WordPress

WordPress has no robots.txt file by default: it generates one when someone asks for it. If a real robots.txt file exists in the site's root folder, it wins and WordPress's version is never seen.

1. Open Settings → Reading and make sure Search engine visibility (Discourage search engines from indexing this site) is unticked. On a live site it should never be on.
2. Yoast SEO: Yoast SEO → Tools → File editor shows the robots.txt and lets you edit it (if the server allows Yoast to write the file).
3. Rank Math: Rank Math SEO → General Settings → Edit robots.txt (visible in Advanced mode). If a physical file exists, Rank Math says so and you have to edit the file itself.
4. If there is a physical file and no plugin edits it, change it over FTP or your host's file manager (see the cPanel tab).

Then clear the page cache: some caching plugins and hosts cache robots.txt too.

cPanel

1. Open Files → File Manager and go to the site's root folder (usually public_html).
2. Right-click robots.txt and choose Edit.
3. Delete the Disallow: / line (or replace it with Disallow:) and Save Changes.

Plesk (Files) and DirectAdmin (File Manager) work the same way. If there is no robots.txt in the folder, the file is generated by the CMS: see the WordPress tab.

nginx

Usually robots.txt is a plain file in the site's root (the root directive of the server block). Edit it there. Some configurations return it straight from nginx, so search for that too:

sudo grep -rn 'robots.txt' /etc/nginx/

A block like this serves a closed robots.txt no matter what the file says:

location = /robots.txt {
    return 200 "User-agent: *\nDisallow: /\n";
}

Delete it (or change the text), then test and reload:

sudo nginx -t && sudo systemctl reload nginx

Apache and LiteSpeed

Edit the robots.txt file in the site's DocumentRoot. If it looks right but the server returns something else, search the configuration for a rewrite or alias:

sudo grep -rn -i 'robots' /etc/apache2/ /var/www/

(/etc/httpd/ instead of /etc/apache2/ on Rocky, Alma and CentOS.) If you changed the virtual host, apply it:

sudo apachectl configtest && sudo systemctl reload apache2

Node.js and Next.js

In the Next.js App Router, robots.txt is either a static app/robots.txt or generated by app/robots.js:

export default function robots() {
    return {
        rules: { userAgent: '*', allow: '/' },
        sitemap: 'https://example.com/sitemap.xml',
    };
}

Look for disallow: '/' there, often behind a condition on the environment. In other Node apps, robots.txt is usually a file in public/.

Laravel

A new Laravel project ships public/robots.txt with an open rule. Edit the file there. If a route serves robots.txt instead (search routes/web.php for robots), check what it returns on production, and that APP_ENV=production is set in the live .env.

Shopify

Shopify generates a sensible robots.txt on its own. It only changes if the theme has a robots.txt.liquid template: Online Store → Themes → … → Edit code → Templates. If it exists and adds Disallow: /, remove that rule, or delete the template to return to Shopify's default.

Wix

Wix generates robots.txt itself. If someone edited it, open the SEO settings in the dashboard, go to the Robots.txt Editor, and remove the Disallow: / rule or reset the file to the default.

Webflow

Open Site settings → SEO, and in the robots.txt field remove Disallow: /. Save and Publish the site: the change goes live only with the next publish.

Squarespace

Squarespace generates robots.txt and it cannot be edited. Its default does not block the whole site. If the checklist reports it anyway, check that the site is public (not private or password protected).