Default robots.txt for WordPress: what core actually serves

Search for the default robots.txt for WordPress and you get templates. Twenty lines, forty lines, blocking /wp-includes/, /wp-content/plugins/, feeds, author archives, trackbacks, readme.html. Every one of them presented as the file your site is missing.

Your site is not missing a file. WordPress is already serving one, most of the recommended lines in those templates are harmful or inert, and the act of creating a physical robots.txt to paste them into turns off the one WordPress was giving you. That last part is the piece nobody states as cause and effect, so let us start with what core actually sends.

What is the default robots.txt in WordPress?

Three lines, from core, with no plugin involved:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

That comes out of do_robots(), the core function that answers requests for /robots.txt. It is worth knowing that the two paths are not hardcoded strings. Core builds them from admin_url(), so a WordPress install in a subdirectory, or one with a moved admin path, outputs a path to match. The Allow line exists because blocking the whole admin directory also blocked admin-ajax.php, which front-end code legitimately calls, and Yoast has said on the record that this was fixed in WordPress 4.4 after they worked with core to remove the old wp-includes disallow in 4.0.

If your site is public you will usually see a fourth line:

Sitemap: https://example.com/wp-sitemap.xml

That one is not from do_robots(). It is added separately by WP_Sitemaps::add_robots(), which has hooked the same output since WordPress 5.5, and it is conditional on the site being public. So the file you see is assembled by two different functions with two different conditions, which is why a private site loses the sitemap line while keeping the admin rules. If wp-sitemap.xml is new to you, core’s own XML sitemap has its own write-up.

Does WordPress create a real robots.txt file?

No, and this is the fact that decides everything else on this page.

WordPress serves a virtual robots.txt through a rewrite rule. The rewrite only fires when the request does not match a real file on disk, which is what the standard Apache RewriteCond %{REQUEST_FILENAME} !-f condition and the equivalent Nginx try_files are doing. So the moment a physical robots.txt exists in your web root, the web server hands that file over and do_robots() never runs.

Everything core was contributing disappears at that instant, silently:

• the Allow: /wp-admin/admin-ajax.php line
• the Sitemap: line pointing at wp-sitemap.xml
• the robots_txt filter, which is how plugins and themes were adding their own rules

Nothing warns you. The site does not error. You simply now own the whole file forever, and it will not pick up changes from core or from any plugin. Rank Math’s own documentation is a good independent confirmation of the precedence: it instructs you to delete the physical file over FTP before its robots.txt editor will do anything at all.

This is also the answer to why you cannot find robots.txt in your WordPress files. There is no file to find until somebody makes one, and making one is a bigger decision than it looks.

Behaviour from the WordPress developer reference for do_robots() and WP_Sitemaps::add_robots(), and the standard WordPress rewrite condition.

Does “Discourage search engines” change robots.txt?

Not in core, and this one gets stated as fact constantly.

do_robots() does read the blog_public option, but it passes that value to the robots_txt filter rather than acting on it. Core itself does not swap in Disallow: / when you tick Discourage search engines from indexing this site. What that checkbox actually does is add a noindex robots meta tag, which is a different mechanism entirely, and which I have written up in the Discourage setting.

The reason it matters here is diagnostic. If you find Disallow: / on a WordPress site, do not blame the checkbox and move on. Something else put it there, usually a staging environment’s physical file that came along with a migration, and until you find that source it can come back.

Should I block /wp-content/ or /wp-includes/?

No. This is the most common recommendation in the templates and it is actively harmful.

Google needs your CSS and JavaScript to render the page, and both live in those directories. Google’s own robots.txt specification carries an example commented to make exactly this point, noting that crawlers disallowed from an includes directory cannot fetch the .css and .js files “but Google needs them for rendering”. The JavaScript SEO documentation is blunter: “Google Search won’t render JavaScript from blocked files or on blocked pages.”

The symptom is easy to recognise once you know it. URL Inspection reports that page resources could not be loaded, and the rendered screenshot shows your page with no styling at all. That is the page Google is judging.

Here is the rest of the template folklore, with a verdict on each:

Rule you keep being told to add	Verdict
Disallow: /wp-includes/	Harmful. Blocks jQuery and core CSS and JS that Google needs to render
Disallow: /wp-content/ or /wp-content/plugins/	Harmful. Blocks theme CSS, plugin JS and your images
Disallow: /wp-admin/	Fine, and already there. Core emits it
Allow: /wp-admin/admin-ajax.php	Redundant. Core has emitted it since 4.4
Disallow: /category/, /tag/, /author/	Harmful. Blocks internal link discovery and produces Indexed, though blocked by robots.txt rows. Use noindex, follow instead
Disallow: /?s=	Counterproductive. Core already noindexes search pages, and blocking the URL stops Google reading that noindex
Disallow: */feed/	Unnecessary. Blocking feeds breaks feed-based discovery for no indexing benefit
Crawl-delay: 10	Inert at Google. Not a supported field
Noindex: inside robots.txt	Inert. Not a supported field, and never was at Google
Disallow: /wp-login.php or readme.html “for security”	Redundant and self-defeating. robots.txt is public, so you are publishing the path you wanted hidden

Google supports four fields and says so: user-agent, allow, disallow and sitemap. Anything else in your file is decoration.

The deeper error under half of that table is treating a disallow as a way to keep pages out of Google. Google states the limit directly: robots.txt “is not a mechanism for keeping a web page out of Google”, and “a page that’s disallowed in robots.txt can still be indexed if linked to from other sites.” If your goal is deindexing, a Disallow rule is not a noindex and never has been.

So what should be in it?

For most WordPress sites, what core already serves. Add a rule only when you can name the URLs it blocks and say what crawling them was costing you.

If you do need a rule, add it through the robots_txt filter in a small plugin rather than by creating a physical file. The filter keeps the virtual file alive, so you keep the core lines and the sitemap line, and your rule survives theme changes and plugin swaps. Yoast leaves the file alone and gives you an editor. Rank Math writes a virtual file of its own with the same core lines plus its own sitemap index.

Three checks I run on any WordPress site before touching anything:

1. Load /robots.txt in an incognito tab. Not the plugin’s editor view, the live response. Those disagree more often than they should when a caching layer is in front.
2. Compare the Disallow path against the real admin URL. On a subdirectory install these come apart, and a rule pointing at the wrong path protects nothing.
3. Confirm with URL Inspection, on a real page, not on the file. The file tells you what you wrote. The live test tells you what Googlebot got, and the resources section tells you whether you have blocked your own stylesheet.

The last one is the one that finds real problems. A robots.txt that looks fine in a text editor and a rendered screenshot with no CSS in it are a common pairing, and only the second one is evidence.

Sources

• WordPress Developer Resources, do_robots(), for the default output, the admin_url()-derived paths and the blog_public handling, checked 7 September 2026
• WordPress Developer Resources, WP_Sitemaps::add_robots(), for the Sitemap line added since 5.5.0 on public sites, checked 7 September 2026
• WordPress Developer Resources, robots_txt filter, for the update-safe way to add rules, checked 7 September 2026
• Google Search Central, Introduction to robots.txt, for “is not a mechanism for keeping a web page out of Google” and the indexed-if-linked limitation, checked 7 September 2026
• Google Search Central, How Google interprets the robots.txt specification, for the four supported fields and the commented includes-directory example, checked 7 September 2026
• Google Search Central, JavaScript SEO basics, for “Google Search won’t render JavaScript from blocked files or on blocked pages”, checked 7 September 2026
• Yoast, WordPress robots.txt: best practice example, for the account of the wp-includes removal in WordPress 4.0 and the admin-ajax fix in 4.4, checked 7 September 2026
• Rank Math, How to edit robots.txt with Rank Math, for its default virtual file and the instruction to delete a physical robots.txt first, checked 7 September 2026