Where is the robots.txt file in WordPress? It is not anywhere. You can connect over SFTP, list every file in the WordPress root, and find no robots.txt sitting next to wp-config.php, because on a standard install WordPress does not keep one. It builds the response fresh every time something asks for it.
That is the whole answer, and it explains most of the confusion that follows from it: why there is nothing to download, why the contents change when you change an unrelated setting, and why the plugin panel you type into can disagree with what a crawler actually receives.
Why there is no file in your WordPress folder
WordPress serves /robots.txt from a function, not from disk. The function is do_robots(), and its full source is published in the developer reference. It sets a plain text content type, fires an action named do_robotstxt, assembles a few lines of text in memory, and echoes them through a filter.
The request never touches a file, so there is nothing to find and nothing to edit in place. This is also why the advice “just edit your robots.txt” fails on a default WordPress site. There is no document for that instruction to apply to.
What WordPress puts in robots.txt by default
The default output is three lines, and they are visible in that same published source:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThat is deliberately small. It keeps crawlers out of the admin directory while leaving admin-ajax.php reachable, because plenty of front end functionality routes through it.
Since WordPress 5.5 there is usually a fourth line. The core sitemaps feature appends the sitemap index, and the source of that method carries a condition that matters more than it looks:
public function add_robots( $output, $is_public ) {
if ( $is_public ) {
$output .= "\nSitemap: " . esc_url( $this->index->get_index_url() ) . "\n";
}
return $output;
}The sitemap line is added only when the site is public. Set the site to non public and the line disappears, with no warning anywhere in the admin.
Assembled from the published source of do_robots() and WP_Sitemaps::add_robots() in the WordPress developer reference.
What quietly rewrites it
Three things change that output, and only one of them is obvious.
A real file wins. If anyone has ever uploaded an actual robots.txt to the site root, the web server hands that file over and WordPress is never consulted. The WordPress documentation makes the same point in reverse when it notes that the site visibility setting only ever affected robots.txt when no real file existed. A stale file left by a previous developer keeps overriding everything you do in the admin, and nothing in WordPress will tell you it is there.
The site visibility setting. As above, switching the site to non public strips the Sitemap: line. What it no longer does is add a sitewide Disallow: /. That behaviour changed in WordPress 5.3, and the setting now prints a noindex, nofollow meta tag instead, which is a different mechanism with a different recovery path. I wrote that one up separately in discourage search engines from indexing this site, because it is the most common single cause of a WordPress site vanishing from Google.
Plugins, through the filter. do_robots() echoes its output through apply_filters( 'robots_txt', $output, $public ), a filter that has existed since WordPress 3.0. Every SEO plugin with a robots.txt editor is hooked there. The box you type into in Yoast or Rank Math is not writing a file, it is supplying a string to that filter. That is exactly why the plugin editor and the live output can disagree when a real file is also present: the plugin is filtering something the server never gets asked for.
How to edit robots.txt in WordPress
There are three routes, they do not stack politely, and the one most people reach for is the one with the least control.
Through the filter, which is what WordPress actually intends. do_robots() ends by passing its assembled string through apply_filters( 'robots_txt', $output, $public ). Hooking that from a small site specific plugin is the core native way to change the output:
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) {
return $output;
}
$output .= "Disallow: /?s=\n";
return $output;
}, 10, 2 );Put it in a plugin rather than in a theme’s functions.php, or the rule leaves with the next theme switch.
Through an SEO plugin. Yoast and Rank Math both ship a robots.txt editor. Neither writes a file. They are hooked to the same filter as above, so the panel is a text box wired to robots_txt with a UI on top. That is fine, and it is worth knowing there is nothing special happening underneath, because it explains the failure mode: if a real file exists, the plugin is still filtering a response the server never generates, and the editor will keep showing you content nobody receives.
By uploading a real file. Drop a robots.txt into the site root and it wins outright. The web server answers the request directly and WordPress never runs, so the filter never fires and every plugin editor on the site becomes decorative. This is occasionally the right call, on a host where you want the file under version control rather than under a plugin. It is more often an accident, and it is the single most common reason a WordPress robots.txt refuses to change no matter what you edit.
Pick one route and stay on it. The debugging cost here comes almost entirely from sites running two at once.
What belongs in a WordPress robots.txt
For most sites, the default plus the sitemap line. That is not a cop out, it is the honest answer: robots.txt controls crawling, and on a normal WordPress site there is very little you want to stop Google from crawling.
The additions that are genuinely defensible are narrow. Internal search result pages, /?s=, are worth disallowing on a busy site, because they generate unbounded low value URLs that exist only because somebody typed something. Beyond that, be suspicious of any rule you cannot justify by naming the crawler and the thing you are stopping it fetching.
The rules to avoid are more useful to list, because they are copied between tutorials constantly:
- Do not block
/wp-content/uploads/. That is where your images live. Blocking it removes them from Google Images, which for a lot of sites is real traffic thrown away for no benefit. - Do not block CSS or JavaScript, which usually means not blocking
/wp-includes/or/wp-content/plugins/wholesale. Google renders pages before judging them, and its own guidance on creating a robots.txt file is explicit that blocking resources needed to render the page can change how the page is understood. A blocked stylesheet does not produce an error anywhere. It produces a worse assessment, silently. - Do not use it to hide anything. The file is public, permanently, at a predictable URL. Listing a directory you would rather nobody found is an index of exactly that.
- Do not disallow a page you want deindexed. This is the one that costs the most. A blocked URL cannot be crawled, so Google cannot see the noindex tag you put on it, and the page can persist in results as a bare link. That whole trap is noindex versus robots.txt, and it is worth reading before you disallow anything you meant to remove.
How do I check what Google actually sees?
Do not read the plugin’s editor screen and assume. Request the URL and read the response, because that is the only version a crawler will ever see.
If you have Search Console access, the robots.txt report shows the fetched content, the fetch status, and when it was last read, which also tells you whether Google is getting a clean 200 at all. A failed fetch is its own separate problem and it reads as failed: robots.txt unreachable rather than as a rule you wrote wrong. A robots.txt returning a server error is treated very differently from one returning 404, and neither of those is what you want.
Also keep straight what the file can and cannot do. It controls crawling, not indexing, and a URL blocked in robots.txt can still surface in results without a snippet. That distinction is the whole of noindex versus robots.txt, and it is worth being certain about before you disallow anything.
What I do on WordPress builds
On the three WordPress locksmith sites I built location pages for, robots.txt was the first thing I read and the last thing I touched. Reading it means fetching the URL, not opening a plugin panel. Touching it usually means leaving it alone, because the WordPress default is already right for almost every site: it blocks the admin directory, it keeps admin-ajax.php open, and it advertises the sitemap.
The few times I have changed it, the reason was never to keep Googlebot away from content. Blocking content you want indexed is how pages get stranded with nothing in the index and no error to explain it. The changes worth making are narrow, and if you cannot name precisely which crawler you are stopping and from fetching what, the honest answer is to leave the default alone.
The honest caveat
I have described what WordPress core does, from core’s own published source, and that is where my confidence ends. Managed hosts do intercept robots.txt at the server or CDN layer, and some security plugins write a real file to disk without announcing it. If the served output matches nothing in this article, that mismatch is your answer, and the way to find the cause is to keep reading the response rather than to keep editing settings.
If what you actually wanted was the contents rather than the location, the default robots.txt WordPress serves goes through the three lines core outputs, where the sitemap line comes from, and why most of the rules the template posts tell you to add are harmful.
Sources
- WordPress developer reference,
do_robots(), for the default output lines, thedo_robotstxtaction added in 2.1.0, and therobots_txtfilter added in 3.0.0. - WordPress developer reference,
WP_Sitemaps::add_robots(), introduced in 5.5.0, for the sitemap line and its public only condition. - WordPress documentation, Settings Reading screen, for the site visibility setting and the note that its old robots.txt behaviour required no real file to exist.
- Google Search Central, Introduction to robots.txt, for what robots.txt controls and what it does not.
- WordPress developer reference,
robots_txtfilter, for the hook every SEO plugin editor writes through. - Google Search Central, Create and submit a robots.txt file, for the warning on blocking resources needed to render a page.