Blocked by robots.txt is the Page indexing status most likely to be correct behaviour wearing a warning colour. The report is telling you Google wanted to fetch a URL and your robots.txt said no. Whether that is a problem depends entirely on whether you meant it, and in the WordPress audits I do, about half the time the site owner has never seen the rule doing the blocking.
What the status actually means
Google’s help entry for the Page indexing report is short and carries one sentence that surprises people: the block “does not guarantee that the page won’t be indexed through some other means”. If Google can find information about the page without loading it, mostly from links pointing at it, the URL can still end up in the index. That version of events has its own status, and I covered it separately in Indexed, though blocked by robots.txt. The status this post is about is the quieter sibling: blocked, and not indexed either.
The robots.txt documentation is blunter than most people expect about what the file is for. It manages crawler traffic; in Google’s own words, “it is not a mechanism for keeping a web page out of Google”. Those are two different jobs, and the whole confusion around this status comes from using the file for the second job when it only does the first.
The decision is a two-question flow
Every URL in this status resolves with the same two questions.
The third branch is the one that trips people, and it is the entire subject of noindex vs robots.txt: a noindex directive lives on the page, and a page Google is forbidden to fetch is a page whose directives Google never sees. Blocking and noindexing the same URL feels like wearing both belt and braces, and it actually disables the noindex.
What I find behind this status on WordPress sites
The rule doing the blocking is usually one of a small set:
- A leftover development block. A
Disallow: /written during staging and carried to production, or the WordPress “discourage search engines” setting doing its subtler equivalent. I wrote up what that checkbox really changes in discourage search engines from indexing this site. - An SEO plugin’s opinion. Yoast, Rank Math and their peers write robots rules of their own, and a migration between plugins can leave both sets active. Finding which file is even answering is its own job on WordPress, because the file is usually virtual; where is robots.txt in WordPress covers the resolution order.
- Faceted and filter URLs blocked on purpose. Blocking
/?filter=and/?orderby=parameter URLs is legitimate crawl management on WooCommerce stores, and those URLs sitting in this status is the system working. This is the “leave it alone” branch. Which parameters to block, and why the rules have to go on after the noindex rather than before, is in faceted navigation SEO for WooCommerce and Shopify. - CSS, JS or API paths blocked by a rule written for pages. A broad
Disallow: /wp-includes/era rule can catch assets that rendering needs. Google renders pages; starve the renderer and the pages that depend on those assets are judged on incomplete information.
One more failure mode belongs on the list because it produces confusing report data rather than a clean block: robots.txt files that do not answer properly at all. If the file itself returns a 4xx, Google’s robots.txt spec says all 4xx errors except 429 are treated as if no robots.txt existed, meaning no restrictions at all, which is the opposite of what a blocked status implies. And if it returns a 5xx, the failure modes get stranger still; I traced those in failed: robots.txt unreachable. When this status appears alongside crawl anomalies, I check the file’s own HTTP status before reading any of its rules.
How I check a specific URL
Fetch your live file at /robots.txt and read it against the failing URL from the report, remembering that the longest matching rule wins and that Allow and Disallow of equal length resolve to allow. For anything non-obvious I stop eyeballing and use the URL Inspection tool in Search Console, which reports the exact verdict Google’s parser reached, including which rule matched. The report’s example URLs also carry a last-crawled date: a URL blocked months ago may already be resolved, because this report trails reality by design.
The count on this status matters less than its trend and its contents. Fifty blocked filter URLs on a store is Tuesday. Five blocked URLs that turn out to be your money pages is an emergency no bigger in the report than the filters were. Read the URLs, not the number.
If one of those URLs is a page you wanted in Google, check what it looks like in the results before you fix anything. A blocked URL that Google has indexed anyway shows no information is available for this page instead of a description, which is the same condition seen from the searcher’s side.
Sources
- Google Search Console Help, Page indexing report, the URL blocked by robots.txt status, including the note that blocking does not guarantee non-indexing, checked 2 September 2026
- Google Search Central, introduction to robots.txt, for the file’s purpose as crawl management and the warning against using it to hide pages, checked 2 September 2026
- Google Search Central, how Google interprets the robots.txt specification, for 4xx handling being treated as no restrictions and the rule precedence behaviour