“Failed: Robots.txt unreachable” is what Search Console shows under Page fetch when you inspect a URL and Google could not read your robots.txt file at the moment it tried. The page itself may be perfect. Google never got as far as looking at it.
This one is worth understanding properly because it behaves unlike every other indexing error. Most errors are per URL: one page has a noindex, one page returns a 404, you fix that page. Robots.txt unreachable is per host. If Google cannot fetch the file, the rules for your entire site are unknown, and a crawler that does not know the rules does not guess. It stops.
What “robots.txt unreachable” actually means
Not missing. Unreachable. The difference decides everything you do next, and it is the single most common misunderstanding I run into on this error.
A robots.txt that returns 404 is fine. Google’s crawl stats documentation says so directly: a 404 for robots.txt is considered a good response, and it is not the same thing as the file being unavailable. No file means no restrictions, so Google crawls everything and moves on. Plenty of small sites have no robots.txt at all and index perfectly.
Unreachable means Google asked and got something that was neither a valid file nor a clean 404. In practice that is one of:
- A 5xx response. The server was up enough to answer and the answer was an error. On any other URL that is the ordinary server error (5xx) status; on robots.txt it triggers everything below.
- A timeout, a reset connection, an invalid response, an HTTP chunking error, or a DNS failure. Google’s robots.txt specification says these are all treated as a server error.
- A redirect chain longer than five hops. Google follows at least five and then gives up and treats it as a 404 for robots.txt.
- A 403 or 401 on the file, often from a firewall or a bot protection rule that decided Googlebot looked suspicious. Technically Google treats 4xx as “no restrictions”, but if your firewall is blocking the robots.txt fetch it is usually blocking page fetches too, and those show up separately as blocked due to access forbidden (403). That is the real problem.
How long does Google stop crawling if robots.txt is unreachable?
There is a documented clock, and almost nobody knows it. Straight from the specification, for the case where Google finds a robots.txt file but cannot fetch it:
Two things fall out of that which change how you triage.
First, twelve hours of nothing. If your server had a bad hour overnight and robots.txt 5xx’d through it, crawling on your site stopped for half a day. Nothing appears in the Page indexing report to announce that, because the pages were never fetched. You just get a quiet dip.
Second, the cache is doing you a favour you cannot see. Google caches robots.txt for up to 24 hours normally, and longer when it cannot refresh it. That cached copy is what keeps a site crawlable through a multi-day outage. It also means a robots.txt change you made this morning may not be in effect yet, which cuts the other way when you are trying to unblock something urgently.
How to check whether it is really unreachable
In order, fastest first:
- Fetch the file yourself and look at the status code, not the content.
curl -I https://example.com/robots.txtand read the first line. A 200 or a 404 is fine. A 500, 502, 503, or a hang is your answer. - Fetch it as Googlebot. Same request with a Googlebot user agent string. If that one fails and the browser one worked, you have a bot protection or WAF rule, not a server problem, and the fix is an allowlist entry rather than anything on the site.
- Search Console, Settings, Crawl stats, then host status, then robots.txt fetching. The graph shows the failure rate for robots.txt requests over time, with a threshold line. This is the only place that tells you when it happened rather than whether it is happening now. Note that crawl stats is missing 15 and 16 August 2026 for many properties at the moment, so there is a hole in that history this month.
- Check the redirect chain if the file redirects at all. Five hops is the documented limit and I have seen sites burn three of them on protocol, host, and trailing slash normalisation before anything intentional happens.
One number in that report catches people out and it is worth knowing while you are in there: if robots.txt has been unavailable for a while, total crawl requests can read far higher than your server logs show. Google counts the crawls it would have made but did not. A crawl total that outruns your logs is a symptom of this error, not evidence of heavy crawling.
The fix order
Nothing here is clever. It is the order that matters.
Make robots.txt return 200 or 404 reliably, and serve it as static as you can. The most frequent cause I see is a robots.txt generated by the application: WordPress plugin, framework middleware, a route in the CMS. On WordPress this is the default rather than the exception, because WordPress builds robots.txt per request instead of keeping a file in the root. When the application is under load or the database is unhappy, the application returns 500, and robots.txt goes down with it. A static file on disk, or one served by the CDN, does not care what the application is doing. That single change removes the whole class of failure.
Then check the file is not varying by user agent. Google’s documentation asks for this explicitly: do not cloak robots.txt to Google, and do not vary it by user agent. Some security plugins and CDN rules do this without telling you.
Then wait, and verify with a live test rather than assuming. Run the live test in URL Inspection on any URL on the host. If the fetch succeeds now, the host is unblocked and crawling resumes; the twelve hour stop applies while the file is unreachable, not as a penalty afterwards. What you should not do is request indexing on fifty URLs to force it. Crawling resumes when the file is readable, and a queue of indexing requests does not make robots.txt fetchable.
Once crawling is back, the pages that were affected will show ordinary statuses again, and any that are still stuck belong to a different problem: see every Search Console indexing status for the one you actually have, and URL is not available to Google if the live test is still failing rather than the indexed data being stale.
Why this one is worth being fussy about
Because it is invisible and it is total. A noindex on one page costs you one page. A robots.txt that flakes for three hours a week costs you crawling across the whole host, repeatedly, and it shows up in your data as a slow indexing problem that no amount of content work fixes. When a site is not getting new pages indexed and everything about the pages looks fine, host status is on my short list of things to check before I touch anything on the page level, alongside whether the sitemap itself still fetches and the internal linking. It is a five minute check and it is occasionally the entire answer.
While you are looking at robots.txt at all, one adjacent thing worth being sure of, because mixing them up causes its own outages: robots.txt controls crawling, not indexing, and blocking a page there does not remove it from Google. That distinction is written out in full in noindex vs robots.txt.
And if you are editing robots.txt with AI crawlers in mind while you are in there, the user agent names matter more than people expect: does robots.txt block ChatGPT covers which of the four OpenAI bots does what.
Sources
- Google Search Central, how Google interprets the robots.txt specification, error handling, caching, and HTTP status code sections
- Google Search Console Help, Crawl Stats report, host status and bad response codes sections
- Google Search Console Help, URL Inspection tool, live test