Does robots.txt block Claude? Yes, Anthropic says its bots honour it, but only for the bot you actually named. Anthropic runs three separate crawlers with three different jobs, and the one-line Disallow most articles hand out blocks one of them and leaves the other two walking straight in.
This is the third of these I have written, after the same question for ChatGPT and the Perplexity version, and the shape is by now familiar: every AI company splits training from user-triggered fetching, and every guide on the internet collapses that back into one rule.
The three bots and what each one is for
Anthropic’s own crawler page opens with “Below is information on the three robots that Anthropic uses”. Three, not one. Here is what it says each does, and what it says you lose by blocking it.
ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Block it and “it signals that the site’s future materials should be excluded from our AI model training datasets.”
Claude-User “supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Block it and you “prevent our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.”
Claude-SearchBot “navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.” Block it and you “prevent our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results.”
The split matters because most people want two different things from those three rows. Keeping your writing out of a training set and being findable when somebody asks Claude a question are not the same decision, and the one-line block treats them as if they were.
Does Anthropic actually respect robots.txt?
It says so, for all three: “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” It also says “Anthropic’s Bots respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.)” and that “we support the non-standard Crawl-delay extension to robots.txt.”
That last one is unusual and worth noting, because Google ignores Crawl-delay entirely. It is also, as the next section shows, the exact line that trips people up.
There is one more instruction on that page that contradicts what a lot of security blogs recommend: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” Block the IPs and the crawler can no longer fetch the file that tells it to stay away. Anthropic publishes its address ranges so you can verify a visitor, not so you can firewall the opt-out.
The mistake that silently cancels your block
This is the part no competing article covers, and it is the reason I see robots.txt files that look protective and do nothing.
Say your file already has a general group:
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/You then read Anthropic’s page, see the Crawl-delay example, and add this at the bottom:
User-agent: ClaudeBot
Crawl-delay: 1You have just removed every restriction from ClaudeBot. The robots.txt standard, RFC 9309, says a crawler “MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group”, and that the * group applies only “if no matching group exists”. Groups are selected, not merged. The moment ClaudeBot has a group of its own, the * group stops applying to it, and its group contains no Disallow at all. The same thing happens if a stray blank line separates User-agent: ClaudeBot from the Disallow: / underneath it.
So if you name a bot, that group has to be complete on its own. Anthropic’s example of a full block is exactly right, and the important detail is that it is repeated per bot:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /Anthropic also says to “do this for every subdomain that you wish to opt out from”, which catches out anyone running a shop or a docs site on its own subdomain. And robots.txt is still not an index removal mechanism, in this context or any other: Disallow is not noindex, it only asks a crawler not to fetch.
On WordPress, the default robots.txt is generated by WordPress itself and does not exist as a file, so editing it means either a real file or a plugin filter. Where robots.txt lives in WordPress covers that.
How do I check if ClaudeBot visited my site?
Read your access logs. This is the step everyone skips, and it is the only way to know whether anything you wrote actually worked.
grep -iE 'ClaudeBot|Claude-User|Claude-SearchBot' access.log | awk '{print $1}' | sort | uniq -c | sort -rnThat gives you hit counts per IP. Take those addresses and check them against the range list Anthropic publishes: it says “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” Anything claiming to be ClaudeBot from an address outside those ranges is somebody else wearing the name, and no robots.txt rule will ever stop it, because a spoofer was never reading the file.
Two expectations worth setting. First, the drop is gradual, not instant, because robots.txt gets cached. Judge it over days. Second, Anthropic publishes user-agent tokens, not full header strings, so match on the token rather than trying to pin an exact Mozilla/5.0 (compatible; ...) line.
While you are in there, be sceptical of any tool or listicle still telling you to block anthropic-ai or Claude-Web. Those names circulate widely and neither appears on Anthropic’s current crawler page. I am not going to claim they are definitively retired, because Anthropic has not said so in as many words. I am saying they are not documented today, and a rule targeting an undocumented token is a rule you cannot verify.
Will blocking Claude hurt my Google rankings?
No, and it is worth being blunt about it because the fear is common. Googlebot does not read your ClaudeBot rules. The lever on the Google side is a separate token, and Google states its effect plainly: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Google-Extended in robots.txt has the detail.
So the two decisions are independent. Blocking Anthropic costs you Claude visibility, on Anthropic’s own account of it, and costs you nothing in Google.
What I do
My own robots.txt has no Claude group in it at all. It is one User-agent: * block that allows everything except /reports/, which holds client deliverables that are public URLs but are not mine to put in anyone’s index. All three Anthropic bots fall through to that group and are allowed, and that is deliberate rather than an oversight: this is a site whose whole job is to be found by people looking for indexing help, and Anthropic’s own middle row says the plain version of it. Blocking Claude-User is blocking the fetch that happens because somebody asked a question this site can answer.
It is also the cleanest illustration of the group rule above. The day I add User-agent: ClaudeBot for any reason, /reports/ stops being disallowed for ClaudeBot unless I repeat that line inside the new group.
If you want the training opt-out without the visibility cost, block ClaudeBot only and leave Claude-User and Claude-SearchBot alone. If you are a publisher whose business is the content itself, block all three and accept the trade. What I would not do is copy a single-line rule off a blog, never check the logs, and assume it worked. That is how sites end up with a robots.txt that reads like a policy and behaves like a welcome mat.
Getting the agents right makes you fetchable. Whether that turns into a mention is a separate question, and AI search visibility covers how to find out.
Sources
- Anthropic Support, does Anthropic crawl data from the web, and how can site owners block the crawler, for the three bots, what each does, what disabling each one costs, the robots.txt and Crawl-delay statements, the per-subdomain instruction and the warning about IP blocking, checked 4 September 2026
- Anthropic, published crawler IP ranges, linked from the page above, for verifying that a visitor claiming to be a Claude bot really is one, checked 4 September 2026
- IETF, RFC 9309, Robots Exclusion Protocol, for group matching, the case-insensitive product token rule, and the
*fallback applying only when no matching group exists, checked 4 September 2026 - Google Search Central, Google common crawlers, for the Google-Extended statement that it does not affect inclusion or ranking in Google Search, checked 4 September 2026