free tool · no login
Robots.txt Tester
Type a domain and this robots.txt tester reads the live file the way the crawlers do. It tells you, crawler by crawler, whether Googlebot, Bingbot, ChatGPT, Claude, Perplexity and the AI training bots may fetch your homepage and any page you name, and which line in the file made that call. It works as a robots.txt checker for any public site, not just your own.
- 21 crawlers in three groups: search engines, AI search bots, and AI training bots.
- Test any other crawler by name, against any page.
- Google's own rules for reading the file, a copy-paste fix, and nothing stored.
Test a site's robots.txt
01 Type the site
A domain like
example.comor any URL on the site. Only the domain is used to find robots.txt.The homepage is always checked. Add a page path to test one section, like your blog or your shop, and add any crawler's name, as robots.txt writes it, to test that one against the same page.
02 Run it
nothing checked yet
What each crawler is allowed to read
Search engine crawlers
The search engines. Googlebot also feeds AI Overviews and AI Mode, so blocking it costs you both.
| Bot | Used for | Homepage | Your page | Why |
|---|
AI search and answer bots
These fetch pages to show, quote or link them in an AI answer. Blocking one takes you out of that product's answers.
| Bot | Used for | Homepage | Your page | Why |
|---|
AI training bots
These collect pages to train future models. Blocking one does not remove you from any search or answer product.
| Bot | Used for | Homepage | Your page | Why |
|---|
llms.txt
Copy-paste fixes
Pick the one that matches what you want and paste it into your robots.txt, above your other groups. A bot that has its own group ignores the User-agent: * group, so the first snippet carries your existing * rules over for you.
How it works
It reads the file the crawlers read
A small reader on this domain fetches
/robots.txtfrom your site, following redirects the way a crawler would, and checks whether/llms.txtexists. Your browser cannot fetch another site's files directly, which is the only reason the reader exists.It finds the group each crawler obeys
A crawler follows the group that names it and ignores every other group, including
User-agent: *. Only when nothing names it does it fall back to*. This is the rule people miss most: one group naming GPTBot quietly replaces your site-wide rules for it.It picks the rule that wins
Inside that group the longest matching rule wins, and Allow wins a tie.
*and$work as Google documents them. The report shows the exact rule, so you can find it in your file.
The status of the file matters too. A missing robots.txt (404) means every crawler may fetch everything. A server error (5xx) or a rate limit (429) on robots.txt is read by Google as "stay out of the whole site", and the tester reports it that way, because that is what the crawlers do. Google also reads only the first 500 KB of the file, and so does this tool.
Three kinds of crawler, three different decisions
Search engine crawlers decide whether you rank at all, and Googlebot now also feeds AI Overviews. AI search bots decide whether ChatGPT, Claude and Perplexity can show and link your pages in their answers today. AI training bots only decide whether your pages go into the next model's training data. Many sites want the first two in and the third out, and that is a perfectly reasonable choice. This site lets all three in, because I want the models to know my work.
Search engine crawlers
Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, Applebot, YandexBot
AI search and answer bots
OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, DuckAssistBot
AI training bots
GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider
For the details of each AI company's crawlers, readdoes robots.txt block ChatGPT,does robots.txt block Claude anddoes robots.txt block Perplexity. If you want training bots out without touching Google, this post walks through it, and this one explains Google-Extended. The bigger picture, beyond access, is in theAI search visibility guide. If Search Console already shows pages as blocked by robots.txt, start there.
What this tester cannot see: firewall and CDN rules, which can block a crawler before it reads robots.txt, and noindex or nosnippet tags on the page itself (theindexability checker reads those, and the canonical). It reads robots.txt as served to a normal browser; a site that serves crawlers a different file will differ. The reader keeps no copy and writes no log. Full detail on the privacy page.
Straight answers
What happened to Google's robots.txt tester?
Google removed the robots.txt tester from Search Console at the end of 2023. Its replacement, the robots.txt report under Settings, shows which robots.txt files Google fetched, when, and any lines it could not read. It does not let you type a URL and ask whether a crawler may fetch it. This tool does that, for Googlebot and for the other crawlers people now care about.
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot collects pages for training. ChatGPT search uses a different crawler, OAI-SearchBot, and pages a user asks ChatGPT to open are fetched as ChatGPT-User. You can block GPTBot and still be found and cited in ChatGPT answers, as long as OAI-SearchBot is allowed. The same split exists at Anthropic (ClaudeBot for training, Claude-SearchBot for search) and at Google (Google-Extended for Gemini training, Googlebot for Search and AI Overviews).
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are built from Google Search, which is crawled by Googlebot. Google-Extended is not a crawler at all. It is a name Google reads in robots.txt to decide whether pages it already fetched may be used for Gemini training. Blocking it changes nothing in Search.
The tester says a page is blocked. Does that keep it out of Google?
Not always. robots.txt stops crawling, not indexing. A blocked page that other sites link to can still show in Google as a bare URL with no description, which Search Console reports as "Indexed, though blocked by robots.txt". To keep a page out of search, let Google crawl it and use a noindex tag. Here is when to use which.
My robots.txt allows everything but a bot still cannot get in. Why?
robots.txt is only a request. A firewall, a CDN bot setting or a security plugin can still refuse the crawler before it ever reads the file. Cloudflare in particular has a one-click setting that blocks AI bots at the network edge. If this test says a bot is allowed and the bot's owner says it cannot reach you, look at those settings next.
Do I need an llms.txt file?
No. It is an optional summary file for AI tools, and no crawler needs it to read your pages. The tester tells you whether one exists because people ask, but robots.txt is what decides access. For the background, here is what llms.txt is and what changed in v2. If you want one anyway, the llms.txt generator drafts it from your sitemap and checks an existing file.
Not sure what your robots.txt should say? Send me the domain and I will tell you, free.Send it on WhatsApp.