“Blocked GPTBot in robots.txt” is usually said as though it settled something. It does not, because OpenAI does not run one crawler. It runs four, they do different jobs, and OpenAI’s own bot documentation says outright that each setting is independent.
Getting this wrong goes both ways. People who wanted to stay out of AI answers are still in them. People who wanted to be cited have quietly excluded themselves.
The four bots and what each one is for
GPTBot is the training crawler. Blocking it is about model training. It has nothing to do with whether you appear in ChatGPT’s answers today.
OAI-SearchBot is the one that matters for visibility. It exists for search in ChatGPT, and the documentation is explicit about the consequence: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing it.
OAI-AdsBot validates advertiser landing pages. The docs state that data it collects is not used for training.
ChatGPT-User is the one that breaks the mental model, and it deserves its own section.
Roles and consequences as stated in OpenAI's bots documentation.
The bot that does not follow your robots.txt
Here is the sentence worth reading twice, from OpenAI’s docs on ChatGPT-User: “ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.”
The reasoning is coherent. If a person pastes a URL into ChatGPT and asks what is on the page, that is a person fetching a page, in the same category as a browser. robots.txt governs automated crawling, and this is not that.
The practical consequence is the part I had to think about on my own site. My robots.txt disallows /reports/, which holds client deliverables. They are public URLs so a client can open one without a login, and they are not mine to put in Google. That disallow works fine for search engines. It does not stop ChatGPT-User from fetching one if somebody hands ChatGPT the link, and neither would blocking every OpenAI agent by name.
That is not an OpenAI problem. It is the same thing robots.txt has always been: a request to well-behaved automated crawlers, not an access control. If a URL genuinely must not be read by whoever holds the link, robots.txt was never the mechanism. Authentication is. I know exactly which of my URLs are in that category, and the honest answer is that “unlisted” is the protection they have, which is a decision rather than an accident.
The docs add one more thing about ChatGPT-User: it “is not used to determine whether content may appear in Search.” So it is not a visibility lever either. It is neither a way in nor a way out.
What to actually put in robots.txt
If you want to be cited in ChatGPT, allow OAI-SearchBot. That is the whole intervention. It is worth checking rather than assuming, because plenty of sites picked up a blanket AI-bot blocklist during the 2023 wave and never revisited it. If yours did, you may have been opted out of ChatGPT search for two years without deciding to be.
If you want out of training but not out of citations, disallow GPTBot and allow OAI-SearchBot. Those settings are independent, which is the entire reason there are separate user agents.
If you want out of everything, disallow GPTBot, OAI-SearchBot and OAI-AdsBot, and accept that ChatGPT-User may still fetch a page a person explicitly asks about.
Two operational notes from the docs. OpenAI says it can take around 24 hours from a robots.txt update for its systems to adjust for search results, so do not conclude anything from a test five minutes after saving. And each bot publishes its IP ranges as JSON at openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json, which is how you verify a hit in your logs was really OpenAI rather than something wearing the user agent string.
The Google half of the same question
Google splits the same way, with different names. Googlebot’s robots.txt rules are the control for Search, AI Overviews and AI Mode included, and there is no separate opt-out for AI features. Google-Extended is the separate token for training and grounding in some other Google systems. I covered what that leaves you with in do I need to optimize for AI Overviews, and the short version is that the answer is the ordinary one. Perplexity splits the same way yet again, one indexing bot that obeys robots.txt and one user-triggered fetcher that mostly does not, which I mapped in does robots.txt block Perplexity. Anthropic runs a third variant of the same split, three bots instead of two, which does robots.txt block Claude works through.
Which is worth saying plainly, because it is the useful conclusion in both cases: being readable to AI systems is downstream of being crawlable and indexable at all. If pages are not getting indexed, no user agent line fixes that, and the diagnosis starts at every Search Console indexing status.
What I have not verified
The other half of this decision is what you offer the crawlers you do allow, which is what llms.txt and its v2 revision are for. Blocking decides who gets in. That file decides what they read first. And a third question arrived in August 2026, when browsers started reading tools a page declares in JavaScript: I went through what WebMCP is and whether it touches SEO, and the answer rhymes with this one.
I have taken OpenAI’s documentation at its word about what each bot does and does not do. I can verify from logs that a request claiming to be a given user agent came from a published IP range. I cannot verify what happens to the data afterwards, and neither can you. That part is a policy statement, not a measurement.
Anthropic splits its crawlers the same way, into three rather than four, and adds a robots.txt trap of its own that can silently undo the block you just wrote. Does robots.txt block Claude is the same question answered for ClaudeBot, Claude-User and Claude-SearchBot.
Access is the first thing to rule out when ChatGPT never cites you, and it is rarely the only thing. The rest of the list, in order, is in how to improve brand visibility in AI search engines.
Robots.txt is only the first of three gates. If yours is permissive and ChatGPT still does not know you exist, why ChatGPT cannot see my website covers the CDN layer and the log checks that come next.
Sources
- OpenAI, Overview of OpenAI Crawlers, for the four user agents, the independence of each setting, the OAI-SearchBot opt-out consequence, the ChatGPT-User robots.txt statement, the roughly 24 hour propagation note, and the published IP range files.
- Google Search Central, AI features and your website, for Googlebot’s robots.txt being the control for AI features in Search.
- Google Search Central, Google-Extended, for the separate training and grounding token.