“ChatGPT does not know my site exists” is now a regular question in my inbox, and almost every article answering it repeats the same five steps without a single link to OpenAI’s own documentation. Some of them are selling a submission product for an index that has no submission endpoint.
There is no ChatGPT equivalent of the URL Inspection tool. You cannot ask OpenAI whether your page is in its index. So the only honest approach is to check each gate in order, confirm what you can confirm, and be clear about where the evidence runs out.
Four crawlers, and only one of them matters for search
OpenAI documents four user agents. Most posts name three and skip the newest one.
OAI-SearchBot is the one that decides whether you show up. OpenAI: “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
GPTBot is for training data. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
ChatGPT-User is a live fetch triggered by a person asking a question. OpenAI notes it “is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.” It also says plainly that ChatGPT-User “is not used to determine whether content may appear in Search.”
OAI-AdsBot validates pages submitted as ads. It does not train models and it has nothing to do with organic citations.
That leads to the single most repeated piece of bad advice in this space: that blocking GPTBot costs you citations. OpenAI says the opposite. “Each setting is independent of the others, for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.” You can refuse to be training data and still be cited.
Source: OpenAI's bots documentation and its ChatGPT search help article.
The gate almost nobody writes about
Here is OpenAI’s own eligibility sentence, and it has two halves: “To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI’s published searchbot IP addresses.”
Robots.txt is only half of it. The host or CDN has to let the requests through, and OpenAI publishes the IP ranges as JSON files, one per agent, so you can check.
This is where most of the real failures I see actually live. Cloudflare classifies AI bots by behaviour, not by name, into groups it calls Search, Agent and Training. A site owner flips on a “block AI scrapers” control, which sounds like a training decision, and it takes out the Search behaviour class. That is exactly OAI-SearchBot. Robots.txt still reads as permissive and nothing in it is wrong. The requests are being refused a layer above.
If you have a permissive robots.txt and no OpenAI traffic in your logs, this is where I would look before anything else. The symptom is a 403 from the edge, which is the same failure mode I covered for Googlebot in blocked due to access forbidden 403.
How I check, in order
Read robots.txt as OpenAI would. A blanket disallow, a wildcard AI block, or a security plugin rule can all catch OAI-SearchBot without ever naming it. I wrote the full walkthrough of that in does robots.txt block ChatGPT.
Grep your access logs for each agent separately. OAI-SearchBot, ChatGPT-User, GPTBot and OAI-AdsBot are four different lines of enquiry. If OAI-SearchBot appears at all, robots.txt is not your problem. If nothing from OpenAI appears anywhere, look at the edge.
Verify the source IPs. Reverse check the addresses against OpenAI’s published JSON lists rather than trusting the user agent string, which anyone can spoof. This also tells you whether the robots.txt fetch itself is arriving, because it comes with its own marker in the user agent.
Then wait a day. OpenAI: “For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” Fix the file, then leave it alone before drawing any conclusion.
What a real citation looks like, and what it does not
Ask ChatGPT something your page is the obvious answer to, and read the response carefully. There are three outcomes and they mean three different things.
A real citation shows a source chip you can hover and click, tied to a specific claim. That is retrieval working.
A navigational link with no substance is what OpenAI describes for opted-out sites: shown as a link, not used as an answer.
A confident description of your business with no source attached at all is the model’s own memory, not retrieval. It proves nothing about your crawlability, and it can be wrong about your prices.
Where the certainty runs out
I would rather say this plainly than pad the post. OpenAI states that ChatGPT search “sometimes partners with other search providers” and links Microsoft and Shopify privacy policies, so Bing is clearly in the mix alongside OpenAI’s own index. What OpenAI does not publish is how much comes from where. Anyone giving you a percentage made it up.
Also undocumented: whether OAI-SearchBot renders JavaScript, how often it recrawls, whether IndexNow reaches OpenAI at all, and whether Bing indexation is required or merely correlated. Three claims you will see stated as fact with no source behind any of them.
And the conclusion I keep arriving at with clients. Once crawl access is confirmed, “not in ChatGPT” is usually not an indexing problem. It is a ranking problem against a query the model rewrote before it searched. Access is table stakes. After that you are competing on the same things you always were, which is what the AI search visibility guide is about, and tracking brand mentions in AI search covers how to measure it over time rather than one prompt at a time.
Sources
- OpenAI bots documentation, listing OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot.
- ChatGPT search, OpenAI Help Center.
- Cloudflare bot categories, Cloudflare developer documentation.