Google-Extended is the robots.txt token sites use to opt out of Google’s AI training, and most robots.txt files I audit that mention it are built on a wrong guess about what it covers. The short version: blocking Google-Extended does not touch Google Search. Not rankings, not snippets, not AI Overviews. It controls something narrower, and knowing exactly where the line sits is the difference between an informed opt-out and a gesture.
What is Google-Extended?
Google-Extended is a standalone product token, listed in Google’s crawler documentation, that publishers can target in robots.txt:
User-agent: Google-Extended
Disallow: /The unusual part is that there is no Google-Extended crawler. The documentation is explicit that it does not have a separate HTTP user agent string, and no request in your logs will ever identify itself as Google-Extended. Google fetches pages with its existing crawlers, Googlebot among them, and the token acts purely as a permission check on what the fetched content may be used for afterwards. So you cannot see it crawl, block it by user agent at the firewall, or verify it by IP. robots.txt is the only place the token exists.
What does blocking Google-Extended actually do?
Per Google’s documentation, the token controls whether your content is used to improve Gemini models, the ones behind the Gemini apps and the Vertex AI API, and whether it is used for grounding, which is the step where a model checks its answer against live web content and cites sources.
What it does not control is longer and matters more:
- Google Search indexing and ranking. Google updated the crawler documentation in February 2024 specifically to state that Google-Extended does not affect Google Search, after enough people assumed otherwise.
- AI Overviews and AI Mode. These are Search features, built on the Search index. Google’s documentation on AI features in Search points to the ordinary Googlebot controls for them: block Googlebot and you leave Search entirely, or use
nosnippet,data-nosnippetandmax-snippetto limit what Search surfaces may quote, AI features included. - Every non-Google AI system. OpenAI, Anthropic and Common Crawl each have their own tokens. I listed who obeys what in does robots.txt block ChatGPT, and Anthropic’s three-bot version in does robots.txt block Claude.
Should I block Google-Extended?
It is a real decision with a real trade-off, which is why I will not give one answer for every site. The cost of leaving it open is that your content feeds model training. The cost of closing it is grounding: when Gemini products check answers against the live web, content that opted out is not available to be cited there, so a blocked site gives up whatever referral surface those citations become.
For this site the robots.txt has no Google-Extended group at all, and that is deliberate rather than a default I never looked at. My work depends on being findable and citable wherever answers get assembled, in Search and outside it, and I have written before about why I keep AI systems readable in do I need to optimize for AI Overviews. A publisher whose entire value is the words on the page can reasonably land the other way. What I push back on in audits is only the middle position, blocking Google-Extended while believing it protects content from AI Overviews. That belief buys nothing: the Search-side controls are nosnippet, data-nosnippet, max-snippet and noindex, and they were the answer before this token existed.
How to check what your own site says
Fetch your robots.txt and search it for Google-Extended. If the token appears, read which paths its group disallows, and remember matching is by group: rules under User-agent: * do not apply to Google-Extended once it has a group of its own. If you inherited the file, this is worth two minutes, because I keep finding blocks added during the 2023 wave of AI opt-outs that the current owner does not know exist. The other direction happens too: a site that decided to opt out but put the rule in a group Google never matches. Give any Perplexity rules in the same file a matching read, because only one of Perplexity’s two user agents obeys robots.txt at all.
One more inherited-file trap: the token is only as effective as the robots.txt serving it, so a file that Google cannot fetch reliably takes every token down with it, which is its own failure mode I covered in robots.txt unreachable.
Google-Extended is only one of the tokens people add in a batch, and the others do not behave the same way. Does robots.txt block Claude covers Anthropic’s three bots and the group-selection rule that can quietly cancel a block when you add one.
The other thing to be clear about is what this token was never for. Blocking Google-Extended does not take you out of AI Overviews or AI Mode, and since June 2026 there is a Search Console setting that does. If that is the outcome you were after, keeping your content out of AI Overviews is the control you actually want.
Google-Extended decides what a model may learn from you, not what today’s answers say about you. The measurement for the second thing is in AI search visibility.
Sources
- Google Search Central, overview of Google crawlers and fetchers, the Google-Extended entry and the February 2024 changelog note that it does not affect Google Search, checked 30 August 2026
- Google Search Central, AI features and your website, on Googlebot and snippet controls governing AI Overviews and AI Mode
- Google Search Central, robots.txt introduction, on group matching rules