Block AI training without blocking Google: that is the exact promise of a setting Cloudflare shipped on 15 September, and for once the promise matches the mechanics. It is called Disallow AI Training, and it solves the problem I hear from site owners constantly, which is that the crawlers reading your site for search and the crawlers feeding model training used to look like one decision. There is also a trap hiding in the same release that can take a site out of Google entirely, so this post covers both, plus the two-minute check I would run on any Cloudflare site this week.
What did Cloudflare actually change?
Cloudflare’s AI Crawl Control has had three levers since July: Search, AI Training and AI Agents. As of 15 September the Training lever has four options: Allow, Disallow AI Training, Block on pages with ads, and Block. The new one in that list, Disallow AI Training, does two different things depending on who the crawler is:
- Mixed-use crawlers stay in. Googlebot, Bingbot and Applebot keep crawling for search. For the companies Cloudflare designates as accountable, the preference is expressed in robots.txt instead of enforced at the edge.
- Training-only crawlers get blocked at the edge. Crawlers from OpenAI, Anthropic, Meta and Amazon that exist for training are stopped outright. Those companies already run separate search and training crawlers, so blocking the training ones costs no search visibility.
The robots.txt part runs through Bot Preference Sync, which prepends rules to your live robots.txt: a Disallow for Google-Extended at Google, and a Disallow for Applebot-Extended at Apple. Your existing lines are preserved below. This is worth saying plainly: Cloudflare did not invent a new signal here. Google-Extended has existed since 2023, and I covered what it does and does not control already. What shipped is a UI that writes it for you, plus real edge enforcement for the crawlers that never respected preferences anyway.
Does blocking AI training hurt Google rankings?
Not through this setting. Google’s own crawler documentation says Google-Extended is not used for Search inclusion or ranking, only to manage whether content helps train and ground Gemini models. Cloudflare’s accountability criteria demand exactly that separation, an opt-out from training that does not affect search results.
Keep two things straight, though, because they are constantly conflated. First, Google-Extended does not keep you out of AI Overviews or AI Mode; that is a separate Search Console setting, and a different decision, one I walked through when Google shipped the control. Second, Google has said nothing publicly about Cloudflare’s feature itself. The pieces are all documented on Google’s side, but the endorsement people are inferring does not exist on the record.
The trap: Block now blocks Google too
Here is the part of the release that deserves the louder headline. Until now, choosing Block on the Training control was a safe-looking maximal option. Since 15 September, Block and Block on pages with ads apply to the mixed-use crawlers as well. Choose Block today and you are refusing Googlebot, Bingbot and Applebot, search crawling included. Cloudflare auto-migrated existing domains that had Block set to the new Disallow AI Training, which was the right call, but anyone who sets Block fresh, assuming it means what it meant in August, is quietly de-indexing their site.
What to check on your site this week
Two minutes, in this order:
- Fetch your own robots.txt in a browser. If Cloudflare is syncing preferences, the rules it wrote sit at the top of the file, above your own. Confirm the
Google-Extendeddisallow is what you intended, and make sure no broad disallow got layered above rules you rely on. If you already hand-wrote Google-Extended and Applebot-Extended lines, decide which copy is the source of truth, because Bot Preference Sync prepends, it does not merge intent. - Open Cloudflare, AI Crawl Control, Training. If it says Disallow AI Training, you have the new behaviour. If it says Block, be certain you mean it, because that now includes search crawlers.
Two more caveats worth carrying. Bing is not wired in yet; Microsoft has not added robots.txt support for this, so today the setting sends Bing nothing, and Bing’s own training opt-out remains the noarchive route, which has its own history. And robots.txt remains a preference, not a wall. The accountable crawlers honor it; the crawlers that never identified themselves ignore it and Cloudflare’s edge block is the only thing that touches them. That is the same reality behind whether robots.txt blocks ChatGPT, and it is why this setting, a preference for the cooperative plus enforcement for the rest, is a genuinely better shape than a text file alone. It is also not llms.txt, which stays a content-discovery proposal with no policy weight.
New domains onboarding after 15 September get presets, and ad-serving sites get Training and AI Agents blocked by default. If you run ads and joined Cloudflare recently, your site may be refusing AI assistants without anyone having chosen that. Whether that is a loss depends on whether AI referrals matter to you, but it should at least be a decision someone made on purpose.
Sources
- Cloudflare blog, stay discoverable in search while disallowing AI training, 15 September 2026, for the four options, accountable-crawler criteria, migration, presets and the under 1% and 17% figures
- Cloudflare blog, Bot Preference Sync, for the robots.txt prepending mechanics
- Google crawler documentation, common crawlers including Google-Extended, for the statement that Google-Extended does not affect Search inclusion or ranking
- Search Console Help, managing AI features in Search, for the separate AI Overviews control
- Search Engine Journal, Cloudflare lets sites disallow AI training, as the secondary report that surfaced the change