Blog AI search
AI search.
AI Overviews, AI Mode, llms.txt, and the crawlers behind ChatGPT, Claude, and Perplexity. 18 posts, newest first.
Block AI training without blocking Google: Cloudflare's new settingCloudflare's Disallow AI Training setting writes Google-Extended into robots.txt while Googlebot keeps crawling. What changed and what to check.
Is Google paying publishers for AI Overviews? The AI contribution pilotGoogle is testing payments for content used in AI answers, managed inside Search Console. What is confirmed, what is not, and what to check now.
Google's Web Search Service API: full web results, but only for partnersGoogle quietly documented a partner-only Web Search Service API returning full web results. What the docs actually say, and who is locked out.
Query fan-out in AI Mode: what Google actually documentsQuery fan-out is how AI Mode answers: it splits a question into subtopics and searches each. What that changes for your pages, and what it does not.
Do AI crawlers render JavaScript? What is documented, and what is notGooglebot renders JavaScript and says so. GPTBot, ClaudeBot and PerplexityBot say nothing at all. Where the evidence actually stops.
Why ChatGPT cannot see my website, and how I check each stepOpenAI runs four crawlers and only one decides whether you appear in ChatGPT search. The check order, and the CDN trap nobody writes about.
Does schema markup help with AI search? What the docs sayNo primary source from Google, OpenAI or Perplexity says schema improves AI citation. What is documented, and what schema does still earn you.
How to track AI Overview traffic in GA4 (and why you cannot)GA4 cannot separate AI Overview visits from ordinary Google clicks, and Google says so in its own docs. What the text fragment trick really does.
Google AI Overviews now auto-expand: what it changes in Search ConsoleGoogle now auto-expands AI Overviews on some queries, pushing blue links down. What it does to your Search Console impressions, and what it does not.
How to keep your content out of AI Overviews and AI ModeA Search Console setting now removes your site from AI Overviews and AI Mode without touching your rankings. Google-Extended never did that job.
Does robots.txt block Claude? ClaudeBot, Claude-User and Claude-SearchBotAnthropic runs three crawlers with three jobs. Blocking ClaudeBot alone leaves two open, and one robots.txt mistake quietly cancels the block.
Does robots.txt block Perplexity? PerplexityBot vs Perplexity-UserPerplexity runs two fetchers with different rules. What blocking PerplexityBot actually does, and why Perplexity-User mostly ignores robots.txt.
Google-Extended in robots.txt: what blocking it doesGoogle-Extended controls Gemini training and grounding, not Search. What blocking the token changes, what it cannot change, and my own choice.
Can you see AI Mode traffic in Search Console?AI Overviews and AI Mode data is in Search Console, mixed into Web search type with no filter. How each is counted, and what you can still work out.
What is WebMCP, and does it affect SEO?ChatGPT's browser now reads tools that websites declare in JavaScript. What WebMCP actually is, and the honest answer on whether it touches search.
llms.txt v2: what changed, and does it help SEO?The llms.txt proposal was revised on 10 August 2026. What v2 actually changes, what it does not do for Google rankings, and the gaps in my own file.
Do I need to optimize for AI Overviews?Google says there are no additional requirements to appear in AI Overviews or AI Mode. What that leaves you with, and what you can actually measure.
Does robots.txt block ChatGPT? Not the way you thinkOpenAI runs four crawlers and they do different jobs. Blocking GPTBot does not remove you from ChatGPT search, and one of them ignores robots.txt.Nothing in that category yet.
Or browse every post, and the evergreen answers in the guides.