Blog AI search
AI search.
AI Overviews, AI Mode, llms.txt, and the crawlers behind ChatGPT, Claude, and Perplexity. 13 posts, newest first.
Why ChatGPT cannot see my website, and how I check each stepOpenAI runs four crawlers and only one decides whether you appear in ChatGPT search. The check order, and the CDN trap nobody writes about.
Does schema markup help with AI search? What the docs sayNo primary source from Google, OpenAI or Perplexity says schema improves AI citation. What is documented, and what schema does still earn you.
How to track AI Overview traffic in GA4 (and why you cannot)GA4 cannot separate AI Overview visits from ordinary Google clicks, and Google says so in its own docs. What the text fragment trick really does.
Google AI Overviews now auto-expand: what it changes in Search ConsoleGoogle now auto-expands AI Overviews on some queries, pushing blue links down. What it does to your Search Console impressions, and what it does not.
How to keep your content out of AI Overviews and AI ModeA Search Console setting now removes your site from AI Overviews and AI Mode without touching your rankings. Google-Extended never did that job.
Does robots.txt block Claude? ClaudeBot, Claude-User and Claude-SearchBotAnthropic runs three crawlers with three jobs. Blocking ClaudeBot alone leaves two open, and one robots.txt mistake quietly cancels the block.
Does robots.txt block Perplexity? PerplexityBot vs Perplexity-UserPerplexity runs two fetchers with different rules. What blocking PerplexityBot actually does, and why Perplexity-User mostly ignores robots.txt.
Google-Extended in robots.txt: what blocking it doesGoogle-Extended controls Gemini training and grounding, not Search. What blocking the token changes, what it cannot change, and my own choice.
Can you see AI Mode traffic in Search Console?AI Overviews and AI Mode data is in Search Console, mixed into Web search type with no filter. How each is counted, and what you can still work out.
What is WebMCP, and does it affect SEO?ChatGPT's browser now reads tools that websites declare in JavaScript. What WebMCP actually is, and the honest answer on whether it touches search.
llms.txt v2: what changed, and does it help SEO?The llms.txt proposal was revised on 10 August 2026. What v2 actually changes, what it does not do for Google rankings, and the gaps in my own file.
Do I need to optimize for AI Overviews?Google says there are no additional requirements to appear in AI Overviews or AI Mode. What that leaves you with, and what you can actually measure.
Does robots.txt block ChatGPT? Not the way you thinkOpenAI runs four crawlers and they do different jobs. Blocking GPTBot does not remove you from ChatGPT search, and one of them ignores robots.txt.Nothing in that category yet.
Or browse every post, and the evergreen answers in the guides.