The problem
After a forum platform upgrade, all sixteen sitemaps in Search Console read “Couldn’t fetch.” The obvious readings were all wrong. The sitemap files opened fine in a browser. The generator was running on schedule. The homepage and forum index were still being crawled normally, which makes “Googlebot is blocked” look like the wrong answer.
It was the right answer. The CDN in front of the site had a crawler block switched on for Googlebot, BingBot and Applebot, returning 403 to the exact requests Search Console was making. In the 24 hours before the fix, the edge had refused 5,590 Googlebot requests and 16,480 BingBot requests. This was never a sitemap problem; it was the whole site quietly going dark.
What I did
- Split edge from origin by pulling the origin IP from cPanel and requesting the sitemaps directly with a Host header, which returned 200 and valid XML under a Googlebot identity and cleared the server
- Ruled out
.htaccess, ModSecurity, Bot Fight Mode, WAF custom rules, rate limiting and page rules before touching a single setting - Traced the block to a master AI bot policy that was locking the per-crawler controls, and changed only that
- Re-applied scraper blocking individually to 26 AI crawlers so the protection the client wanted stayed in place, with search engines left exempt
- Found the canonical sitemap the upgrade had orphaned, submitted it, and removed 17 stale entries pointing at old static snapshots and individually submitted children
- Confirmed the result in Search Console rather than in the live URL test, which reports as Google’s inspection tool and is not exempt at the edge
The evidence
Search Console, 8 August 2026: sitemap index processed successfully, all seven child sitemaps reading Success, 135,934 URLs discovered.
The grey bars mask the client’s domain. Open the image for full size.
The full client report is published as delivered, with two changes: the site’s domain is masked in the screenshot and in the text, and one account-security instruction meant only for the owner is removed. Every figure, status and date is untouched.