Crawl budget is the most over-discussed and least measured thing in technical SEO. Most sites do not have a crawl budget problem. The ones that do usually cannot tell you which URLs are eating it, because the reports that would tell them sit in two different places and neither one is labelled crawl budget.
There is also a housekeeping detail worth knowing before you read anything else on this. Google moved the guide. It used to live at a URL under search/docs/crawling-indexing with the title “Large site owner’s guide to managing your crawl budget”. It now lives on the crawling docs at developers.google.com/crawling/docs/crawl-budget and it is called “Optimize your crawl budget”. Almost every article you will find still links the old address.
Do you even have a crawl budget problem?
Google is unusually blunt about who this applies to. The guide names three groups:
“Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”
“Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”
“Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed”
And then, immediately after: “The numbers given here are a rough estimate. These are not exact thresholds.”
The third group is the one nobody quotes and the one that catches real sites. It has no size attached to it. A four hundred page store with sixty thousand parameter URLs is in that group, and by page count it looks tiny.
The Page indexing documentation says the same thing from the other side: “If your site has fewer than 500 pages, you probably don’t need to use this report.” Under that size, crawl budget is almost never your problem.
The advice that surprises everyone
Straight from the guide, in a section about reducing crawl waste:
“Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.”
That contradicts most crawl budget advice on the internet. Noindex is an indexing instruction, not a crawling one. Googlebot has to fetch the page to read the tag, so every noindexed page costs one full fetch, every single recrawl, forever, and returns nothing.
Noindex is still the correct tool for getting a page out of the index. It is simply the wrong tool for saving crawl.
One more line from the same guide that changes how you think about the ceiling: “While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers.” Googlebot for search, the image crawler, the video crawler and the rest are all drawing from one bucket.
What each Search Console status actually costs you
This is the table I build for clients, because “fix your crawl budget” is not an instruction anyone can act on. Cost here means fetches spent per recrawl cycle.
Fetch cost by status. Source: Google's Optimize your crawl budget guide and the Page indexing report documentation.
The two zero rows are the ones people misread. Blocked by robots.txt is genuinely free, which is why it is the right lever for parameter noise Google has never indexed. Discovered - currently not indexed is also free, and yet it is the status Google names as a symptom of a crawl budget problem. Those URLs are not consuming anything. They are queued behind the URLs that are.
Soft 404 is the one I would fix first on almost any site. It answers with a 200, so nothing ever ages it out of the crawl schedule. A genuine 404 stops getting requested. A soft 404 keeps getting fetched indefinitely.
For redirects, note that each hop is a separate request in your logs and in Crawl Stats. Google types them under “Other file type”, which is why redirect chains hide so well. My write-up of Page with redirect in Search Console covers the chain audit.
Reading Crawl Stats without fooling yourself
Crawl Stats sits in Settings, not in the main navigation, and there is one restriction that catches people out. The documentation states: “This report is available only for root-level properties.” If your property is a URL prefix for a subfolder, this report does not exist for you. Add a domain property.
Inside the report, the split that matters is By purpose: Discovery versus Refresh. Discovery is Google fetching URLs it has not seen before. Refresh is Google re-checking pages it already knows.
A healthy site running a content programme shows real discovery traffic. A site drowning in parameter URLs shows discovery pinned high while the indexed count sits flat, which means the budget is being spent finding junk. A site Google has lost interest in shows discovery near zero with refresh continuing on a small set of pages.
Watch the average response time on the same screen. Crawl capacity moves with how fast your server answers. If response times climb, crawl rate falls, and no amount of sitemap tidying will get it back. If your Crawl Stats numbers look wrong or absent, I have written separately about missing data in the Crawl Stats report.
The order I actually work in
Fix soft 404s first. They are permanent waste, they are usually a small list, and the fix is a real 404 or 410.
Flatten redirect chains second. Every hop is a fetch and chains grow silently through site migrations.
Block parameter URLs in robots.txt third, but only patterns Google has never indexed. If a URL is already in the index, blocking it freezes it there, because Google can no longer fetch the page to learn it should go. The full argument is in my post on noindex versus robots.txt.
Leave the noindexed pages alone unless the count is large. Yes, they cost a fetch each. They also cost you nothing to keep, and swapping them to a robots.txt block is exactly the mistake in the previous paragraph.
If you want the full list of what every status means before you decide what to fix, the Search Console indexing status guide covers all of them.
Sources
- Optimize your crawl budget, Google crawling documentation.
- Crawl Stats report, Google Search Console Help.
- Page Indexing report, Google Search Console Help.