Google is not indexing all my products is one of the most common complaints I audit, and on stores with thousands of SKUs the reflex explanation is always crawl budget. Sometimes it is. More often the budget is being spent, just not on products, and the fix is subtraction rather than submission. I have run one of these campaigns end to end, 8,500 URLs on a single site brought to 91% indexed, so this post is the sequence I actually used, not a checklist of everything that could theoretically help.
Is crawl budget even your problem?
Google’s crawl budget documentation is unusually specific about who should care: sites with over a million pages that change weekly, sites with more than 10,000 pages that change daily, or sites where Search Console flags a large share of URLs as Discovered, currently not indexed. A 10,000-SKU store does not hit the first two bars, but catalogs churn, and the third condition is the one that catches them: if your Page indexing report shows a fat slice of Discovered, currently not indexed, Google is declining to spend crawl on you, and budget thinking applies no matter your size.
One myth to kill early: there is no per-site cap on how many pages Google will index. Mueller has said so directly. The limit you are hitting is willingness, not capacity.
The order matters: what I removed first
The campaign logic is simple to state: before you ask Google to crawl more of what you want, stop it crawling what you do not. The removal order I used, first to last:
- Internal search and sort parameters. Search result URLs, sort orders, per-page settings. Pure crawl waste, blocked in robots.txt. Not noindex, and the reason is the least understood rule in this area: noindex does not save crawl budget. Google has to crawl a URL to see a noindex tag, so the crawl is already spent. Robots.txt stops the fetch itself. Google’s faceted navigation guidance makes exactly this distinction.
- Unvalidated facet combinations. Colour times size times price band multiplies into millions of URLs no one searches for. I covered which facets deserve to live in faceted navigation SEO, so the short version: a facet page earns indexing when there is search demand for the combination, and earns crawling only if it earns indexing.
- Variant and near-duplicate SKUs. Canonicals to the parent product, so the crawl that does happen consolidates instead of fragmenting.
- Thin and stale pages. Discontinued products, empty categories and tag archives, the stuff index bloat is made of.
Only after those four cuts did submission start, sectioned sitemaps first, priority pages through URL Inspection last. Per-URL submission cannot carry a catalog: the daily allowance is a rounding error against 10,000 SKUs, which is the arithmetic in how many URLs you can request indexing per day.
Read Crawl Stats while you do it
The feedback loop for all of this is the Crawl Stats report, and I watched two numbers through the campaign. Total crawl requests: as the robots.txt cuts landed, requests did not drop, they moved, away from parameter URLs and into products and categories, which is the entire point. Average response time: this is the quiet cap on everything, because a slow store gets crawled gently no matter how clean its architecture is. Google’s crawl budget doc lists server health as a direct input to the crawl rate. How to read the Crawl Stats report covers where each number lives.
Expect lag. A robots.txt change is picked up quickly, but the index numbers move on Google’s schedule, not yours: I saw weeks, not days, between a cut and the corresponding shift in the Page indexing report, which matches the timelines in how long Google takes to index a new page.
What I would tell a 10,000-SKU store owner
Do not start by submitting anything. Pull the Page indexing report, split the not-indexed URLs by pattern, and count how much of your crawl is going to URLs you would never want ranked. On every large store I have audited, that number, not Google’s capacity, was the reason the products were waiting.
Sources
- Google, optimize your crawl budget, for the three thresholds and server health as a crawl-rate input, checked 14 September 2026. Note the guide moved from its old search/docs address, which most articles still link
- Google Search Central, managing crawling of faceted navigation URLs, for robots.txt versus noindex on parameter URLs
- Google Search Console Help, Crawl Stats report and Page indexing report
- Search Engine Journal, Google: no limit to how many pages of a site we index, for Mueller on there being no per-site index cap
- My own project ledger for the 8,500-URL campaign and its 91% indexed outcome, and audit work on WooCommerce and Shopify stores for the patterns described