Noindex vs robots.txt is the mix-up I fix most often on other people’s sites. Both look like ways to keep a page out of Google, they work through completely different mechanisms, and the most popular way of using them, both at once, is the one combination that fails.
The short version: noindex removes a page from Google’s index. Robots.txt only stops Google from crawling it, and a URL Google cannot crawl can still be indexed. If your goal is “not in Google”, the tool is noindex, and the page must stay crawlable so Google can see it.
What robots.txt actually does
Robots.txt is a crawl control. A Disallow line tells Googlebot not to fetch matching URLs, and that is the entire contract. Google’s robots.txt introduction says it outright: it is not a mechanism for keeping a page out of Google.
Here is the part that surprises people. If other pages link to a URL blocked by robots.txt, Google can index the URL anyway, from the links alone, without ever fetching the page. You get the ugly version of a result: the bare URL or anchor text as the title, no description. Search Console reports these as “Indexed, though blocked by robots.txt”, and I see that status on nearly every site whose owner tried to remove pages with robots.txt. The block did not remove anything. It froze the page in the index in its worst possible form, because Google can no longer read what is there. I have pulled that status apart on its own in indexed, though blocked by robots.txt, including the cases where leaving it alone is the right call.
What robots.txt is for: keeping crawlers out of infinite spaces and junk fetches. Faceted filter combinations, internal search results, cart and session URLs, endless calendars. Crawl management, not index management. It is also worth keeping robots.txt itself reachable while you do that, because a file Google cannot fetch stops crawling across the whole host: see failed: robots.txt unreachable for what Google does in that case.
What noindex actually does
Noindex is an index control. A <meta name="robots" content="noindex"> tag in the head, or an X-Robots-Tag: noindex HTTP header on files with no head to put one in, tells Google: you may crawl this, do not keep it in results. Google’s blocking indexing documentation confirms the effect: when Googlebot crawls the page and sees the tag, the page is dropped from results, whoever links to it.
The catch is in that sentence: when Googlebot crawls the page. Noindex only works if Google can fetch the page and read the tag. Which produces the classic failure:
Can I use noindex and robots.txt together?
On the same URL, for removal: no, and this is the mistake that keeps the combination famous. Block a page in robots.txt and add noindex to it, and the block wins in the worst way: Googlebot is forbidden from fetching the page, so it never sees the noindex, so the URL stays indexable from links. Google’s documentation carries this exact warning: for noindex to work, the page must not be blocked by robots.txt. The belt and the braces cancel each other.
The order matters even when you clean up. If a blocked URL is already indexed, unblock it first, let Google recrawl and read the noindex, and only consider re-blocking after it is gone. And do not bother putting a noindex rule inside robots.txt itself: Google ended support for that in 2019, effective 1 September 2019. Lines like Noindex: in robots.txt do nothing now.
Which one for which job
- Page must disappear from Google: noindex, page crawlable. For urgency, pair it with the Removals tool in Search Console, which hides the page fast while the noindex makes it permanent. Success looks like the page landing in the URL marked ‘noindex’ status, which is the report confirming it read your directive.
- Crawler wasting budget on parameter or filter URLs: robots.txt, and accept the ghost-URL risk on anything well linked.
- Private or sensitive content: neither. Robots.txt is public and readable by anyone, and noindex still lets the page be fetched. Authentication is the only real control there.
- Page should never have existed: delete it and return 404 or 410. Nothing removes a URL from the index more definitively than the page ceasing to exist.
If you are staring at unexpected statuses while you sort this out, I keep a guide to every indexing status Search Console reports and what each one actually means. And to check what Google currently holds for a batch of URLs before and after the fix, the bulk URL inspection tool on this site reads per-URL status straight from the API, which beats eyeballing the Pages report by hand.
If you are on WordPress, there is one setting that has quietly switched sides between these two mechanisms, and it is worth reading before you go hunting anywhere else: discourage search engines from indexing this site. The other WordPress case where people reach for the wrong one of these two is archives, which I sorted out in should you noindex category and tag pages. And before you edit anything at all, it is worth knowing that the WordPress robots.txt you are about to change is not a file: WordPress builds it per request, which is why the plugin panel and the served output can disagree.
The clearest live example of this I write about is WooCommerce. Cart and checkout pages are already noindexed by core, and blocking them in robots.txt is what keeps them in the results. Why WooCommerce cart and checkout pages show up in Google has the full fix order.
Sources
- Google Search Central, introduction to robots.txt
- Google Search Central, block search indexing with noindex
- Google Search Central blog, a note on unsupported rules in robots.txt, July 2019