X-Robots-Tag is the HTTP header version of the robots meta tag, and it exists for one reason: a PDF, an image or a CSV has no <head> to put a meta tag in. Any rule the meta tag supports, noindex included, can be sent as a response header instead, which makes it the only page-level indexing control that works on every file type you serve. Google documents both in the same specification, and the header half is the one most sites have never touched.
I checked this site while writing this post: no URL on shahidali.co sends an X-Robots-Tag header, which is the correct configuration here and still worth having verified rather than assumed. More on that below, because the check is one command.
What the header looks like
The rule rides in the HTTP response, before any HTML:
HTTP/1.1 200 OK
Date: Sun, 31 Aug 2026 12:00:00 GMT
X-Robots-Tag: noindexGoogle’s specification adds three details that matter in practice. Multiple headers can be combined, or rules comma-separated in one header. A user agent can be named before the rules, so X-Robots-Tag: googlebot: noindex binds only Googlebot, and rules without a user agent bind all crawlers. And when rules conflict, the most restrictive one wins, exactly as with stacked meta tags. Header name, user agent and values are all case insensitive.
The server configuration is a few lines. Google’s own examples: Apache sets it with Header set X-Robots-Tag "noindex, nofollow" inside a <Files ~ "\.pdf$"> block, nginx with add_header X-Robots-Tag "noindex, nofollow"; in a location matching \.pdf$. On Vercel, where this site runs, the same thing is a headers entry in vercel.json, the mechanism I already use to set a content security policy, so adding X-Robots-Tag to a path pattern is one more object in an existing array.
Why robots.txt cannot do this job
This is the confusion that keeps a generation of PDFs in Google. Disallowing /downloads/ in robots.txt stops the crawling, not the indexing. A URL Google cannot fetch can still be indexed from the links pointing at it, which is precisely the “indexed, though blocked by robots.txt” status in Search Console. Worse, the two mechanisms cancel each other: Google’s specification says robots rules can only be read and followed if crawlers are allowed to access the page. Block the file in robots.txt and the noindex header on it is never seen. Crawl access is the delivery mechanism for the removal instruction.
So the working combination for “keep this file out of results” is the counterintuitive one: allow the crawl, send X-Robots-Tag: noindex, and let Googlebot fetch its own eviction notice. For HTML pages the same logic applies to the meta tag, a split I walked through in noindex vs robots.txt; the header simply extends it to everything that is not HTML.
How to check what your site sends
One request per file type is enough, and you do not need anything beyond curl:
curl -sI https://example.com/whitepaper.pdf | grep -i x-robots-tagNo output means no header, which is fine if you want the file indexed and a gap if you do not. Running that against this site is how I verified nothing here sends the header. That is deliberate: the one section I keep out of Google, /reports/, is blocked in robots.txt instead, a choice with a documented trade-off I wrote up in the indexed-though-blocked post, because for client-facing report URLs I care about stopping the crawl itself, not just the listing. If my priority ever flips to guaranteed absence from results, the migration path is exactly this post: drop the disallow, add X-Robots-Tag: noindex on that path in vercel.json, and let the URLs get crawled out of the index.
Two closing cautions from Google’s specification. The header, like the meta tag, is honoured at the next crawl, not instantly; for urgent removals the Removals tool is the fast lane. And check what your stack already sends before adding anything: plugins, CDNs and app frameworks sometimes attach an X-Robots-Tag you did not write, and a stray noindex header on real content is invisible in the HTML, which is exactly why it belongs in a technical audit’s header pass rather than a view-source skim. When that stray header bites, Search Console files the page under URL marked ‘noindex’, and the header is the last of the four places anyone looks.
The same curl check settles questions about the other directives you can send this way. It is how I confirmed what noarchive still does in 2026, which turns out to be nothing at Google and quite a lot at Bing.
Sources
- Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, header syntax, server examples, and the crawl access requirement, checked 31 August 2026
- Google Search Central, Block Search indexing with noindex, on the two noindex forms and the robots.txt interaction
- My own header checks on shahidali.co, 31 August 2026