Does robots.txt block Perplexity? For half of it. Perplexity documents two user agents, and only one of them is a crawler in the robots.txt sense. PerplexityBot builds the search index that decides whether your site can be surfaced and linked in Perplexity’s results, and it is the one a robots.txt rule reaches. Perplexity-User fetches a page at the moment a user asks a question that needs it, and Perplexity’s own documentation says it “generally ignores robots.txt rules” because a human requested the fetch. If you wrote a Disallow line expecting it to end all Perplexity traffic in your logs, that is why it did not.
This is the same two-sided misunderstanding I unpacked for OpenAI’s four crawlers in does robots.txt block ChatGPT, and it fails in the same two directions here. People who block PerplexityBot to keep their content away from AI mostly remove their own citations, and people who wanted a total exit find the user-triggered fetches still arriving.
PerplexityBot indexes, Perplexity-User fetches
Straight from Perplexity’s crawler documentation, which is short enough to read in full:
PerplexityBot “is designed to surface and link websites in search results on Perplexity.” The same paragraph carries the sentence most people do not expect: “It is not used to crawl content for AI foundation models.” Perplexity’s recommendation is to allow it if you want to appear in results, and it publishes its source IP addresses so the user agent string can be verified rather than trusted.
Perplexity-User “supports user actions within Perplexity.” When someone asks a question, it may visit a page to help answer and link it. Also not used for training, per the same document. And the load-bearing line: since a user requested the fetch, it generally ignores robots.txt. That makes it a user agent in the original sense, closer to a browser acting for a person than to a crawler filling an index, the same category OpenAI’s ChatGPT-User occupies.
Both publish their IP ranges as JSON, and I pulled the two files while writing this on 31 August: perplexitybot.json currently lists 8 address ranges and perplexity-user.json lists 4. Those files are the answer to a question robots.txt cannot settle, whether a request claiming to be Perplexity really is, which matters because scrapers borrow well-known user agent strings precisely to inherit their welcome.
So what does blocking PerplexityBot get you?
Removal from Perplexity’s search surface over time: no longer indexed, so no longer surfaced and linked as a result. What it does not get you, on Perplexity’s own description, is a training opt-out, because the bot is not collecting for model training in the first place. If your reason for blocking was “I do not want to train their models”, the block is aimed at the wrong mechanism, and if your reason was “I do not want Perplexity citing me”, note what you are giving up: the citation is a link, and a user-triggered Perplexity-User visit can still read a public page when someone asks about it directly.
There is also a verification wrinkle worth stating plainly: robots.txt is a request, and compliance is claimed by the company, not enforced by the protocol. Perplexity has faced public accusations of fetching content beyond what its documentation describes, which it disputes. I am not adjudicating that here; I am noting that the published IP files are the tool the documentation gives you for holding the claim testable in your own logs, and a firewall rule keyed to verified ranges is enforcement in a way a Disallow line never is.
What my robots.txt does about it
Nothing, deliberately. This site’s robots.txt allows every crawler and disallows one private path, the same policy I walked through for Google-Extended: my pages exist to be found, an AI answer that cites me is distribution, and the operator asking “should I block AI bots” almost always means “will blocking cost me visibility”, to which the honest answer here is that blocking PerplexityBot costs exactly the visibility and prevents none of the training people worry about, because the training is not what this bot does. A site with a different economics, paywalled content, licensing leverage, should reason differently, and for that site the order of operations is: robots.txt for the compliant crawler, verified-IP firewall rules for enforcement, and no expectation that either fully stops user-triggered fetches of public URLs.
The same split shows up at every AI company, so the same reasoning applies each time you meet a new one. Does robots.txt block Claude covers Anthropic, which runs three bots rather than two.
If Perplexity can fetch you and still never cites you, the next thing to check is whether your answers are liftable. That list is in how to improve brand visibility in AI search engines.
Sources
- Perplexity, crawlers documentation, both user agent descriptions and the robots.txt behaviour, checked 31 August 2026
- Perplexity, published IP ranges: perplexitybot.json and perplexity-user.json, counted 31 August 2026
- My own robots.txt policy on shahidali.co, checked 31 August 2026