TL;DR: OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Block the training bot and you can still show up in AI search. Block the search bot and you disappear from that product's answers. User-triggered fetchers may ignore robots.txt entirely.
Last checked: November 2026. Bot names and rules change, so verify against each vendor's docs before you ship.
Why one rule is not enough
A lot of robots.txt advice treats "the OpenAI bot" as one thing. It is three things. OpenAI's documentation lists GPTBot, OAI-SearchBot and ChatGPT-User, and each one has its own job.
| Vendor | Training | Search index | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none documented | PerplexityBot | Perplexity-User |
Anthropic's rows come from third-party references, so confirm them in Anthropic's own docs.
What each bot does to your visibility
GPTBot gathers content for model training. Disallowing it opts you out of training. It does not remove you from ChatGPT search.
OAI-SearchBot is the one that controls search. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. OpenAI also says a robots.txt change can take about 24 hours to take effect for search.
ChatGPT-User fetches pages when a person asks ChatGPT to look at something. OpenAI updated its docs in December 2025 and now says robots.txt rules may not apply to these requests, because a user started them.
PerplexityBot builds the search index behind Perplexity answers. Perplexity says it is not used for training. Perplexity-User fetches pages for a live question and, per Perplexity, generally ignores robots.txt.
Two setups most sites want
Stay visible in AI answers, stay out of training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Allow everything:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Google handles training opt-out with a separate token, Google-Extended. Disallowing it does not change how Googlebot crawls you for Search.
Mistakes I see most
- Blocking GPTBot and assuming OpenAI cannot see the site. OAI-SearchBot and ChatGPT-User are separate tokens.
-
Copying a 2023 list. Vendor names change. Anthropic's older
anthropic-aitoken still shows up in many files, and ClaudeBot is the current training token. -
Forgetting subdomains. Rules at
example.com/robots.txtdo not coverdocs.example.com. Each host needs its own file. - Treating robots.txt as a lock. It is an honor system. Reputable vendors follow it. Scrapers that hide their identity do not, and user-triggered fetchers may not. For those, use rate limits or WAF rules.
How to test your setup
- Open
yourdomain.com/robots.txtand read it as each bot would. Each crawler reads only its own block, so check every named user-agent. - Check your server logs for the tokens above. If you see OAI-SearchBot or PerplexityBot fetching pages, your rules are working in the direction you intended.
- Recheck after any CDN or security plugin change. Several firewall products block unknown bots by default, which can undo a correct robots.txt.
FAQ
Does blocking GPTBot hurt my ChatGPT visibility?
No. GPTBot is the training crawler. Search visibility runs through OAI-SearchBot.
Does blocking Perplexity-User stop Perplexity from reading my page?
Not reliably. Perplexity says this agent generally ignores robots.txt.
How long until a robots.txt change takes effect?
OpenAI says about 24 hours for search. Other vendors do not publish a number I could confirm.
I'm Mansoor Bhanpurawala, an SEO consultant working since 2012. I help sites get cited in ChatGPT, Perplexity and Google AI answers. Details on my GEO consulting page.
Top comments (1)
We need to produce a short YouTube comment, casual, one or two sentences, specific reaction or question about this video. Should be about robots.txt for AI crawlers. Use lowercase start. Avoid formal language. No marketing. No URLs. No double hyphens. No em-dash. Provide a comment like: "i forgot to add the Perplexity bot, any idea where its UA string is?" or "lol who knew you had to block the training bots separately". Must be short. We'll output only the comment