robots.txt for AI Crawlers: Which Bots to Allow and Which to Block

Mansoor Bhanpurawala (Digital Mansoor) — Sat, 14 Nov 2026 10:37:24 +0000

TL;DR: OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Block the training bot and you can still show up in AI search. Block the search bot and you disappear from that product's answers. User-triggered fetchers may ignore robots.txt entirely.

Last checked: November 2026. Bot names and rules change, so verify against each vendor's docs before you ship.

Why one rule is not enough

A lot of robots.txt advice treats "the OpenAI bot" as one thing. It is three things. OpenAI's documentation lists GPTBot, OAI-SearchBot and ChatGPT-User, and each one has its own job.

Vendor	Training	Search index	User-triggered fetch
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	none documented	PerplexityBot	Perplexity-User

Anthropic's rows come from third-party references, so confirm them in Anthropic's own docs.

What each bot does to your visibility

GPTBot gathers content for model training. Disallowing it opts you out of training. It does not remove you from ChatGPT search.

OAI-SearchBot is the one that controls search. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. OpenAI also says a robots.txt change can take about 24 hours to take effect for search.

ChatGPT-User fetches pages when a person asks ChatGPT to look at something. OpenAI updated its docs in December 2025 and now says robots.txt rules may not apply to these requests, because a user started them.

PerplexityBot builds the search index behind Perplexity answers. Perplexity says it is not used for training. Perplexity-User fetches pages for a live question and, per Perplexity, generally ignores robots.txt.

Two setups most sites want

Stay visible in AI answers, stay out of training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Allow everything:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Google handles training opt-out with a separate token, Google-Extended. Disallowing it does not change how Googlebot crawls you for Search.

Mistakes I see most

Blocking GPTBot and assuming OpenAI cannot see the site. OAI-SearchBot and ChatGPT-User are separate tokens.
Copying a 2023 list. Vendor names change. Anthropic's older anthropic-ai token still shows up in many files, and ClaudeBot is the current training token.
Forgetting subdomains. Rules at example.com/robots.txt do not cover docs.example.com. Each host needs its own file.
Treating robots.txt as a lock. It is an honor system. Reputable vendors follow it. Scrapers that hide their identity do not, and user-triggered fetchers may not. For those, use rate limits or WAF rules.

How to test your setup

Open yourdomain.com/robots.txt and read it as each bot would. Each crawler reads only its own block, so check every named user-agent.
Check your server logs for the tokens above. If you see OAI-SearchBot or PerplexityBot fetching pages, your rules are working in the direction you intended.
Recheck after any CDN or security plugin change. Several firewall products block unknown bots by default, which can undo a correct robots.txt.

FAQ

Does blocking GPTBot hurt my ChatGPT visibility?
No. GPTBot is the training crawler. Search visibility runs through OAI-SearchBot.

Does blocking Perplexity-User stop Perplexity from reading my page?
Not reliably. Perplexity says this agent generally ignores robots.txt.

How long until a robots.txt change takes effect?
OpenAI says about 24 hours for search. Other vendors do not publish a number I could confirm.

I'm Mansoor Bhanpurawala, an SEO consultant working since 2012. I help sites get cited in ChatGPT, Perplexity and Google AI answers. Details on my GEO consulting page.