AI crawlers explained: which bots to block, which to allow

Rajhussain Kanani — Wed, 14 Oct 2026 15:08:46 +0000

Originally published on CheckSEO.

The short answer

AI crawlers are the bots AI companies send to read web pages, and they do three different jobs. Training crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot collect content for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI answers can cite and link to them. User-triggered fetchers such as ChatGPT-User and Perplexity-User load a page live because someone asked about it.

Each has its own robots.txt token, so blocking AI isn't all-or-nothing. The common middle ground is to disallow the training bots and allow the search bots, which opts you out of training while keeping you citable in AI answers. Just remember robots.txt is a request, not a lock.

The AI crawler list, grouped by purpose

Every token below comes from the operator's own documentation, checked in September 2026. Two entries aren't crawlers at all: Google-Extended and Applebot-Extended are robots.txt tokens that control how content fetched by Googlebot and Applebot may be used, so they never appear in server logs.

Bot token	Operator	Purpose	Respects robots.txt
`GPTBot`	OpenAI	Collects content that may be used to train OpenAI's foundation models	Yes
`ClaudeBot`	Anthropic	Collects content that may be used to train Claude models	Yes
`CCBot`	Common Crawl	Builds a free public web archive, a common source of AI training data	Yes
`Google-Extended`	Google	Token, not a crawler: controls use in Gemini training and grounding	Yes (a robots.txt control)
`Applebot-Extended`	Apple	Token, not a crawler: controls use in training Apple's models	Yes (a robots.txt control)
`Meta-ExternalAgent`	Meta	Crawls for AI training and indexing for Meta products	Yes
`Bytespider`	ByteDance	Widely listed as ByteDance's AI data crawler	No published policy found
`OAI-SearchBot`	OpenAI	Surfaces sites in ChatGPT search answers	Yes
`Claude-SearchBot`	Anthropic	Indexes content to improve Claude's search results	Yes
`PerplexityBot`	Perplexity	Surfaces and links sites in Perplexity results; not used for training	Yes (disputed)
`Meta-WebIndexer`	Meta	Helps Meta AI cite and link content	Yes
`ChatGPT-User`	OpenAI	Visits a page when a ChatGPT user needs it	OpenAI says robots.txt may not apply
`Claude-User`	Anthropic	Fetches a page when a Claude user asks	Yes
`Perplexity-User`	Perplexity	Fetches a page to answer a user's question	Perplexity says it generally ignores robots.txt
`Meta-ExternalFetcher`	Meta	Fetches links at a user's request, including for AI agents	Meta says it may bypass robots.txt

What blocking each group actually costs

Blocking training crawlers is the low-cost option. OpenAI documents GPTBot and OAI-SearchBot as independent settings, so disallowing GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search. It only works going forward: Anthropic describes a ClaudeBot block as excluding a site's future materials from its training datasets.

Blocking search crawlers takes you out of the answers. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.

Blocking user-triggered fetchers is the least predictable. Anthropic says disallowing Claude-User stops Claude retrieving your pages for user queries, but OpenAI and Perplexity say their fetchers act for a person and may not follow robots.txt.

Google works differently. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. AI Overviews and AI Mode are part of Search, so blocking Google-Extended won't remove you from them.

Copy-paste robots.txt rules

The file goes at the root of every host you want covered, since each subdomain needs its own. Allow about a day for changes to register. One trap: a crawler that finds a group naming it ignores the User-agent: * group entirely, so copy any path rules you rely on, such as Disallow: /admin/, into the named groups too.

Block training, allow AI search (the middle ground most sites want):

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Stay visible in AI search answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Allow everything:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt is a request, not enforcement

Nothing technically stops a bot ignoring robots.txt. If a bot must stay out, enforce it at the edge:

Verify who is really visiting. User agent strings are trivial to fake. OpenAI, Anthropic, Perplexity, Common Crawl and Apple publish their IP addresses, so match both the name and the source before trusting or blocking a request.
Block by behaviour at your CDN or firewall. Cloudflare's AI bot policies block Search, Agent and Training bots separately. Other firewalls can match user agent plus IP range.
Keep robots.txt in place anyway. Anthropic warns that blocking its IP addresses can stop it reading your robots.txt, so an IP block alone may not register as a lasting opt-out.

Check which AI crawlers your site blocks

I built a free AI Crawler Access Checker that fetches a domain's robots.txt and evaluates it for 16 user agents. For each one it reports allowed, blocked or partly restricted, plus the exact rule that decided it. No signup.