Sooner or later someone forwards you a screenshot of the access log and asks whether to block "the AI bots". It is a reasonable question with an unreasonable premise: there is no single population there. There are at least three, they want different things, and blocking them costs you different things.
Here is the map I wish I had the first time I was asked.
Three jobs, not one
Training crawlers collect pages to train a future model. Nothing you do today changes a model that already shipped, and nothing about your search ranking depends on them.
Live-fetch bots fetch one page because a user asked something right now. Block these and you disappear from the answer that was about to cite you.
Answer-engine indexers build an index used to answer questions with citations. Closest thing to search, and the one where blocking most directly costs you referral traffic.
The same company usually runs one of each, under different names. That is the part people miss.
| User-agent | Operator | Job | What blocking it costs you |
|---|---|---|---|
GPTBot |
OpenAI | Training | Nothing in the short term |
OAI-SearchBot |
OpenAI | Search index for ChatGPT | Visibility in ChatGPT search |
ChatGPT-User |
OpenAI | Live fetch on user request | Your page cannot be opened when a user asks about it |
ClaudeBot |
Anthropic | Training | Nothing in the short term |
Claude-Web |
Anthropic | Live fetch | Same as ChatGPT-User
|
PerplexityBot |
Perplexity | Index for cited answers | Citations, and the referral traffic with them |
Google-Extended |
Gemini training only | Nothing in Search — see below | |
CCBot |
Common Crawl | Open dataset many models train on | Nothing directly; wide indirect reach |
Bytespider |
ByteDance | Training | Nothing in the short term |
Applebot-Extended |
Apple | Apple Intelligence training | Nothing in Search — Applebot is separate |
Amazonbot |
Amazon | Assistant and search | Alexa-surface visibility |
meta-externalagent |
Meta | Training | Nothing in the short term |
The one that causes the most damage by accident
Google-Extended is not Googlebot.
It controls whether your content is used to train Gemini. It has no effect on crawling for Search, on indexing, or on ranking. Blocking it does not deindex you and does not cost you a position.
Googlebot is what crawls for Search. If you block that one, you leave the index.
I have seen the two conflated in both directions: people who avoid blocking training because they fear losing rankings they were never going to lose, and — worse — people who block Googlebot believing they are opting out of AI. The names are similar; the consequences are not.
Same pattern at Apple: Applebot crawls for Siri and Spotlight, Applebot-Extended is the training opt-out. Different bots, different consequences.
The trade-off nobody can settle for you
For scale, we crawled the robots.txt of a large sample of sites for a study on this exact question: 17.8% block GPTBot against 2.5% for Googlebot, and 12.3% block PerplexityBot, a crawler that does not train models at all. That last figure is the accidental-damage bucket, measured in the wild.
The argument for blocking training crawlers is simple: your content is an asset, training on it is a use, and you get nothing back.
The argument against is subtler. Assistants that cite sources send traffic, and increasingly they are the search interface. A site that is invisible to them is invisible in a place where people now ask questions. And the distinction between "training" and "answering" is blurrier in practice than the bot names suggest.
The position most people land on, once it is laid out this way:
-
Block training —
GPTBot,ClaudeBot,CCBot,Google-Extended,Bytespider,Applebot-Extended,meta-externalagent. -
Allow live-fetch and answer indexes —
ChatGPT-User,Claude-Web,OAI-SearchBot,PerplexityBot. -
Never touch
Googlebot,Bingbot.
Which comes out as:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
Sitemap: https://example.com/sitemap.xml
That is a position, not a default. A documentation site that wants to be the answer everywhere might block nothing. A publisher whose archive is the product might block everything. Both are coherent; drifting into one by accident is not.
Three things to know before you edit the file
A specific group replaces the wildcard group — it does not extend it. If you write a group for GPTBot, that group is the whole ruleset for GPTBot. Rules under User-agent: * are not inherited. This is the most common way a hand-written file ends up doing the opposite of what was intended.
robots.txt is a request, not a fence. Well-behaved crawlers obey it. Nothing enforces it. If a bot must be stopped rather than asked, that is a job for authentication or blocking at the edge, and the operator list above is the wrong tool.
Test the file before you ship it. The precedence rules are unintuitive: the longest matching rule wins, not the first, and on a tie Allow beats Disallow. A file that reads correctly to a human can behave differently to a parser.
I maintain two small browser-based tools for exactly this: a robots.txt generator with the AI-crawler presets above, and a tester that shows which line decided each verdict. Neither uploads anything — both run on the page.
Top comments (1)
i actually blocked all of these last month without thinking about the tradeoffs, now im kinda scared about the SEO impact