AI companies read your site for three different reasons, and robots.txt lets you treat each one differently. Most "block AI bots" advice lumps them together, which can quietly remove you from AI answers you'd want to be in.
Three kinds of AI bots
-
Training crawlers collect pages to train models:
GPTBot(OpenAI),ClaudeBot(Anthropic),Google-Extended(Gemini),Applebot-Extended,meta-externalagent,Amazonbot,CCBot(Common Crawl) andBytespider(ByteDance). -
AI search crawlers index pages so assistants can cite you:
OAI-SearchBot,Claude-SearchBotandPerplexityBot. -
User-triggered fetchers open one page because someone asked about it:
ChatGPT-User,Claude-UserandPerplexity-User.
Google-Extended and Applebot-Extended aren't crawlers at all. They're labels Google and Apple check before using pages their normal bots already fetched for AI training.
The policy most small sites want
Block training, keep AI search:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Disallow:
One trap: a bot that finds a group with its own name ignores your User-agent: * rules. If you hide /admin/ in the * group and then add a group for a named bot, repeat /admin/ there.
Google works differently
Blocking Google-Extended does not remove you from Google Search, AI Overviews or AI Mode. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what AI Overviews show from a page, Google points to nosnippet, max-snippet and noindex. Blocking Googlebot removes you from Google entirely, so never do it by accident.
What robots.txt can't do
It's a request, not a lock. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. Anthropic says all its bots respect it. For scrapers that ignore the file, you need your host's or CDN's bot blocking. And llms.txt blocks nothing.
What big sites do
On November 18, 2026, we checked the robots.txt files of 16 well-known sites. The New York Times, the BBC, Reddit and Amazon block every AI bot we tested. Meta's and ByteDance's crawlers were fully blocked on 12 sites and ClaudeBot on 11, but Google-Extended on only 6. Three Indian news sites, Times of India, Hindustan Times and Mint, fully blocked only Meta's crawler.
If you run a blog, a tool site or a small business, blocking training while keeping AI search is usually the best trade: you stay the source assistants link to, without donating your writing to the next model.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google: Google's common crawlers (Google-Extended)
- Google: AI features and your website
- Google: How Google interprets the robots.txt specification
- Google: Optimizing for generative AI features in Search
- Google: exploring controls for Search generative AI features
- Perplexity: Perplexity crawlers
- Apple: About Applebot
- Meta: Meta web crawlers
- Amazon: Amazonbot and Amazon crawlers
- Common Crawl: CCBot
- IETF: RFC 9309, Robots Exclusion Protocol
- Cloudflare: Block AI bots
- Cloudflare: Declaring your AIndependence (Bytespider)
- Bing: Clearer preferences for training and grounding (Oct 30, 2026)
- llms.txt proposal
This is the short version. The full guide, with every step and source, is on GeekSpark.
Top comments (0)