DEV Community

Azeem Rana
Azeem Rana

Posted on Originally published at choruswire.com AI-assisted

How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More

Originally published on ChorusWire.

Each AI company documents separate robots.txt agents for training, search and user-triggered fetches. Blocking one does not block the others, and robots.txt is voluntary.

To communicate that you do not want certain crawlers collecting your site's content for AI training, use the robots.txt controls documented by each provider. These controls differ by company: some block crawling, while others restrict how already-crawled content may be used. They do not guarantee exclusion from every AI training dataset. The names involved include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple) and CCBot (Common Crawl, an open repository of web crawl data). Blocking those does not hide you from AI search or stop fetches a user triggers by hand, because those use different agents. And robots.txt is a request, not a lock: it works only on crawlers that choose to honor it, per the robots.txt standard, RFC 9309. All details below are as of Nov. 10, 2026, from each company's own documentation.

Training, search and user fetches are different agents

The companies split their robots on purpose. OpenAI says each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to signal that its content should not train OpenAI's foundation models. Anthropic makes the same three-way split: ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for fetches made when a person asks Claude a question.

Company Agent Purpose, per the company Robots.txt note, per the company
OpenAI GPTBot Content that may be used for training Disallow signals no training use
OpenAI OAI-SearchBot Surfaces sites in ChatGPT search Opted-out sites will not be shown in search answers
OpenAI ChatGPT-User Fetches triggered by a user Rules may not apply
Anthropic ClaudeBot Collects content that could contribute to training Restricting it excludes future material from training
Anthropic Claude-SearchBot, Claude-User Search quality; user-asked fetches Disabling may reduce visibility
Google Google-Extended Gemini training and grounding No effect on Google Search inclusion
Perplexity PerplexityBot, Perplexity-User Search results; user-asked fetches User fetcher generally ignores rules
Apple Applebot-Extended Opt-out of generative model training Applebot may still crawl for search
Common Crawl CCBot Open repository of web crawl data Disallow stops its crawl

What each one does and does not cover

  • OpenAI. Per OpenAI's bots page, ChatGPT-User is not used to crawl automatically and is not used to decide what appears in Search. Because a user starts those requests, robots.txt rules may not apply. OpenAI lists a fourth agent, OAI-AdsBot, which checks pages submitted as ChatGPT ads. Search opt-outs can take about 24 hours to take effect, OpenAI says.
  • Anthropic. Anthropic says its bots honor robots.txt and that blocking IP addresses may not work reliably, since it stops the bot from reading your robots.txt. It also says it began recognizing a machine-readable "no AI training" preference signal, based on an IETF draft, on Oct. 30, 2026, and that a rule naming ClaudeBot takes priority over the general signal.
  • Google. Google says Google-Extended controls whether crawled content may train future Gemini models and ground answers in Gemini Apps and Vertex AI, and that it has no separate user agent string. The crawling itself uses existing Google agents. Google also says its common crawlers always obey robots.txt for automatic crawls, while user-triggered fetchers generally ignore it.
  • Perplexity. Per Perplexity's crawler page, PerplexityBot is for search and is not used to crawl content for AI foundation models. Perplexity-User supports user questions and generally ignores robots.txt because a user requested the fetch.
  • Apple. Apple says Applebot data may help train its foundation models, and disallowing Applebot-Extended opts out of that. Doing so does not stop Applebot from crawling pages for Spotlight, Siri and Safari. To opt out of use in AI-generated answers, Apple points to the nosnippet meta tag.
  • Common Crawl. Common Crawl is a non-profit that maintains an open repository of web crawl data. It says to add a Disallow rule for CCBot to stop its crawling, and warns that some crawlers falsely identify themselves as CCBot.

A short example

This robots.txt example blocks GPTBot and ClaudeBot, requests that CCBot not crawl the site, and uses Google-Extended to opt out of specified Gemini-related uses. It leaves the separately documented search crawlers unrestricted. Place it in the top-level directory of each domain.

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: CCBot

Disallow: /

Add Applebot-Extended the same way if you want Apple's training opt-out; it is a control token for that opt-out, not a separate crawling process. Anthropic says to repeat the rule for every subdomain you want to opt out. To limit speed instead of blocking, Anthropic supports a non-standard Crawl-delay line; Apple says Applebot does not follow crawl-delay.

What blocking does not do

Robots.txt is voluntary. The standard says crawlers are requested to honor the rules, that they are not a form of access authorization, and that the protocol is not a substitute for real content security. It also warns that listing paths in the file makes them publicly discoverable. For private material, use authentication.

The companies describe opt-outs as forward-looking signals about use in training, not as proof of removal: Anthropic, for example, says restricting ClaudeBot signals that future materials should be excluded. We found no company page promising deletion of anything already collected.

Names and rules change. Google says it intends to support the AIPREF training-preference signal in early 2027, so check each company page before relying on this list.

What we could not verify

We read the pages above but did not test whether any bot obeys your file in practice, and we did not verify how many sites block each agent. Anthropic's page describes its new preference signal but we did not confirm the exact syntax it reads.

Sources


ChorusWire is an independent publisher of sourced music, sports and tech explainers. This article was AI-assisted and human-reviewed. Read more at choruswire.com.

Top comments (0)