<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Azeem Rana</title>
    <description>The latest articles on DEV Community by Azeem Rana (@choruswire).</description>
    <link>https://dev.to/choruswire</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4243613%2Fd24415bc-8563-4553-8d0c-d3efe2db92b8.png</url>
      <title>DEV Community: Azeem Rana</title>
      <link>https://dev.to/choruswire</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/choruswire"/>
    <language>en</language>
    <item>
      <title>How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More</title>
      <dc:creator>Azeem Rana</dc:creator>
      <pubDate>Sat, 14 Nov 2026 14:32:22 +0000</pubDate>
      <link>https://dev.to/choruswire/how-to-block-ai-crawlers-with-robotstxt-gptbot-claudebot-google-extended-and-more-92c</link>
      <guid>https://dev.to/choruswire/how-to-block-ai-crawlers-with-robotstxt-gptbot-claudebot-google-extended-and-more-92c</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://choruswire.com/tech/it/block-ai-crawlers-robots-txt-gptbot-claudebot-google-extended/" rel="noopener noreferrer"&gt;ChorusWire&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Each AI company documents separate robots.txt agents for training, search and user-triggered fetches. Blocking one does not block the others, and robots.txt is voluntary.&lt;/p&gt;

&lt;p&gt;To communicate that you do not want certain crawlers collecting your site's content for AI training, use the robots.txt controls documented by each provider. These controls differ by company: some block crawling, while others restrict how already-crawled content may be used. They do not guarantee exclusion from every AI training dataset. The names involved include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple) and CCBot (Common Crawl, an open repository of web crawl data). Blocking those does not hide you from AI search or stop fetches a user triggers by hand, because those use different agents. And robots.txt is a request, not a lock: it works only on crawlers that choose to honor it, per the &lt;a href="https://www.rfc-editor.org/rfc/rfc9309.html" rel="noopener noreferrer"&gt;robots.txt standard, RFC 9309&lt;/a&gt;. All details below are as of Nov. 10, 2026, from each company's own documentation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Training, search and user fetches are different agents
&lt;/h2&gt;

&lt;p&gt;The companies split their robots on purpose. &lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;OpenAI says&lt;/a&gt; each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to signal that its content should not train OpenAI's foundation models. &lt;a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler" rel="noopener noreferrer"&gt;Anthropic&lt;/a&gt; makes the same three-way split: ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for fetches made when a person asks Claude a question.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;Agent&lt;/th&gt;
&lt;th&gt;Purpose, per the company&lt;/th&gt;
&lt;th&gt;Robots.txt note, per the company&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;GPTBot&lt;/td&gt;
&lt;td&gt;Content that may be used for training&lt;/td&gt;
&lt;td&gt;Disallow signals no training use&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;OAI-SearchBot&lt;/td&gt;
&lt;td&gt;Surfaces sites in ChatGPT search&lt;/td&gt;
&lt;td&gt;Opted-out sites will not be shown in search answers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;ChatGPT-User&lt;/td&gt;
&lt;td&gt;Fetches triggered by a user&lt;/td&gt;
&lt;td&gt;Rules may not apply&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;ClaudeBot&lt;/td&gt;
&lt;td&gt;Collects content that could contribute to training&lt;/td&gt;
&lt;td&gt;Restricting it excludes future material from training&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;Claude-SearchBot, Claude-User&lt;/td&gt;
&lt;td&gt;Search quality; user-asked fetches&lt;/td&gt;
&lt;td&gt;Disabling may reduce visibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google&lt;/td&gt;
&lt;td&gt;Google-Extended&lt;/td&gt;
&lt;td&gt;Gemini training and grounding&lt;/td&gt;
&lt;td&gt;No effect on Google Search inclusion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Perplexity&lt;/td&gt;
&lt;td&gt;PerplexityBot, Perplexity-User&lt;/td&gt;
&lt;td&gt;Search results; user-asked fetches&lt;/td&gt;
&lt;td&gt;User fetcher generally ignores rules&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apple&lt;/td&gt;
&lt;td&gt;Applebot-Extended&lt;/td&gt;
&lt;td&gt;Opt-out of generative model training&lt;/td&gt;
&lt;td&gt;Applebot may still crawl for search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Common Crawl&lt;/td&gt;
&lt;td&gt;CCBot&lt;/td&gt;
&lt;td&gt;Open repository of web crawl data&lt;/td&gt;
&lt;td&gt;Disallow stops its crawl&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What each one does and does not cover
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;OpenAI.&lt;/strong&gt; Per &lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;OpenAI's bots page&lt;/a&gt;, ChatGPT-User is not used to crawl automatically and is not used to decide what appears in Search. Because a user starts those requests, robots.txt rules may not apply. OpenAI lists a fourth agent, OAI-AdsBot, which checks pages submitted as ChatGPT ads. Search opt-outs can take about 24 hours to take effect, OpenAI says.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anthropic.&lt;/strong&gt; &lt;a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler" rel="noopener noreferrer"&gt;Anthropic says&lt;/a&gt; its bots honor robots.txt and that blocking IP addresses may not work reliably, since it stops the bot from reading your robots.txt. It also says it began recognizing a machine-readable "no AI training" preference signal, based on an IETF draft, on Oct. 30, 2026, and that a rule naming ClaudeBot takes priority over the general signal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google.&lt;/strong&gt; &lt;a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers" rel="noopener noreferrer"&gt;Google says&lt;/a&gt; Google-Extended controls whether crawled content may train future Gemini models and ground answers in Gemini Apps and Vertex AI, and that it has no separate user agent string. The crawling itself uses existing Google agents. Google also says its common crawlers always obey robots.txt for automatic crawls, while &lt;a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers" rel="noopener noreferrer"&gt;user-triggered fetchers&lt;/a&gt; generally ignore it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Perplexity.&lt;/strong&gt; Per &lt;a href="https://docs.perplexity.ai/guides/bots" rel="noopener noreferrer"&gt;Perplexity's crawler page&lt;/a&gt;, PerplexityBot is for search and is not used to crawl content for AI foundation models. Perplexity-User supports user questions and generally ignores robots.txt because a user requested the fetch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Apple.&lt;/strong&gt; &lt;a href="https://support.apple.com/en-us/119829" rel="noopener noreferrer"&gt;Apple says&lt;/a&gt; Applebot data may help train its foundation models, and disallowing Applebot-Extended opts out of that. Doing so does not stop Applebot from crawling pages for Spotlight, Siri and Safari. To opt out of use in AI-generated answers, Apple points to the nosnippet meta tag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Common Crawl.&lt;/strong&gt; &lt;a href="https://commoncrawl.org/ccbot" rel="noopener noreferrer"&gt;Common Crawl&lt;/a&gt; is a non-profit that maintains an open repository of web crawl data. It says to add a Disallow rule for CCBot to stop its crawling, and warns that some crawlers falsely identify themselves as CCBot.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A short example
&lt;/h2&gt;

&lt;p&gt;This robots.txt example blocks GPTBot and ClaudeBot, requests that CCBot not crawl the site, and uses Google-Extended to opt out of specified Gemini-related uses. It leaves the separately documented search crawlers unrestricted. Place it in the top-level directory of each domain.&lt;/p&gt;

&lt;p&gt;User-agent: GPTBot&lt;/p&gt;

&lt;p&gt;Disallow: /&lt;/p&gt;

&lt;p&gt;User-agent: ClaudeBot&lt;/p&gt;

&lt;p&gt;Disallow: /&lt;/p&gt;

&lt;p&gt;User-agent: Google-Extended&lt;/p&gt;

&lt;p&gt;Disallow: /&lt;/p&gt;

&lt;p&gt;User-agent: CCBot&lt;/p&gt;

&lt;p&gt;Disallow: /&lt;/p&gt;

&lt;p&gt;Add Applebot-Extended the same way if you want Apple's training opt-out; it is a control token for that opt-out, not a separate crawling process. Anthropic says to repeat the rule for every subdomain you want to opt out. To limit speed instead of blocking, Anthropic supports a non-standard Crawl-delay line; Apple says Applebot does not follow crawl-delay.&lt;/p&gt;

&lt;h2&gt;
  
  
  What blocking does not do
&lt;/h2&gt;

&lt;p&gt;Robots.txt is voluntary. The &lt;a href="https://www.rfc-editor.org/rfc/rfc9309.html" rel="noopener noreferrer"&gt;standard&lt;/a&gt; says crawlers are requested to honor the rules, that they are not a form of access authorization, and that the protocol is not a substitute for real content security. It also warns that listing paths in the file makes them publicly discoverable. For private material, use authentication.&lt;/p&gt;

&lt;p&gt;The companies describe opt-outs as forward-looking signals about use in training, not as proof of removal: Anthropic, for example, says restricting ClaudeBot signals that future materials should be excluded. We found no company page promising deletion of anything already collected.&lt;/p&gt;

&lt;p&gt;Names and rules change. Google says it intends to support the AIPREF training-preference signal in early 2027, so check each company page before relying on this list.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not verify
&lt;/h2&gt;

&lt;p&gt;We read the pages above but did not test whether any bot obeys your file in practice, and we did not verify how many sites block each agent. Anthropic's page describes its new preference signal but we did not confirm the exact syntax it reads.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;OpenAI: Overview of OpenAI Crawlers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler" rel="noopener noreferrer"&gt;Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers" rel="noopener noreferrer"&gt;Google: List of Google's common crawlers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers" rel="noopener noreferrer"&gt;Google: List of Google user-triggered fetchers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.perplexity.ai/guides/bots" rel="noopener noreferrer"&gt;Perplexity: Perplexity Crawlers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://commoncrawl.org/ccbot" rel="noopener noreferrer"&gt;Common Crawl: CCBot&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://support.apple.com/en-us/119829" rel="noopener noreferrer"&gt;Apple: About Applebot&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9309.html" rel="noopener noreferrer"&gt;IETF: RFC 9309, Robots Exclusion Protocol&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;ChorusWire is an independent publisher of sourced music, sports and tech explainers. This article was AI-assisted and human-reviewed. Read more at &lt;a href="https://choruswire.com" rel="noopener noreferrer"&gt;choruswire.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>seo</category>
      <category>security</category>
    </item>
  </channel>
</rss>
