<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: SigzenAI</title>
    <description>The latest articles on DEV Community by SigzenAI (@sigzenai).</description>
    <link>https://dev.to/sigzenai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247252%2F6f1a1bf4-b46e-4823-8013-2f0632078aa7.png</url>
      <title>DEV Community: SigzenAI</title>
      <link>https://dev.to/sigzenai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sigzenai"/>
    <language>en</language>
    <item>
      <title>Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers</title>
      <dc:creator>SigzenAI</dc:creator>
      <pubDate>Mon, 16 Nov 2026 08:49:34 +0000</pubDate>
      <link>https://dev.to/sigzenai/your-robotstxt-is-probably-blocking-the-wrong-ai-bots-heres-how-to-sort-training-search-and-d51</link>
      <guid>https://dev.to/sigzenai/your-robotstxt-is-probably-blocking-the-wrong-ai-bots-heres-how-to-sort-training-search-and-d51</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why is this suddenly a developer problem?
&lt;/h2&gt;

&lt;p&gt;For years, &lt;code&gt;robots.txt&lt;/code&gt; was a set-and-forget file. Then AI crawlers arrived, and someone on the marketing or legal side asked engineering to "block the AI bots".&lt;/p&gt;

&lt;p&gt;The traffic numbers explain the pressure. Cloudflare's &lt;a href="https://blog.cloudflare.com/agentic-internet-bot-report/" rel="noopener noreferrer"&gt;July 2026 crawler report&lt;/a&gt; found that &lt;strong&gt;52% of crawler requests on its network were for AI training in June 2026&lt;/strong&gt;, up from 22% in spring 2025. More than half of all web traffic is now non-human.&lt;/p&gt;

&lt;p&gt;(Note that Cloudflare sells bot-management products and argues for blocking by default, and the figures cover only its own network. They are still the best public view we have.)&lt;/p&gt;

&lt;p&gt;The catch: the quick fix, a blanket &lt;code&gt;Disallow&lt;/code&gt; for anything that looks like AI, usually blocks the bots that put you &lt;em&gt;into&lt;/em&gt; AI answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are the three types of AI bot?
&lt;/h2&gt;

&lt;p&gt;Before touching the file, label every AI user agent by what it actually does:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Example user agents&lt;/th&gt;
&lt;th&gt;If you block it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Training&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Collects pages to train future models&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GPTBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Your content is (per the vendor) not used for training. AI answers are unaffected in the short term.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Search / answer&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Indexes pages so the assistant can cite them in live answers&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OAI-SearchBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;You stop appearing in ChatGPT search answers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;User-triggered&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Fetches a page when a user asks the assistant about it&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ChatGPT-User&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Inconsistent: OpenAI notes that robots.txt rules may not apply to these fetches&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Google works differently, and this is where most teams slip up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Googlebot&lt;/code&gt;&lt;/strong&gt; governs Search, &lt;em&gt;including&lt;/em&gt; AI Overviews and AI Mode. Block it and you vanish from all three.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Google-Extended&lt;/code&gt;&lt;/strong&gt; is a control token for some of Google's AI uses. Blocking it does &lt;strong&gt;not&lt;/strong&gt; remove you from AI Overviews or AI Mode.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vendors rename and add agents regularly. Check the official lists, such as &lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;OpenAI's crawler docs&lt;/a&gt; and &lt;a href="https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers" rel="noopener noreferrer"&gt;Google's crawler overview&lt;/a&gt;, before you edit anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a sensible default look like?
&lt;/h2&gt;

&lt;p&gt;If you're a business whose buyers use ChatGPT or Gemini to research vendors, you usually want to be &lt;em&gt;found&lt;/em&gt; in AI answers. A reasonable starting point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# --- AI search / answer crawlers: allow (keeps you in live AI answers)
User-agent: OAI-SearchBot
Allow: /

# --- User-triggered fetchers: allow (a real person asked about your page)
User-agent: ChatGPT-User
Allow: /

# --- Training crawlers: a business decision (see below)
# Option A: allow (your content can shape how future models describe you)
User-agent: GPTBot
Allow: /
# Option B: block training only
# User-agent: GPTBot
# Disallow: /

# --- Google: never block Googlebot if you want Search + AI Overviews
User-agent: Googlebot
Allow: /

# Google-Extended: opt-out token for some AI uses; does NOT affect AI Overviews
# User-agent: Google-Extended
# Disallow: /

# Keep private areas private for everyone
User-agent: *
Disallow: /admin/
Disallow: /api/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Treat this as a template, not gospel. Publishers whose content &lt;em&gt;is&lt;/em&gt; the product (news, research, courses) have a very different calculation from a SaaS or services company.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why isn't robots.txt enough on its own?
&lt;/h2&gt;

&lt;p&gt;This is the step most audits miss: &lt;strong&gt;your CDN or WAF can override robots.txt.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If Cloudflare, Akamai, Fastly or your WAF has a "block AI bots" toggle switched on, your carefully written &lt;code&gt;Allow&lt;/code&gt; lines do nothing. The bot never reaches the file. So check both layers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;robots.txt:&lt;/strong&gt; what you &lt;em&gt;say&lt;/em&gt; is allowed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CDN / firewall bot rules:&lt;/strong&gt; what is &lt;em&gt;actually&lt;/em&gt; allowed through&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A quick way to see what a bot experiences is to request a page with its user agent and check the status code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s -o /dev/null -w &lt;span class="s2"&gt;"%{http_code}\n"&lt;/span&gt; \
  -A &lt;span class="s2"&gt;"Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"&lt;/span&gt; \
  https://example.com/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A &lt;code&gt;403&lt;/code&gt; or a challenge page means something upstream is blocking it, whatever robots.txt says. This isn't a perfect test: some CDNs verify real bots by IP address, so a spoofed user agent can be treated differently. It does catch the most common misconfiguration, a blanket bot block.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's a 5-step checklist for your next sprint?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Read robots.txt and the CDN/WAF bot settings together.&lt;/strong&gt; The network-level rule wins.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;List every AI user agent&lt;/strong&gt; you allow or block, and tag each one &lt;em&gt;training&lt;/em&gt;, &lt;em&gt;search&lt;/em&gt; or &lt;em&gt;user-triggered&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Allow search crawlers and user fetchers&lt;/strong&gt; unless you have a specific reason not to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make the training decision with the business&lt;/strong&gt;, meaning whoever owns content licensing. It isn't purely an engineering call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recheck after every change and every quarter.&lt;/strong&gt; New agents appear, and defaults change.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How can you check your site quickly?
&lt;/h2&gt;

&lt;p&gt;If you want a quick read on where you stand, the team at &lt;strong&gt;&lt;a href="https://sigzenai.com/insights/block-ai-training-crawlers-keep-answer-bots" rel="noopener noreferrer"&gt;Sigzen AI&lt;/a&gt;&lt;/strong&gt; has written up the full decision table behind this post, including what happens with each block/allow combination.&lt;/p&gt;

&lt;p&gt;They also built a free &lt;a href="https://sigzenai.com/free-tools/ai-crawler-check" rel="noopener noreferrer"&gt;AI crawler check&lt;/a&gt;. Enter a domain and it reads your robots.txt against 16 named AI crawlers (OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance, Common Crawl and others), returning &lt;em&gt;allowed&lt;/em&gt;, &lt;em&gt;blocked&lt;/em&gt; or &lt;em&gt;partly blocked&lt;/em&gt; for each. It only reads robots.txt, so pair it with the &lt;code&gt;curl&lt;/code&gt; test above to catch CDN-level blocks.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;How does your team handle this? Do you block training bots, allow everything, or let the CDN decide? I'd like to hear in the comments.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written by the Sigzen AI editorial team. Sigzen AI helps B2B brands get described accurately in ChatGPT, Gemini, Copilot and Google AI answers, and builds AI agents for operations teams. More of our research is at &lt;a href="https://sigzenai.com/insights" rel="noopener noreferrer"&gt;sigzenai.com/insights&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>seo</category>
      <category>webdev</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
