<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Daniel Di Cerbo</title>
    <description>The latest articles on DEV Community by Daniel Di Cerbo (@danieldicerbo).</description>
    <link>https://dev.to/danieldicerbo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4241951%2Fe6982617-13ee-4332-87d9-c5dc5782f320.png</url>
      <title>DEV Community: Daniel Di Cerbo</title>
      <link>https://dev.to/danieldicerbo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/danieldicerbo"/>
    <language>en</language>
    <item>
      <title>How to Check Which AI Crawlers Your robots.txt Actually Blocks</title>
      <dc:creator>Daniel Di Cerbo</dc:creator>
      <pubDate>Fri, 13 Nov 2026 02:55:47 +0000</pubDate>
      <link>https://dev.to/danieldicerbo/how-to-check-which-ai-crawlers-your-robotstxt-actually-blocks-37jl</link>
      <guid>https://dev.to/danieldicerbo/how-to-check-which-ai-crawlers-your-robotstxt-actually-blocks-37jl</guid>
      <description>&lt;p&gt;To check which AI crawlers your robots.txt blocks, take each crawler's user-agent token, find the group in your file that matches it, and read only that group's rules. Then confirm the result in your server logs, because robots.txt states what you want. It doesn't enforce it.&lt;/p&gt;

&lt;p&gt;Most people assume one blanket &lt;code&gt;Disallow&lt;/code&gt; covers every bot. It often doesn't. Here's how the matching works, which bots do which job, and a short audit you can run on any site.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why one file gives different bots different answers
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9309" rel="noopener noreferrer"&gt;RFC 9309&lt;/a&gt; sets the rule. A crawler looks for groups whose &lt;code&gt;User-agent&lt;/code&gt; line matches its token, case-insensitive. If it finds one, it follows that group and ignores the &lt;code&gt;*&lt;/code&gt; group completely. It only falls back to &lt;code&gt;*&lt;/code&gt; when nothing names it.&lt;/p&gt;

&lt;p&gt;Take this file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;GPTBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Googlebot&lt;/span&gt;
&lt;span class="nc"&gt;Allow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it per crawler:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GPTBot&lt;/strong&gt; has its own group, so it reads only &lt;code&gt;Disallow: /&lt;/code&gt; and is blocked from everything.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Googlebot&lt;/strong&gt; has its own group, so it may fetch everything, &lt;code&gt;/private/&lt;/code&gt; included. The &lt;code&gt;*&lt;/code&gt; rule never applies to it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ClaudeBot&lt;/strong&gt; has no named group, so it falls back to &lt;code&gt;*&lt;/code&gt; and may crawl everything except &lt;code&gt;/private/&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One file, three different answers. The blanket rule covered only one of the three bots.&lt;/p&gt;

&lt;p&gt;Inside the group a crawler follows, the longest matching path wins, and when an &lt;code&gt;Allow&lt;/code&gt; and a &lt;code&gt;Disallow&lt;/code&gt; match equally, the &lt;code&gt;Allow&lt;/code&gt; wins. So read the rule for the exact URL you care about, not whichever line comes first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Know which bot does which job
&lt;/h2&gt;

&lt;p&gt;The major AI companies run separate bots for training, for search, and for fetching a page when a user asks. Blocking the wrong one either does nothing or takes you out of answers you wanted to be in.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;Training&lt;/th&gt;
&lt;th&gt;Search and answers&lt;/th&gt;
&lt;th&gt;Fetches on a user's request&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;GPTBot&lt;/td&gt;
&lt;td&gt;OAI-SearchBot&lt;/td&gt;
&lt;td&gt;ChatGPT-User&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;ClaudeBot&lt;/td&gt;
&lt;td&gt;Claude-SearchBot&lt;/td&gt;
&lt;td&gt;Claude-User&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Perplexity&lt;/td&gt;
&lt;td&gt;None listed&lt;/td&gt;
&lt;td&gt;PerplexityBot&lt;/td&gt;
&lt;td&gt;Perplexity-User&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google&lt;/td&gt;
&lt;td&gt;Google-Extended (a token, see below)&lt;/td&gt;
&lt;td&gt;Googlebot&lt;/td&gt;
&lt;td&gt;None listed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three things trip people up:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Blocking GPTBot doesn't remove you from ChatGPT search.&lt;/strong&gt; OAI-SearchBot decides that. &lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;OpenAI's crawler docs&lt;/a&gt; list each bot and what it's for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google-Extended isn't a crawler.&lt;/strong&gt; &lt;a href="https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers" rel="noopener noreferrer"&gt;Google's docs&lt;/a&gt; say it has no separate user-agent string. Googlebot does the crawling and reads the token as a control. You'll never see it in your logs, and blocking it doesn't remove you from Google Search.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User-triggered fetchers may ignore robots.txt.&lt;/strong&gt; When a person asks an assistant to read a page, the vendor treats that as the user's request, not a crawl. OpenAI says robots.txt rules may not apply to ChatGPT-User.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The audit, step by step
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fetch the live file from every hostname&lt;/strong&gt;, for example &lt;code&gt;https://example.com/robots.txt&lt;/code&gt;. Each subdomain needs its own file. Fetch the live file rather than reading the one in your repo, because a CDN can change what's served. Cloudflare's AI bot settings, for one, can add rules to it or block bots at the edge whatever the file says.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Note the status code.&lt;/strong&gt; Under RFC 9309, a 4xx response means crawlers may fetch everything. A 5xx or an unreachable file means they should assume they can fetch nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;List every named group.&lt;/strong&gt; Write down each &lt;code&gt;User-agent&lt;/code&gt; line.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;For each bot you care about&lt;/strong&gt; (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot), find its group, or &lt;code&gt;*&lt;/code&gt; if it has none, and read only that group.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test a real URL&lt;/strong&gt;, not just the homepage. The answer for &lt;code&gt;/blog/post&lt;/code&gt; can differ from the answer for &lt;code&gt;/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Re-test after every edit.&lt;/strong&gt; Adding a named group for one bot can quietly change what that bot may crawl, as the example above shows.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you'd rather not do this by hand, I built a free &lt;a href="https://verand.ai/tools/ai-crawler-access-checker" rel="noopener noreferrer"&gt;AI crawler access checker&lt;/a&gt;: enter one URL and it shows, for each major AI crawler, whether it's allowed and which rule decides.&lt;/p&gt;

&lt;h2&gt;
  
  
  Confirm it in your server logs
&lt;/h2&gt;

&lt;p&gt;The file shows intent. Logs show behaviour. Search your access logs for the tokens above and read the pattern:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A request for &lt;code&gt;/robots.txt&lt;/code&gt;, then nothing for disallowed paths:&lt;/strong&gt; a compliant crawler obeying your file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;403s or challenge pages:&lt;/strong&gt; your server, firewall or CDN is blocking the bot. That's a separate layer from robots.txt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;200s on paths you disallowed:&lt;/strong&gt; either you've misread which group applies, or the bot is ignoring the file.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;User-agent strings are easy to fake, so check that a request is genuine before you act on it. OpenAI publishes IP ranges for its bots, and Google documents a reverse DNS check for Googlebot.&lt;/p&gt;

&lt;h2&gt;
  
  
  A common policy: be cited, don't be trained on
&lt;/h2&gt;

&lt;p&gt;Many sites want assistants to find and cite their pages but don't want those pages used for training. In robots.txt, that looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;GPTBot&lt;/span&gt;
&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ClaudeBot&lt;/span&gt;
&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Google&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Extended&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;OAI&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;SearchBot&lt;/span&gt;
&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Claude&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;SearchBot&lt;/span&gt;
&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;PerplexityBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;Disallow: /private/&lt;/code&gt; repeated in the search group. Once those bots have their own group, they ignore &lt;code&gt;*&lt;/code&gt;, so any path you want kept private has to be restated there. It's the same trap as in the first example.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the list current
&lt;/h2&gt;

&lt;p&gt;New AI user agents appear constantly. The community-maintained &lt;a href="https://github.com/ai-robots-txt/ai.robots.txt" rel="noopener noreferrer"&gt;ai.robots.txt&lt;/a&gt; project lists 208 as of November 2026, and it can generate robots.txt and server configs for you. Compare your named groups against it every few months, and re-run the audit whenever you change the file.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>ai</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
