<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Grubbomatic </title>
    <description>The latest articles on DEV Community by Grubbomatic  (@grubbomatic).</description>
    <link>https://dev.to/grubbomatic</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4187521%2F8ffe5471-e562-405d-90f3-bc1d87ca2088.png</url>
      <title>DEV Community: Grubbomatic </title>
      <link>https://dev.to/grubbomatic</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/grubbomatic"/>
    <language>en</language>
    <item>
      <title>robots.txt asks, the server answers. Method and data from the AI Access Index</title>
      <dc:creator>Grubbomatic </dc:creator>
      <pubDate>Wed, 18 Nov 2026 22:58:06 +0000</pubDate>
      <link>https://dev.to/grubbomatic/robotstxt-asks-the-server-answers-method-and-data-from-the-ai-access-index-98g</link>
      <guid>https://dev.to/grubbomatic/robotstxt-asks-the-server-answers-method-and-data-from-the-ai-access-index-98g</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: we build Crawl Readiness, a website check that also sells paid audits and monitoring. We also maintain Machine Readiness, the standard Crawl Readiness measures against. No standards body or AI company has adopted it. Both are made by Tundrastone (Ontario, Canada).&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On 1 November 2026 we checked the top 1,000 hostnames of &lt;a href="https://tranco-list.eu/list/8PXYV" rel="noopener noreferrer"&gt;Tranco list 8PXYV&lt;/a&gt; for edition 1 of the monthly &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11" rel="noopener noreferrer"&gt;AI Access Index&lt;/a&gt;; 684 answered. Of the 547 tested sites that allow at least one of four AI crawlers in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot), 62 (11.3%) refused at least one crawler they allow, with a 401, 403, 429 or 503. Leaving out the 17 sites whose only refusals were 429s, it is 45 (8.2%). Of the 124 tested sites that block at least one of the four, 79 (63.7%) answered at least one crawler they block with a 2xx status.&lt;/p&gt;

&lt;p&gt;Our test sends a user-agent string that names each crawler, from a cloud runner, not the vendors' IP addresses, so a site that verifies crawlers by address could refuse the test and admit the vendor's own crawler. The 11.3% counts sites that turned away a test request, so it is more exposed to that than the 63.7% is. A page served to a user-agent string does not show what the vendor's crawler would get.&lt;/p&gt;

&lt;p&gt;robots.txt is a request, not a lock: it asks, and only the server, or a CDN or firewall in front of it, can refuse. A 2xx for a crawler that robots.txt asks to stay away breaks no rule, so we call it served, not a fault. In the standard we maintain, server enforcement is optional (A5, a MAY). Open or closed, a policy is a choice, and a block that holds is reported as working as intended.&lt;/p&gt;

&lt;h2&gt;
  
  
  Method
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sample.&lt;/strong&gt; Tranco list 8PXYV, top 1,000, downloaded on 2026-10-11 and frozen. 684 answered; 316 did not (some are infrastructure hostnames with no website). 599 of the 684 could be tested, meaning the server's answers can be read against its robots.txt; 85 could not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run.&lt;/strong&gt; 13:27 to 13:37 UTC: each site once, six at a time, nothing retried.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Server test.&lt;/strong&gt; The check's own request is the baseline. The homepage is then requested again as each of the four crawlers, keeping only the status. A crawler counts as blocked when robots.txt disallows it; a site with no robots.txt allows all four.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Recount the figures from the CSV
&lt;/h2&gt;

&lt;p&gt;Save &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv" rel="noopener noreferrer"&gt;results.csv&lt;/a&gt; and this script as recount.mjs, then run &lt;code&gt;node recount.mjs results.csv&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// split(",") is safe: only the last column holds commas.&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;readFileSync&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"node:fs"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;","&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromEntries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;four&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"GPTBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"OAI-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ClaudeBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PerplexityBot"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;has&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;four&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;answered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;tested&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;answered&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;integrity&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="s2"&gt;"unknown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;allows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tested&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"_robots"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"allowed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tested&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"_robots"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"blocked"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;refused&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;allows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"_outcome"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"refused-while-allowed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;refusedNot429&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;refused&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;four&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"_outcome"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"refused-while-allowed"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"_status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="s2"&gt;"429"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;served&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"_outcome"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"served-while-blocked"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;opened&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;homepage_status&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"answered"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;answered&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tested"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tested&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"allow base"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;allows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"refused"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;refused&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"without 429-only"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;refusedNot429&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"block base"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"served"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;served&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"own request opened"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opened&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"served"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opened&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;served&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints 684 and 599, then 547, 62 and 45, then 124 and 79, then 91 and 74. In the &lt;code&gt;integrity&lt;/code&gt; column, &lt;code&gt;both&lt;/code&gt; (2 sites) counts in each, so 77 + 2 = 79 and 60 + 2 = 62. In summary.json the headline counts are under &lt;code&gt;server&lt;/code&gt;. Each row also has a score and a grade. A site whose robots.txt blocks AI crawlers in a pattern the check reads as deliberate is graded Restricted by Design, a choice rather than a fault. Its blocked crawlers still cost points, so its score falls.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One pass, homepages only, from one vantage point. A 429 can be a rate limit, so the lower figure leaves 429-only refusals out.&lt;/li&gt;
&lt;li&gt;Served means a 2xx status and nothing more, so a 200 challenge page counts. Among tested sites that block at least one crawler and whose homepage opened for the check's own request, 74 of 91 (81.3%) answered a blocked crawler with a 2xx, against 79 of 124 above.&lt;/li&gt;
&lt;li&gt;This is edition 1, so there is no trend. In October we posted &lt;a href="https://dev.to/grubbomatic/your-robotstxt-says-yes-your-cdn-says-no-we-tested-the-top-1000-sites-883"&gt;a study&lt;/a&gt; and said we would re-run it monthly. The Index is a new series, not that re-run: the study used a browser request as its baseline and a different network, so we do not set the two side by side. Edition 2 is scheduled for 1 December 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://hasdata.com/blog/ai-crawler-block-index" rel="noopener noreferrer"&gt;HasData&lt;/a&gt; published a declared-versus-served test of GPTBot in its AI Crawler Block Index, a different index with a similar name, using a July 2026 baseline and one US datacenter IP. &lt;a href="https://blog.cloudflare.com/bot-preference-sync/" rel="noopener noreferrer"&gt;Cloudflare&lt;/a&gt; describes the same gap between a robots.txt and its edge rules, and &lt;a href="https://tollbit.com/state-of-the-bots/q1-q2-2026/" rel="noopener noreferrer"&gt;TollBit&lt;/a&gt; reports on bots that bypass disallow rules. Methods differ, so we do not compare the numbers.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Data and licences
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv" rel="noopener noreferrer"&gt;results.csv&lt;/a&gt;, &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11/changes.csv" rel="noopener noreferrer"&gt;changes.csv&lt;/a&gt; (a header row only in edition 1) and &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11/summary.json" rel="noopener noreferrer"&gt;summary.json&lt;/a&gt; are CC BY-NC 4.0. Quote the numbers anywhere with credit, press included. Commercial use of the files, such as in a product or a dataset you sell, needs permission: &lt;a href="mailto:support@crawlreadiness.com"&gt;support@crawlreadiness.com&lt;/a&gt;. Cite as "Crawl Readiness, AI Access Index, November 2026, &lt;a href="http://www.crawlreadiness.com/ai-access-index/2026-11." rel="noopener noreferrer"&gt;www.crawlreadiness.com/ai-access-index/2026-11.&lt;/a&gt;"&lt;/li&gt;
&lt;li&gt;The Index page said CC BY 4.0 from 1 November until the change merged at 16:48 UTC on 3 November, about 51 hours.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;rank&lt;/code&gt; column is Tranco's; check Tranco's list page before reuse.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://machinereadiness.com/crawlers.json" rel="noopener noreferrer"&gt;crawler registry&lt;/a&gt; and the &lt;a href="https://www.crawlreadiness.com/research/ai-crawler-access-october-2026" rel="noopener noreferrer"&gt;October study&lt;/a&gt; data are CC BY 4.0.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;p&gt;Crawl Readiness checks that a website is set up for AI the way its owner intends: readable by the AI crawlers it wants reading it, closed to the ones it wants closed, and with a server that keeps to what its robots.txt says. Readiness is not a promise of being mentioned. Corrections: comment here or write to &lt;a href="mailto:support@crawlreadiness.com"&gt;support@crawlreadiness.com&lt;/a&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The AI Access Index, November 2026: &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11" rel="noopener noreferrer"&gt;https://www.crawlreadiness.com/ai-access-index/2026-11&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The standard Crawl Readiness measures against, Machine Readiness 1.1: &lt;a href="https://machinereadiness.com/standard" rel="noopener noreferrer"&gt;https://machinereadiness.com/standard&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The free check, no signup: &lt;a href="https://www.crawlreadiness.com/" rel="noopener noreferrer"&gt;https://www.crawlreadiness.com/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>research</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)</title>
      <dc:creator>Grubbomatic </dc:creator>
      <pubDate>Thu, 15 Oct 2026 22:00:06 +0000</pubDate>
      <link>https://dev.to/grubbomatic/your-robotstxt-says-yes-your-cdn-says-no-we-tested-the-top-1000-sites-883</link>
      <guid>https://dev.to/grubbomatic/your-robotstxt-says-yes-your-cdn-says-no-we-tested-the-top-1000-sites-883</guid>
      <description>&lt;p&gt;Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets. So on 11 October 2026 we tested that part too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The one-line result:&lt;/strong&gt; of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.&lt;/p&gt;

&lt;p&gt;That is the gap between policy and reality. Here is how we got the number, what else fell out of the run, and how to test your own site in a minute.&lt;/p&gt;

&lt;h2&gt;
  
  
  Method, short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sample:&lt;/strong&gt; the Tranco top 1,000 (list 8PXYV, generated 10 Oct 2026). Tranco averages several popularity lists, so it is not one vendor's view.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reach:&lt;/strong&gt; 688 of the 1,000 answered at all. 549 answered a browser user agent with a 2xx, and those 549 are the base for the parity numbers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parity test:&lt;/strong&gt; fetch the homepage five times, as a browser and then as &lt;code&gt;GPTBot&lt;/code&gt;, &lt;code&gt;OAI-SearchBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt; and &lt;code&gt;PerplexityBot&lt;/code&gt;, with each vendor's published user-agent string. A 401, 403, 429 or 503 for the crawler when the browser got a 200 counts as "refused".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Robots:&lt;/strong&gt; parse robots.txt for the 35 crawler tokens we score (training, search and answer crawlers, user-triggered fetchers), per vendor documentation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Everything else:&lt;/strong&gt; llms.txt, agents.json, markdown copies, JSON-LD, feeds, meta robots, forms, scored with the same checker anyone can run on crawlreadiness.com.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One run, 11 October 2026, 15 minutes, from one IP in North America. The per-site CSV, the per-crawler CSV and a summary JSON are published under CC BY 4.0 with the study.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the server said when robots.txt said yes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Crawler&lt;/th&gt;
&lt;th&gt;Refused by the server&lt;/th&gt;
&lt;th&gt;…while robots.txt allowed it&lt;/th&gt;
&lt;th&gt;…excluding 429-only refusals&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPTBot&lt;/td&gt;
&lt;td&gt;14.4%&lt;/td&gt;
&lt;td&gt;11.1%&lt;/td&gt;
&lt;td&gt;10.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ClaudeBot&lt;/td&gt;
&lt;td&gt;14.8%&lt;/td&gt;
&lt;td&gt;10.9%&lt;/td&gt;
&lt;td&gt;9.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PerplexityBot&lt;/td&gt;
&lt;td&gt;11.3%&lt;/td&gt;
&lt;td&gt;8.4%&lt;/td&gt;
&lt;td&gt;7.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OAI-SearchBot&lt;/td&gt;
&lt;td&gt;9.8%&lt;/td&gt;
&lt;td&gt;8.0%&lt;/td&gt;
&lt;td&gt;7.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things to notice.&lt;/p&gt;

&lt;p&gt;First, OAI-SearchBot is the crawler ChatGPT search uses to fetch pages, and it is refused on 8% of sites that explicitly allow it. On those sites, most likely, a bot-management rule or a "block AI bots" switch also closes the door their robots.txt leaves open.&lt;/p&gt;

&lt;p&gt;Second, the 429s are a judgement call, which is why they get their own column. A 429 is "slow down", not "go away". But a crawler that is rate-limited on the very first request to the homepage, from one IP, is for practical purposes blocked, so the headline keeps them and the third column shows the figure without them.&lt;/p&gt;

&lt;p&gt;The mirror image exists too: for GPTBot, 53 sites disallow it in robots.txt and serve it anyway. robots.txt is advisory in both directions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What robots.txt itself says
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;82.7% of reachable sites have a robots.txt at all.&lt;/li&gt;
&lt;li&gt;25.1% block at least one search or answer crawler, the ones that fetch pages for AI search and answers. 10.2% block the majority.&lt;/li&gt;
&lt;li&gt;26.0% block at least one training crawler. Only 2.6% do the thing the guides recommend: block training crawlers only, and keep the search and answer crawlers open.&lt;/li&gt;
&lt;li&gt;2.5% carry a Content-Signal directive. 62.4% list a sitemap.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The emerging files, honestly counted
&lt;/h2&gt;

&lt;p&gt;Adoption numbers for llms.txt get quoted a lot, and most of them are inflated by soft 404s: sites that answer 200 with an HTML "not found" page for any path. We fetched every reported llms.txt again and kept only the ones that are a text file beginning with a Markdown heading.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Any 2xx answer&lt;/th&gt;
&lt;th&gt;A real file&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llms.txt&lt;/td&gt;
&lt;td&gt;34.3%&lt;/td&gt;
&lt;td&gt;15.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llms-full.txt&lt;/td&gt;
&lt;td&gt;21.9%&lt;/td&gt;
&lt;td&gt;not re-verified&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;agents.json&lt;/td&gt;
&lt;td&gt;0.3%&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A markdown copy of the page (Accept: text/markdown or .md)&lt;/td&gt;
&lt;td&gt;3.5% served, 1.5% linked&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WebMCP form annotations&lt;/td&gt;
&lt;td&gt;1 site of the 254 with forms&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So the real llms.txt number for the top 1,000 is about one site in seven, not one in three. If you are building anything that reads these files, test for the soft 404 first: check the content type, and check that the first line is a heading.&lt;/p&gt;

&lt;h2&gt;
  
  
  Readiness overall
&lt;/h2&gt;

&lt;p&gt;Scored with our checker (v2, which counts only the crawlers that fetch pages for AI search and answers, and treats a deliberate block as a valid choice rather than a failure): mean 84.5, median 90. 355 sites are AI Ready, 106 are Restricted by Design, 88 are JavaScript-Dependent (the crawler is allowed in, but the content renders client-side, which most crawlers do not run), and 126 could only be partly checked because they blocked our checker too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test your own site
&lt;/h2&gt;

&lt;p&gt;The quickest check is from a terminal. If the second command gets a 403 and the first a 200, your robots.txt is not the whole story:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-sI&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0"&lt;/span&gt; https://example.com/ | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;
curl &lt;span class="nt"&gt;-sI&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt; &lt;span class="s2"&gt;"OAI-SearchBot/1.0; +https://openai.com/searchbot"&lt;/span&gt; https://example.com/ | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The study's parity script is a single dependency-free Node file. Point it at any CSV of domains:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-O&lt;/span&gt; https://www.crawlreadiness.com/research/ai-crawler-access-2026-10/parity.mjs
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"example.com"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; list.csv
node parity.mjs list.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And if you want the full readiness check with the fix list, the free checker at crawlreadiness.com runs the same code the study ran, and each of the 51 crawlers has a reference page with its documented user agent, its purpose, whether it honours robots.txt, and the exact robots.txt lines to allow or block it: &lt;a href="https://www.crawlreadiness.com/crawlers" rel="noopener noreferrer"&gt;https://www.crawlreadiness.com/crawlers&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do if you are in the 14%
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Look at the bot-management layer before robots.txt. On Cloudflare that is Security → Bots, where "Block AI bots" blocks search and answer crawlers along with training crawlers. On other CDNs and WAFs, look for a managed rule that matches the user-agent strings above.&lt;/li&gt;
&lt;li&gt;Decide per purpose, not per vendor. Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search) is a legitimate and common choice; the 2.6% of top sites that do it carefully show it works.&lt;/li&gt;
&lt;li&gt;Verify by request, not by reading the file. The curl pair above takes ten seconds.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Data and reproduction
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Study page, with definitions and the per-crawler table: &lt;a href="https://www.crawlreadiness.com/research/ai-crawler-access-october-2026" rel="noopener noreferrer"&gt;https://www.crawlreadiness.com/research/ai-crawler-access-october-2026&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;results.csv (one row per site), crawlers.csv (one row per crawler token), summary.json and parity.mjs, all CC BY 4.0, linked from the study page.&lt;/li&gt;
&lt;li&gt;Update, November 2026: we said we would re-run this monthly. The follow-up is the &lt;a href="https://www.crawlreadiness.com/ai-access-index/2026-11" rel="noopener noreferrer"&gt;AI Access Index&lt;/a&gt;, the same 1,000 hostnames checked on the 1st of every month. It is measured from a different network, with the check's own request as the baseline instead of a browser, so its figures start a new series rather than continuing these.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you find a site we have misclassified, the per-site row shows every status code we got; open an issue or write to &lt;a href="mailto:support@crawlreadiness.com"&gt;support@crawlreadiness.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>opensource</category>
      <category>research</category>
    </item>
  </channel>
</rss>
