The latest articles on DEV Community by ranaiman (@ran__8243a506d14cf64). https://dev.to/ran__8243a506d14cf64 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4177704%2F20f34b96-d326-49bf-9994-20a3a59bef8c.png https://dev.to/ran__8243a506d14cf64 en ranaiman Thu, 15 Oct 2026 13:35:54 +0000 https://dev.to/airanked/which-ai-crawlers-do-261-top-websites-block-study-data-41e3 https://dev.to/airanked/which-ai-crawlers-do-261-top-websites-block-study-data-41e3 <p>We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.</p> <h2> The headline numbers </h2> <ul> <li>59 of the 261 sites (22.6%) disallow GPTBot, the crawler OpenAI uses to collect training data.</li> <li>35% of sites block at least one training crawler.</li> <li>21% of sites block at least one search crawler, the bots that fetch pages for live answers rather than for model training.</li> <li>CCBot (Common Crawl) and Bytespider (ByteDance) are the most blocked user agents, at 28.7% each.</li> <li>OAI-SearchBot, the crawler behind ChatGPT search results, is the least blocked at 11.5%.</li> <li>Only one site in the whole set blocks Googlebot: Reddit.</li> </ul> <h2> News sites drive most of the blocking </h2> <p>The sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.</p> <h2> Training crawlers vs search crawlers </h2> <p>The gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.</p> <h2> Get the data </h2> <p>The CSV with every site, every crawler and every rule is open on GitHub: <a href="https://github.com/airanked/ai-crawler-access-study" rel="noopener noreferrer">ai-crawler-access-study</a></p> <p>The full write up, including method and per category breakdowns, is here: <a href="https://airanked.ai/research/ai-crawler-access-report" rel="noopener noreferrer">AI crawler access report</a></p> <p>If you want to check your own robots.txt against these crawlers, AI Ranked has a <a href="https://airanked.ai/tools/robots-txt-ai-checker" rel="noopener noreferrer">free in-browser robots.txt checker</a>.</p> ai seo webdev data