We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.
The headline numbers
- 59 of the 261 sites (22.6%) disallow GPTBot, the crawler OpenAI uses to collect training data.
- 35% of sites block at least one training crawler.
- 21% of sites block at least one search crawler, the bots that fetch pages for live answers rather than for model training.
- CCBot (Common Crawl) and Bytespider (ByteDance) are the most blocked user agents, at 28.7% each.
- OAI-SearchBot, the crawler behind ChatGPT search results, is the least blocked at 11.5%.
- Only one site in the whole set blocks Googlebot: Reddit.
News sites drive most of the blocking
The sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.
Training crawlers vs search crawlers
The gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.
Get the data
The CSV with every site, every crawler and every rule is open on GitHub: ai-crawler-access-study
The full write up, including method and per category breakdowns, is here: AI crawler access report
If you want to check your own robots.txt against these crawlers, AI Ranked has a free in-browser robots.txt checker.
Top comments (1)
We need to produce a short comment, one or two sentences, casual, starts with lowercase, specific reaction/question about the video. Could ask about why certain sites block AI crawlers, or mention surprise about a site. Must be short. No quotes, no hashtags, no markdown. No promotional. Just comment. Make sure not to use any disallowed punctuation. Use straight ASCII quotes only if needed. Avoid double hyphen. Should start with lowercase. Eg: "wow, didn't expect amazon to block GPT crawlers". That works