DEV Community

crawlerpeek
crawlerpeek

Posted on Originally published at crawlerpeek.com

We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

#ai #seo #webdev #robotstxt

On 29 October 2026 we fetched the robots.txt of 151 large websites (news, shops, SaaS, docs, social networks, education, SEO) and ran every file through a linter. 135 files could be read.

The short version

  • 44% block at least one of the six best-known AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, OAI-SearchBot) from /.
  • ClaudeBot is blocked more often than GPTBot: 55 sites (41%) vs. 42 (31%).
  • Training yes, search no: 22 sites block GPTBot but leave OAI-SearchBot (ChatGPT search) open. 20 block both.
  • News blocks, software does not: 19 of 34 news sites block GPTBot, 2 of 33 SaaS sites do, and no developer-docs or SEO site does.
  • A third carry dead lines: 44 files still name anthropic-ai or Claude-Web, which are not on Anthropic's current crawler list.
  • The syntax is fine: one line in 135 files silently does nothing. The bigger issue is delivery: 9 sites answered /robots.txt with HTTP 403, 3 with an HTML page.

Who blocks which crawler

Crawler Blocked from / Share of 135
ClaudeBot 55 41%
Bytespider 55 41%
CCBot 53 39%
Applebot-Extended 48 36%
meta-externalagent 43 32%
GPTBot 42 31%
Google-Extended 38 28%
PerplexityBot 38 28%
Claude-SearchBot 30 22%
ChatGPT-User 26 19%
OAI-SearchBot 20 15%
Googlebot 1 1%

Crawlers that collect training data are blocked two to three times as often as the ones behind AI search and user-triggered fetches.

A 403 on robots.txt does not keep bots out

Nine sites, including cnn.com, nih.gov and npmjs.com, answered our identifiable non-browser client with HTTP 403 for /robots.txt. Under RFC 9309 a 4xx answer means a crawler may fetch everything, and Google treats it the same way. A firewall that blocks /robots.txt hides the rules instead of enforcing them. (Real crawlers may get a different answer; we can only see what an unknown client gets.)

Gzip nobody asked for

www.forbes.com sends its robots.txt gzip-compressed even when the request asks for no compression. Real crawlers decode it. Our own CLI did not, until this study: before 0.3.1 it read the file as binary and found no rules. If you test robots.txt with your own scripts, decode Content-Encoding.

Check your own file 

npx crawlerpeek lint https://example.com   # syntax, legacy tokens, extensions
npx crawlerpeek scan https://example.com   # every sitemap URL against robots.txt
Enter fullscreen mode Exit fullscreen mode

Or in the browser, without installing anything: robots.txt validator and sitemap robots.txt check.

The full write-up with the sector table, the 13 sites that block every AI crawler, Content-Signal and RSL adoption, and the method is on crawlerpeek.com. The CLI and MCP server are open source: github.com/crawlerpeek/crawlerpeek.

Top comments (0)