Short answer: most big sites block AI training crawlers, and they treat Anthropic more harshly than OpenAI. On 1 November 2026 we read the robots.txt of 30 well-known sites. 28 answered. Of those 28, 25 block ClaudeBot and 17 block GPTBot. Eight let GPTBot in and keep ClaudeBot out, including The Guardian, The Washington Post, The Wall Street Journal and The Verge. Only 2 of the 28 publish an llms.txt file. You can check any list of sites the same way with AI Crawler Checker, an Apify Actor. It costs $0.003 per site and needs no key.
The numbers (robots.txt, path "/", 1 November 2026)
28 of 30 sites answered. npr.org did not answer within 15 seconds. stackoverflow.com returned HTTP 418 to an automated client, and we do not get around that.
| Crawler | Who runs it | What it is for | Sites blocking it (of 28) |
|---|---|---|---|
| ClaudeBot | Anthropic | Training | 25 |
| Bytespider | ByteDance | Training | 24 |
| CCBot | Common Crawl | Training dataset | 23 |
| Applebot-Extended | Apple | Training | 23 |
| PerplexityBot | Perplexity | Search index | 21 |
| meta-externalagent | Meta | Training | 20 |
| Amazonbot | Amazon | Training and Alexa | 18 |
| GPTBot | OpenAI | Training | 17 |
| Google-Extended | Gemini training | 16 | |
| Claude-SearchBot | Anthropic | Search | 16 |
| Claude-User | Anthropic | Fetches a page for a user | 16 |
| Perplexity-User | Perplexity | Fetches a page for a user | 14 |
| ChatGPT-User | OpenAI | Fetches a page for a user | 10 |
| OAI-SearchBot | OpenAI | ChatGPT search | 8 |
Search crawlers are treated more gently than training crawlers. OAI-SearchBot, which decides whether a page can appear in ChatGPT search, is blocked by only 8 of the 28. Six sites close every AI search crawler: nytimes.com, cnn.com, reddit.com, yelp.com, imdb.com and amazon.com. Six leave them all open: wikipedia.org, medium.com, github.com, etsy.com, tripadvisor.com and webmd.com.
GPTBot allowed, ClaudeBot blocked
These eight sites give OpenAI's training crawler a different answer from Anthropic's: theguardian.com, washingtonpost.com, wsj.com, theatlantic.com, theverge.com, wired.com, arstechnica.com, allrecipes.com.
Most of them have announced content licensing deals with OpenAI. We have not checked each deal, so read this as a pattern, not proof. The robots.txt itself is the primary source. Here is The Washington Post's, fetched the same day:
User-agent: ClaudeBot
Disallow: /
Allow: /creativegroup/
GPTBot has no group of its own there, so it falls under User-agent: *, which only closes a few print and email paths.
Why this matters if you publish
If you want your pages cited in ChatGPT, Perplexity or Claude answers, the search and user crawlers are the ones to leave open. You can block the training crawlers and keep the search ones, as theguardian.com does for OpenAI. A blanket Disallow: / for User-agent: * also blocks every AI crawler that has no group of its own. reddit.com does this, so it shows up as blocking all of them.
llms.txt is still rare: github.com and etsy.com publish one, and the 26 others do not.
Run it on your own list
curl -X POST "https://api.apify.com/v2/acts/appdata-labs~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"websites": ["nytimes.com", "theguardian.com", "reddit.com", "github.com"]}'
This is the call behind the tables above, with the list cut short. The full 30-site run took 44 seconds. It uses Apify's run-sync-get-dataset-items endpoint. AI agents connected to the Apify MCP server can call it as a tool. Each row is at most 1.3 KB.
Each row has:
domain-
robotsTxtStatus:found,none,refusedorerror -
bots: an allowed or blocked verdict for each of the 16 crawlers blockedBotsblocksAllAi-
aiSearchAccessandaiTrainingAccess:full,partialornone llmsTxtFound
The Actor matches rules the way RFC 9309 says: the longest match wins, Allow beats Disallow on a tie, and * and $ wildcards are handled. Set path to check a page other than the home page.
FAQ
Does robots.txt actually stop these crawlers?
It is a request, not a lock. OpenAI, Anthropic, Google, Apple and Perplexity all say their crawlers honour it. This Actor reports what each site asks for. It does not test what a crawler really does.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot collects training data. Claude-User fetches a page because a person asked Claude to read it. Claude-SearchBot builds Claude's search index. OpenAI splits its crawlers the same way: GPTBot, ChatGPT-User and OAI-SearchBot.
Am I charged for sites that refuse to answer?
No. A site that answers 401, 403, 418 or 429, or does not answer at all, still gets a row, but that row is free.
Can I watch for changes?
Yes. Schedule the Actor weekly and compare blockedBots between runs. A site that adds a crawler shows up straight away.
Related: AI Search Readiness Audit scores one page from 0 to 100 on how easily ChatGPT, Perplexity and AI Overviews can cite it, and lists what to fix.
Top comments (1)
We need to produce a short YouTube comment, casual, one or two sentences, starts with lowercase, specific reaction or question about this video. Avoid formal language. Should ask something about which websites block GPTBot and ClaudeBot. Could mention surprise about certain site blocking. Maybe ask about why Anthropic is treated harsher. Keep it short. No punctuation issues. No double hyphen. No markdown. Just comment text. Let's craft: "i was shocked to see reddit block ClaudeBot but not GPTBot, any idea why they