Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.
The one-line result: of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.
That is the gap between policy and reality. Here is how we got the number, what else fell out of the run, and how to test your own site in a minute.
Method, short version
- Sample: the Tranco top 1,000 (list 8PXYV, generated 10 Oct 2026). Tranco averages several popularity lists, so it is not one vendor's view.
- Reach: 688 of the 1,000 answered at all. 549 answered a browser user agent with a 2xx, and those 549 are the base for the parity numbers.
-
Parity test: fetch the homepage five times, as a browser and then as
GPTBot,OAI-SearchBot,ClaudeBotandPerplexityBot, with each vendor's published user-agent string. A 401, 403, 429 or 503 for the crawler when the browser got a 200 counts as "refused". - Robots: parse robots.txt for the 35 crawler tokens we score (training, search and answer crawlers, user-triggered fetchers), per vendor documentation.
- Everything else: llms.txt, agents.json, markdown copies, JSON-LD, feeds, meta robots, forms, scored with the same checker anyone can run on crawlreadiness.com.
One run, 11 October 2026, 15 minutes, from one IP in North America. The per-site CSV, the per-crawler CSV and a summary JSON are published under CC BY 4.0 with the study.
What the server said when robots.txt said yes
| Crawler | Refused by the server | …while robots.txt allowed it | …excluding 429-only refusals |
|---|---|---|---|
| GPTBot | 14.4% | 11.1% | 10.0% |
| ClaudeBot | 14.8% | 10.9% | 9.1% |
| PerplexityBot | 11.3% | 8.4% | 7.3% |
| OAI-SearchBot | 9.8% | 8.0% | 7.3% |
Two things to notice.
First, OAI-SearchBot is the crawler that decides whether ChatGPT search can cite you, and it is refused on 8% of sites that explicitly allow it. Those sites have paid for a bot-management rule, or flipped a "block AI bots" switch, that also closes the door they wanted open.
Second, the 429s are a judgement call, which is why they get their own column. A 429 is "slow down", not "go away". But a crawler that is rate-limited on the very first request to the homepage, from one IP, is for practical purposes blocked, so the headline keeps them and the third column shows the figure without them.
The mirror image exists too: for GPTBot, 53 sites disallow it in robots.txt and serve it anyway. robots.txt is advisory in both directions.
What robots.txt itself says
- 82.7% of reachable sites have a robots.txt at all.
- 25.1% block at least one search or answer crawler, the ones that fetch pages to cite them. 10.2% block the majority.
- 26.0% block at least one training crawler. Only 2.6% do the thing the guides recommend: block training crawlers only, and keep the search and answer crawlers open.
- 2.5% carry a Content-Signal directive. 62.4% list a sitemap.
The emerging files, honestly counted
Adoption numbers for llms.txt get quoted a lot, and most of them are inflated by soft 404s: sites that answer 200 with an HTML "not found" page for any path. We fetched every reported llms.txt again and kept only the ones that are a text file beginning with a Markdown heading.
| File | Any 2xx answer | A real file |
|---|---|---|
| llms.txt | 34.3% | 15.1% |
| llms-full.txt | 21.9% | not re-verified |
| agents.json | 0.3% | |
| A markdown copy of the page (Accept: text/markdown or .md) | 3.5% served, 1.5% linked | |
| WebMCP form annotations | 1 site of the 254 with forms |
So the real llms.txt number for the top 1,000 is about one site in seven, not one in three. If you are building anything that reads these files, test for the soft 404 first: check the content type, and check that the first line is a heading.
Readiness overall
Scored with our checker (v2, which counts only the crawlers that fetch pages for AI search and answers, and treats a deliberate block as a valid choice rather than a failure): mean 84.5, median 90. 355 sites are AI Ready, 106 are Restricted by Design, 88 are JavaScript-Dependent (the crawler is allowed in, but the content renders client-side, which most crawlers do not run), and 126 could only be partly checked because they blocked our checker too.
Test your own site
The quickest check is from a terminal. If the second command gets a 403 and the first a 200, your robots.txt is not the whole story:
curl -sI -A "Mozilla/5.0" https://example.com/ | head -1
curl -sI -A "OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com/ | head -1
The study's parity script is a single dependency-free Node file. Point it at any CSV of domains:
curl -O https://www.crawlreadiness.com/research/ai-crawler-access-2026-10/parity.mjs
echo "example.com" > list.csv
node parity.mjs list.csv
And if you want the full readiness check with the fix list, the free checker at crawlreadiness.com runs the same code the study ran, and each of the 51 crawlers has a reference page with its documented user agent, its purpose, whether it honours robots.txt, and the exact robots.txt lines to allow or block it: https://www.crawlreadiness.com/crawlers
What to do if you are in the 14%
- Look at the bot-management layer before robots.txt. On Cloudflare that is Security → Bots, where "Block AI bots" blocks search and answer crawlers along with training crawlers. On other CDNs and WAFs, look for a managed rule that matches the user-agent strings above.
- Decide per purpose, not per vendor. Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a legitimate and common choice; the 2.6% of top sites that do it carefully show it works.
- Verify by request, not by reading the file. The curl pair above takes ten seconds.
Data and reproduction
- Study page, with definitions and the per-crawler table: https://www.crawlreadiness.com/research/ai-crawler-access-october-2026
- results.csv (one row per site), crawlers.csv (one row per crawler token), summary.json and parity.mjs, all CC BY 4.0, linked from the study page.
- We will re-run it monthly and keep the earlier runs published, so the trend becomes visible.
If you find a site we have misclassified, the per-site row shows every status code we got; open an issue or write to support@crawlreadiness.com.
Top comments (1)
We need to produce a short comment, one sentence or up to two, casual, start with lowercase, mention specific reaction or question about this video. No promotional content. Should be something like "noticed that a lot of those sites had CDN rules that overrode robots.txt, any idea why they do that?" Must follow guidelines: no quotes? The comment text only. No punctuation issues. Avoid double hyphens. Use straight ASCII quotes if needed. Probably no need for quotes. We'll produce something like: "lol surprised to see