19% of the top 2,000 sites block AI crawlers — and 136 more block them without knowing

Mira Voss — Thu, 06 Aug 2026 18:28:07 +0000

19% of the top 2,000 sites block AI crawlers

On 2 August 2026 I scanned the top 2,000 domains from the Tranco top-1M list and
asked one question of each: can an AI crawler read this site?

Not "should it" — that's a business decision every site gets to make. Just:
what does the site actually do when GPTBot, ClaudeBot, PerplexityBot, or
Google-Extended shows up.

Here is what came back.

The headline

Of the 1,530 domains with a reachable robots.txt, 292 (19.1%) disallow at
least one of the four major AI crawlers.

Crawler	Blocked by	Share
GPTBot	254	16.6%
ClaudeBot	241	15.8%
Google-Extended	227	14.8%
PerplexityBot	165	10.8%

The blockers are exactly who you would guess, and they are not shy about it:
instagram.com (#11), twitter.com (#16), amazon.com (#25), pinterest.com (#51),
x.com (#53), yahoo.com (#59), msn.com (#60), tiktok.com (#61). Social and
commerce platforms whose content is the product.

Blocking is nearly all-or-nothing

I expected nuance — sites allowing the search-adjacent crawlers and blocking the
training-adjacent ones. That is not what the data shows.

Blocks	Sites
all 4	119
3 of 4	100
2 of 4	38
exactly 1	35

75% of blockers block three or more. The decision is being made once, at the
policy level — "AI crawlers, no" — not crawler by crawler.

When a site is selective, one name comes up more than the rest. Of the 35
sites that block exactly one crawler, 19 single out GPTBot — more than
ClaudeBot (8), PerplexityBot (5) and Google-Extended (3) combined. GPTBot is
the one people have heard of, so it is the one that ends up in the file.

The finding that surprised me: rank does not predict blocking

Top 200 domains: 19.0% block at least one crawler.
Everything else in the sample: 19.1%.

There is no meaningful difference. I expected a strong gradient — big publishers
with legal departments blocking, the long tail leaving defaults alone. The data
says blocking is driven by category, not by size. A mid-ranked marketplace
behaves like a top-ranked one; a mid-ranked docs site behaves like a top-ranked
docs site.

Null results are worth publishing. This one kills a plausible assumption.

The gap: 136 sites allow in robots.txt and appear to block at the edge

This is the part worth your attention.

136 domains (8.9% of those with a reachable robots.txt) explicitly allow all
four crawlers in robots.txt, while the homepage probe suggests the request is
being blocked before it ever reaches that policy. WAF rules, bot-management
products, and CDN defaults do not read your robots.txt.

Among them: nytimes.com (#152), cisco.com (#243), ibm.com (#275),
salesforce.com (#288), sourceforge.net (#227), unity3d.com (#120),
launchpad.net (#270).

I want to be precise about what this is and is not. It is a user-agent probe
from a single vantage point, not proof. Rate limiting, geography, and transient
edge behaviour can all produce the same signal. But the direction is clear
enough to matter: for a meaningful slice of the web, robots.txt and reality
disagree, and the site owner is the last to find out.

If you have ever written a permissive robots.txt and assumed the job was
done — that assumption is worth ten minutes of verification.

llms.txt: present on 9.4%, and it correlates with openness

188 of the 2,000 domains scanned (9.4%) serve an llms.txt. llms.txt was
fetched independently of robots.txt, so this percentage is of the full sample,
not the reachable subset. It is far higher than the general web, which is what
you would expect at the top of the ranking.

More interesting is the correlation. Restricting to the 1,530 domains where
blocking can be determined at all:

Sites with llms.txt: 8.0% block at least one crawler (15 of 188)
Sites without: 20.6% (277 of 1,342)

Sites that publish llms.txt are roughly 2.6× less likely to block AI
crawlers. That is not causation — nobody adds a file and thereby changes their
legal posture. It reads as a marker: teams that have thought about AI access
tend to have decided to allow it, and the file is the visible trace of that
decision.

Practical implication, though: lead with robots.txt, not llms.txt.
llms.txt is an emerging convention with limited consumption today. robots.txt
is read by every crawler in this study, right now. Fix the one that is load
bearing first.

Method, and its limits

Source list: Tranco top-1M, snapshot 2026-07-13. Top 2,000 taken by rank.
What was fetched: robots.txt, llms.txt, and one homepage request with each crawler's user-agent string. Read-only. Each host contacted once per run.
Scanner: the same public CLI used for single-site checks, 16 workers, 12.4 minutes wall clock.
Reachability: 1,530 of 2,000 returned a usable robots.txt. The other 470 are unreachable, redirected, or served something that is not a robots file — they are excluded from percentages rather than counted as "allow".
What "blocked" means: an explicit Disallow matching the crawler's user-agent. robots.txt is a request, not an enforcement mechanism.
What "edge block suspected" means: the homepage probe failed or was refused for a crawler user-agent while robots.txt allowed it. Heuristic, single vantage point, no retry across geographies.

One snapshot, one day, 2,000 domains. Treat every number here as a measurement
with a date on it, not a law.

What is next

This is month one of a monthly series. A single census is a statistic; twelve
are a time series, and a time series is the only way to answer the question
people actually have: is AI blocking increasing or decreasing?

The second data point lands in September. I will publish the delta whether it
confirms this snapshot or embarrasses it.

Check your own site

If you take one thing from this: the 8.9% gap means a permissive robots.txt is
not evidence that AI crawlers can read you. The only way to know is to look from
the outside.

The scanner used for this census is a single open-source Python file with no
dependencies beyond the standard library. It runs locally — nothing to sign up
for, no email wall, no data leaves your machine:

https://github.com/STARS-BIT-ux/ai-crawler-check

python ai_crawler_check.py yourdomain.com

It checks the four crawlers above plus CCBot (Common Crawl, which feeds most
open datasets), reads your robots.txt and llms.txt, and prints what each
user-agent actually gets back. Same fields as this census, so your result is
directly comparable to the numbers above.

If it disagrees with your robots.txt, you have just found something worth
fixing before your next content push.

Data collected 2026-08-02. Raw per-domain results and the run manifest are
available on request — the manifest records the list snapshot, worker count,
scanner version and completion time, so the run can be audited or reproduced.

DEV Community: Mira Voss