Everyone reads the headline as "Cloudflare blocks AI crawlers." The measurement says something
narrower and more interesting: on the sites whose own robots.txt says yes, Cloudflare refuses
Amazon's and Meta's crawlers far more often than it refuses OpenAI's or Anthropic's — and for
OAI-SearchBot it refuses slightly less often than sites with no CDN at all.
All figures re-derived from the live Agent Web Index
aggregate on 2026-09-19: 47,315 domains from the Tranco list with a measured verdict, 46,868 of them
with the edge vendor read from the response headers. Dataset CC BY 4.0 (CSV on the hub).
What is being counted
One unit: a domain × crawler pair whose robots.txt allows that crawler. For each pair we made
a real GET with that crawler's published user-agent. If robots.txt says yes and the server says no
— 403, challenge, block page — that refusal contradicts the site's own stated policy, and somebody
other than the site owner made it.
The edge vendor comes from the headers of the same response (cf-ray, akamai-grn, x-amz-cf-id,
x-fastly-request-id…), so it costs zero extra requests. Sites with no recognisable signature form
the control group: "no known edge", 12,384 domains.
Cloudflare, per crawler
26,543 domains behind Cloudflare. 13.1% of the 119,112 robots-allowed pairs are refused anyway. But
that single number hides the whole story:
| Crawler | Cloudflare | No known edge | Ratio |
|---|---|---|---|
| Amazonbot | 29.9% (2,115/7,062) | 13.8% (922/6,677) | 2.17× |
| Meta-ExternalAgent | 20.7% (1,471/7,120) | 12.4% (832/6,727) | 1.67× |
| Bytespider | 26.1% (80/306) | 20.1% (37/184) | 1.30× |
| GPTBot | 14.0% (3,593/25,736) | 11.9% (1,357/11,448) | 1.18× |
| ClaudeBot | 14.6% (3,768/25,873) | 13.0% (1,511/11,644) | 1.12× |
| Applebot | 9.1% (29/320) | 7.3% (14/192) | 1.24× |
| PerplexityBot | 8.8% (2,314/26,302) | 8.2% (973/11,885) | 1.07× |
| OAI-SearchBot | 8.5% (2,239/26,393) | 9.0% (1,087/12,040) | 0.94× |
Read the right-hand column, not the left. Sites block AI crawlers for their own reasons everywhere,
CDN or not; the ratio isolates what being behind Cloudflare adds. For ChatGPT's search crawler it
adds nothing. For Anthropic and Perplexity it adds a few percent. For Amazon's crawler it roughly
doubles the chance of being turned away from a page the site said it could have.
The shape tells you whether a human decided
Run the same breakdown on the other vendors and the contrast is the finding:
| Edge | Domains | Refusal rate | Widest gap between crawlers | Every crawler, vs control |
|---|---|---|---|---|
| Akamai | 327 | 42.2% | 1.1× | 3.2–4.9× |
| Google (Cloud/GFE) | 763 | 36.6% | 1.3× | 2.2–4.4× |
| Sucuri | 61 | 21.4% | 21.0× | 0.2–3.3× |
| AWS CloudFront | 3,111 | 16.5% | 1.2× | 1.3–1.8× |
| Cloudflare | 26,543 | 13.1% | 3.5× | 0.9–2.2× |
| Fastly | 1,454 | 8.3% | 1.7× | 0.6–0.8× |
| Vercel | 570 | 5.0% | 1.4× | 0.4–0.6× |
| Netlify | 197 | 2.3% | 3.2× | 0.1–0.3× |
Read the last two columns together. Akamai refuses far more traffic than Cloudflare — and refuses
every AI crawler at almost exactly the same rate, roughly three to five times the control on all of
them. That is a wall nobody aimed: a bot-management default that never learned these user-agents
apart, and it is by far the most punishing thing that can sit in front of a site that said yes.
Cloudflare refuses less overall, but selectively, and selectivity is the signature of a list — a
set of names, maintained by someone, applied on behalf of 26,543 sites that mostly never opened the
setting.
Sucuri is the cleanest example of a list there is: ClaudeBot 35%, GPTBot 29%, OAI-SearchBot 30%,
PerplexityBot 2%. Twenty-one to one, across 61 domains whose own robots.txt allows all four.
Nobody writes that shape by accident.
And the bottom of the table is a finding in its own right: behind Fastly, Vercel, Netlify, Imperva,
Azure Front Door or Varnish, a robots-allowed crawler is refused less often than on a site with no
recognisable edge at all — down to 0.1× for Netlify. The edge is not one thing. Two vendors apart on
this table differ more, for a site owner who wanted to be readable, than most of the choices that
owner actually agonises over.
Why this is hard to see
You cannot get these numbers from robots.txt, and you cannot get them from a CDN's own
transparency page, which only covers its own customers and never compares them to a control. You
have to make the request as each crawler, from outside, at scale, and keep the response headers. The
index does exactly that, continuously; every cell above is a live aggregate, not a one-off study.
The per-vendor × per-crawler table is on the hub at
/ai-readiness/#edge, with the pair count
behind every cell. There is also an MCP endpoint, no key, if you would rather have your assistant
pull it: POST https://shop.lumnika.com/ai-readiness/mcp, tool edge_blocking.
Limits, stated first
- One observation point (Europe) and the homepage only. A vendor whose rules vary by region or by path will read differently from elsewhere.
- Columns do not share a denominator. Amazonbot and Meta-ExternalAgent were added to the probe later, so their columns cover the domains measured since — a more recent slice of the same list, not a different list. Compare within a row first.
- Small vendors, small denominators. Sucuri's 21× gap rests on 321 robots-allowed pairs across 61 domains, Akamai's rates on 1,582 pairs across 327. The direction is unambiguous at that size; the second decimal is not. Cloudflare (119,112 pairs) is the only cell where precision is cheap.
- "No known edge" is an upper bound, not a vendor. A domain measured before a signature was added to the table stays in that bucket until it is re-requested; a re-request of 250 of them found 14% already carrying a recognised signature. So the named vendors are undercounts.
- For part of the domains the edge was read in a later pass than the crawler verdicts, so a site that changed CDN in between appears under its current one.
-
Google-ExtendedandApplebot-Extendedare robots.txt tokens, not crawlers — they make no HTTP request of their own and are excluded from every number here about server behaviour.
If your site is in the table and you did not choose this, the per-domain page shows exactly which
crawler got which answer: https://shop.lumnika.com/ai-readiness/<your-domain>.
Top comments (0)