This started as a comment @wrencalloway left on my last post. It was sharp enough that it deserved more than a reply — so here's the full version.
You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs.
Except the crawler never fetches your URL. It asks the CDN, and the CDN says 403.
Your content is perfect and unreachable. This is the layer underneath the one everyone talks about — and it's invisible in every tool you'd normally reach for.
Two different kinds of "no"
robots.txt is a note taped to the door. It says "please don't come in." A polite crawler reads it and turns around. A rude one ignores it and walks straight past. Either way, the note never touches your bytes — it's a request, enforced entirely by the visitor's own manners.
A WAF or CDN rule is the door. It answers the request itself, at the edge, before anything reaches your origin: 403 Forbidden, 429 Too Many Requests, or a JavaScript challenge the bot can't solve. The HTML behind that door could be a masterpiece or a blank page. The bot sees neither. It sees the status code.
People spend weeks perfecting the note and never check whether the door is locked.
This isn't hypothetical — here's the data
Cloudflare Radar publishes what actually happens to AI-crawler traffic across its network. From the AI Insights dashboard (7-day view, as of 18 July 2026):
Of all HTTP responses served to AI bots and crawlers:
- 200 OK — 73.6%
- 403 Forbidden — 5.2%
- 429 Too Many Requests — 1.3%
- 503 — 1.1%
AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026.
Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused.
And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a bot-fight score set too high. The same dashboard's robots.txt tracker shows GPTBot, CCBot and ClaudeBot as the most-disallowed crawlers across the top domains — much of it inherited, not authored.
Most-disallowed AI user agents across top domains. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.
Why you will never notice on your own
Open your site. It loads. Of course it does — your browser isn't on the blocklist. Residential IP, Chrome User-Agent, you pass every check.
Open robots.txt. Allow: /. Looks perfectly welcoming.
View source. Clean HTML, structured data, the works. (You already fixed that part.)
Every tool you'd reach for confirms you're fine — because every one of them comes from you, not from GPTBot. The block is aimed at a class of visitor you are never going to be.
The only place it shows: server logs
The browser lies, because the browser is you. robots.txt lies, because it's advisory. The access log doesn't lie. It has one row per real request, with the visitor's User-Agent and the exact status you returned. It's the only record of what GPTBot actually got.
Grep for the ones that matter and count their status codes:
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Google-Extended" access.log \
| awk '{print $9}' \
| sort | uniq -c
200 means it got the page. 403 means the edge refused it. 429 means you rate-limited it into silence. The failure mode nobody catches looks like this: robots.txt wide open, logs full of 403s. Everything you can see says yes; the thing that decides says no.
I've read this exact story more than once. Months of content, zero citations in ChatGPT or Perplexity, and an audit that finally finds Cloudflare had been handing GPTBot a 403 the entire time.
The trap inside the fix
The obvious repair — allow anything whose User-Agent contains GPTBot — is wrong, because a User-Agent is a string anyone can type. A price scraper labels itself GPTBot and strolls through your generous new rule.
Cloudflare's own transparency tracking makes the point: it flags which operators can even be verified. As of mid-2026, ByteDance shows as unverified — which is exactly why User-Agent alone can't be trusted.
AI bot transparency tracking. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.
So real verification isn't the name. It's the origin. Two ways:
1. Verify by published IP ranges. OpenAI, Anthropic and the rest publish their crawler IPs — openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and their equivalents. Reverse-DNS the requesting IP, confirm the hostname belongs to the vendor, then forward-resolve it back to the same IP. If either step fails, it's a spoofer wearing the name.
2. Verify by cryptographic signature. This is where it's heading. Web Bot Auth — a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421) — has crawlers sign each request with an Ed25519 key, so identity is proven, not claimed. It's integrated into Cloudflare's Verified Bots program with a dedicated Signed Agents directory, and ChatGPT's agent was in the first signed cohort in 2025. It isn't only Cloudflare, either — AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents.
Note the status honestly: Web Bot Auth is an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.
The point for you: "block the bad bots" and "let the AI in" are the same problem, and the User-Agent solves neither. Verify by origin or by signature — otherwise you block the crawler you want and admit the one you don't.
What to actually do
Check the logs, not the browser. Pull a week of access logs, filter to the AI User-Agents, look at the status codes. That's the whole audit — an afternoon, not a project.
If you see 403/429, find the rule. It's almost always a managed ruleset, a "block AI bots" setting, or an over-eager bot-fight score — not a line you wrote. Allowlist the crawlers you want by verified IP or Web Bot Auth, never by User-Agent alone.
The rule I took away
Your HTML is the last thing in the request's path, not the first. Before a bot reads a single byte of it, the request has to clear DNS, the CDN, the WAF, the rate limiter, the bot-management score. Any one of them can end the request with a status code you never see — on a page that renders flawlessly for you.
"It works in my browser" was always a weak claim. For crawlers it isn't even the right question. The right question is: what status code did the bot get? And the only honest answer is in the logs.
Related reading:
Top comments (2)
ngl i always just assume it's a rendering issue, never thought about the cdn blocking the bot entirely. makes so much sense though.
Genuinely made my day to see the comment turned into this. You have done great work here, thank you.