Two posts ago I told people to check their server logs for AI crawlers getting a 403. @wrencalloway replied with the obvious next question: fine, you found the block - now how do you let the right crawlers through without letting everyone in?
This is that post - and honestly, he co-wrote it. The three-gate model, the "identity isn't intent" framing, the rotation problem that breaks IP allowlists: those are his, pulled straight from our thread. I did the write-up. The thinking was a two-person job. Credited throughout.
Upfront, so nobody mistakes this for a war story: I haven't shipped this on my own edge yet. It's research toward doing it properly. If you run this in production, the comments are where I learn something.
One question that's actually two
You want AI crawlers to reach your content - that's how you get cited in ChatGPT, Perplexity and Gemini. You don't want every scraper on the internet walking in behind them.
"Let the good ones in, keep the bad ones out" sounds like one job. It's two, and they pull against each other. Open the door wider for GPTBot and you've opened it for anything willing to type GPTBot into a header. Close it and you vanish from the answers you were trying to appear in.
The whole problem is identity: how do you know a request is really from the crawler it claims to be? Everything below is three increasingly honest answers to that.
Answer 1 - the name (and why it's worthless)
The lazy fix: allow anything whose User-Agent contains GPTBot.
A User-Agent is a string. Anyone can type it. A scraper labels itself GPTBot, sails through, and your gesture toward AI visibility is now an open door for exactly the traffic you were keeping out.
The name isn't identity. It's a claim. You need something the visitor can't fake.
Answer 2 - the origin (IP)
Better: check where the request actually came from. Done right, it's a two-step DNS dance, not a flat IP lookup:
- Reverse-DNS the requesting IP → get a hostname
- Confirm the hostname belongs to the vendor -
*.crawl.openai.com,*.anthropic.com,*.perplexity.ai,*.googlebot.com - Forward-resolve that hostname → it must return the original IP
Forward-confirm reverse DNS. A scraper can claim a hostname; it can't make the vendor's DNS resolve back to its IP.
A scraper can claim a hostname. It can't make the vendor's DNS forward-resolve to its own IP. Both directions have to agree.
Some vendors also publish raw ranges to match against:
- OpenAI - gptbot.json, searchbot.json, chatgpt-user.json
- Perplexity - perplexitybot.json, perplexity-user.json
- Google - crawler ranges
Where origin breaks - three cracks, none small
Not everyone publishes. This one surprised me: Anthropic does not publish IP ranges for ClaudeBot. The IPs in their docs are for the Claude API, not the crawler. Their own guidance for identifying ClaudeBot is User-Agent plus reverse-DNS against *.anthropic.com. So for one of the three biggest crawlers, "just match the published range" isn't an option that exists.
Google won't separate training traffic by IP. Google-Extended - the control for Gemini and AI Overviews training - is a robots.txt token, not an IP set. There's no google-extended.json. You can allow or deny it in robots.txt, but you cannot distinguish it at the network layer. IP verification doesn't reach it at all.
Rotation. This is the crack @wrencalloway named, and it's the sharpest:
A stale allowlist and a 20-minute 403 window are both real costs, and "just poll the JSON more often" is a band-aid on a design that shouldn't have required polling.
Every published range is a snapshot of a moving target. The vendor rotates, or ships a new block late, and you choose between an allowlist that's already wrong and a window where legitimate crawlers eat a 403. You're not verifying identity anymore. You're babysitting CIDR blocks and hoping the JSON is current.
Answer 3 - the proof (Web Bot Auth)
This is where the trust model changes shape instead of just getting fussier.
Web Bot Auth - a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421 HTTP Message Signatures) - has each crawler sign its requests with an Ed25519 key. The public key lives at a well-known directory on the vendor's own domain. You verify the signature. Identity is proven by cryptography, not inferred from where the packet came from.
The shift is the entire point: from "where did this connection originate" to "can you prove you hold the key." That survives rotation. It doesn't care which datacenter the request left. No CIDR to babysit, no JSON to poll.
It's real and moving fast:
- Integrated into Cloudflare's Verified Bots program, with a Signed Agents directory
- ChatGPT's agent was in the first signed cohort, 2025
- AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents
Status, honestly: an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.
Where proof breaks
It only helps for crawlers that actually sign. And the scrapers you most want to block are exactly the ones who'll never adopt it. Signatures verify the honest. They do nothing to the dishonest.
So why not just wait for signatures to win?
Because signatures answer a narrower question than it looks. Here's the reframing @wrencalloway landed on, and it's the sharpest thing in either of our threads:
Signatures move the question from "who are you" to "what are you doing."
Sit with that. A valid signature proves identity, not intent. A legitimate crawler having a bad day and a stolen key behave identically at the signature layer - both present a valid signature. Verification hands you a name. It says nothing about whether the holder of that name is currently hammering your origin at 400 requests a second.
Which means there is always a floor under everything else: behaviour. Rate limits, request patterns, what a client reaches for once it's through the door. Verification gets you a name. Behaviour decides whether the name gets to keep acting like one.
The model: three gates, not one
Put the three answers in order and they stop competing. They stack.
- Signed? Verify the signature → fast-path it. (Web Bot Auth - proof)
- Not signed, but claims a known crawler? Reverse-DNS + forward-confirm, or a published range. (IP - origin)
- Through either gate - now watch it. Rate limits and behaviour, always on, for everyone. (the floor)
Three gates, not one: proof, then origin, then an always-on behaviour floor. Identity proves who; only behaviour proves what.
@wrencalloway's word for this was additive, not a replacement, and once you see the gates it's obvious why. Signatures don't replace the IP layer - they shrink its blast radius. Every honest crawler that signs steps out of the pool you're forced to judge by circumstance. What's left for the IP-and-vibes gate is smaller and dirtier - a population that's mostly bad actors, which is a much easier one to be aggressive with. And behind both, behaviour never stops watching, because identity was never the same thing as intent.
Spotting a fake while the world catches up
Until signatures are universal, most of the work is catching User-Agent liars. The tells are consistent:
- Reverse-resolves to a general host (Hetzner, OVH, DigitalOcean) with no vendor match
- NXDOMAIN - no reverse-DNS at all
- Goes straight for
/admin,/.env,/wp-login- real crawlers read content, not secrets - Hundreds of requests a minute, no Crawl-Delay
- A typo in the name:
GPT-Bot, or a version that doesn't exist (ClaudeBot/2.x)
None is proof alone. Together they're a scraper wearing a badge.
The rule I took away
The whole progression is one idea getting more honest with itself:
- The name asks the visitor to identify themselves. Trusts a string.
- The origin asks where they came from. Trusts an address that rotates or goes unpublished.
- The proof asks them to prove they hold a key. Trusts math - but only about who, never what.
Each gate moves trust closer to something the visitor can't fake. None of them is the finish line, because the finish line doesn't exist: even perfect identity tells you nothing about intent. You verify by proof where you can, by origin where you can't, and you hold the line with behaviour for everyone - because a name was never a promise of good behaviour.
I haven't built this yet. When I do, it's signatures first, IP as fallback, behaviour underneath - and I'll write up what broke, because something always does. If you're already running it, tell me what I'm underestimating. Last time that's exactly how this post got written.
Related reading:
Top comments (3)
I particularly appreciated the breakdown of the three-gate model and the discussion around verifying crawler identity through IP checks and DNS resolution. The point about Anthropic not publishing IP ranges for ClaudeBot highlights the complexities of implementing this approach. I've found that using a combination of User-Agent verification and IP checks can be effective, but it's crucial to stay up-to-date with the latest vendor-published ranges and adapt to changes in their authentication methods. Have you considered exploring alternative authentication methods, such as cryptographic signatures or token-based authentication, to further enhance crawler verification?
Thanks @topstar_ai ! Cryptographic signatures are actually the third section of the post - Web Bot Auth, the Cloudflare-led IETF draft that has crawlers sign each request with an Ed25519 key. That's exactly the "prove you hold the key" layer that fixes the rotation problem you mentioned: it survives IP changes because it doesn't depend on where the request came from.
The catch I get into further down: signatures only help for crawlers that actually sign, and the scrapers you most want to block never will. So it ends up additive to IP checks, not a replacement - which is why the model has three gates instead of one.
Token-based auth is interesting but I haven't seen a vendor adopt it for crawling at scale - the industry seems to be converging on Web Bot Auth instead. Have you seen token approaches used anywhere in production?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.