DEV Community: Evgenii Slepinin

Verify the crawler, not its name: Web Bot Auth, IP checks, and the third gate

Evgenii Slepinin — Sun, 19 Jul 2026 13:27:03 +0000

Two posts ago I told people to check their server logs for AI crawlers getting a 403. @wrencalloway replied with the obvious next question: fine, you found the block - now how do you let the right crawlers through without letting everyone in?

This is that post - and honestly, he co-wrote it. The three-gate model, the "identity isn't intent" framing, the rotation problem that breaks IP allowlists: those are his, pulled straight from our thread. I did the write-up. The thinking was a two-person job. Credited throughout.

Upfront, so nobody mistakes this for a war story: I haven't shipped this on my own edge yet. It's research toward doing it properly. If you run this in production, the comments are where I learn something.

One question that's actually two

You want AI crawlers to reach your content - that's how you get cited in ChatGPT, Perplexity and Gemini. You don't want every scraper on the internet walking in behind them.

"Let the good ones in, keep the bad ones out" sounds like one job. It's two, and they pull against each other. Open the door wider for GPTBot and you've opened it for anything willing to type GPTBot into a header. Close it and you vanish from the answers you were trying to appear in.

The whole problem is identity: how do you know a request is really from the crawler it claims to be? Everything below is three increasingly honest answers to that.

Answer 1 - the name (and why it's worthless)

The lazy fix: allow anything whose User-Agent contains GPTBot.

A User-Agent is a string. Anyone can type it. A scraper labels itself GPTBot, sails through, and your gesture toward AI visibility is now an open door for exactly the traffic you were keeping out.

The name isn't identity. It's a claim. You need something the visitor can't fake.

Answer 2 - the origin (IP)

Better: check where the request actually came from. Done right, it's a two-step DNS dance, not a flat IP lookup:

Reverse-DNS the requesting IP → get a hostname
Confirm the hostname belongs to the vendor - *.crawl.openai.com, *.anthropic.com, *.perplexity.ai, *.googlebot.com
Forward-resolve that hostname → it must return the original IP

Forward-confirm reverse DNS. A scraper can claim a hostname; it can't make the vendor's DNS resolve back to its IP.

A scraper can claim a hostname. It can't make the vendor's DNS forward-resolve to its own IP. Both directions have to agree.

Some vendors also publish raw ranges to match against:

OpenAI - gptbot.json, searchbot.json, chatgpt-user.json
Perplexity - perplexitybot.json, perplexity-user.json
Google - crawler ranges

Where origin breaks - three cracks, none small

Not everyone publishes. This one surprised me: Anthropic does not publish IP ranges for ClaudeBot. The IPs in their docs are for the Claude API, not the crawler. Their own guidance for identifying ClaudeBot is User-Agent plus reverse-DNS against *.anthropic.com. So for one of the three biggest crawlers, "just match the published range" isn't an option that exists.

Google won't separate training traffic by IP. Google-Extended - the control for Gemini and AI Overviews training - is a robots.txt token, not an IP set. There's no google-extended.json. You can allow or deny it in robots.txt, but you cannot distinguish it at the network layer. IP verification doesn't reach it at all.

Rotation. This is the crack @wrencalloway named, and it's the sharpest:

A stale allowlist and a 20-minute 403 window are both real costs, and "just poll the JSON more often" is a band-aid on a design that shouldn't have required polling.

Every published range is a snapshot of a moving target. The vendor rotates, or ships a new block late, and you choose between an allowlist that's already wrong and a window where legitimate crawlers eat a 403. You're not verifying identity anymore. You're babysitting CIDR blocks and hoping the JSON is current.

Answer 3 - the proof (Web Bot Auth)

This is where the trust model changes shape instead of just getting fussier.

Web Bot Auth - a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421 HTTP Message Signatures) - has each crawler sign its requests with an Ed25519 key. The public key lives at a well-known directory on the vendor's own domain. You verify the signature. Identity is proven by cryptography, not inferred from where the packet came from.

The shift is the entire point: from "where did this connection originate" to "can you prove you hold the key." That survives rotation. It doesn't care which datacenter the request left. No CIDR to babysit, no JSON to poll.

It's real and moving fast:

Integrated into Cloudflare's Verified Bots program, with a Signed Agents directory
ChatGPT's agent was in the first signed cohort, 2025
AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents

Status, honestly: an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.

Where proof breaks

It only helps for crawlers that actually sign. And the scrapers you most want to block are exactly the ones who'll never adopt it. Signatures verify the honest. They do nothing to the dishonest.

So why not just wait for signatures to win?

Because signatures answer a narrower question than it looks. Here's the reframing @wrencalloway landed on, and it's the sharpest thing in either of our threads:

Signatures move the question from "who are you" to "what are you doing."

Sit with that. A valid signature proves identity, not intent. A legitimate crawler having a bad day and a stolen key behave identically at the signature layer - both present a valid signature. Verification hands you a name. It says nothing about whether the holder of that name is currently hammering your origin at 400 requests a second.

Which means there is always a floor under everything else: behaviour. Rate limits, request patterns, what a client reaches for once it's through the door. Verification gets you a name. Behaviour decides whether the name gets to keep acting like one.

The model: three gates, not one

Put the three answers in order and they stop competing. They stack.

Signed? Verify the signature → fast-path it. (Web Bot Auth - proof)
Not signed, but claims a known crawler? Reverse-DNS + forward-confirm, or a published range. (IP - origin)
Through either gate - now watch it. Rate limits and behaviour, always on, for everyone. (the floor)

Three gates, not one: proof, then origin, then an always-on behaviour floor. Identity proves who; only behaviour proves what.

@wrencalloway's word for this was additive, not a replacement, and once you see the gates it's obvious why. Signatures don't replace the IP layer - they shrink its blast radius. Every honest crawler that signs steps out of the pool you're forced to judge by circumstance. What's left for the IP-and-vibes gate is smaller and dirtier - a population that's mostly bad actors, which is a much easier one to be aggressive with. And behind both, behaviour never stops watching, because identity was never the same thing as intent.

Spotting a fake while the world catches up

Until signatures are universal, most of the work is catching User-Agent liars. The tells are consistent:

Reverse-resolves to a general host (Hetzner, OVH, DigitalOcean) with no vendor match
NXDOMAIN - no reverse-DNS at all
Goes straight for /admin, /.env, /wp-login - real crawlers read content, not secrets
Hundreds of requests a minute, no Crawl-Delay
A typo in the name: GPT-Bot, or a version that doesn't exist (ClaudeBot/2.x)

None is proof alone. Together they're a scraper wearing a badge.

The rule I took away

The whole progression is one idea getting more honest with itself:

The name asks the visitor to identify themselves. Trusts a string.
The origin asks where they came from. Trusts an address that rotates or goes unpublished.
The proof asks them to prove they hold a key. Trusts math - but only about who, never what.

Each gate moves trust closer to something the visitor can't fake. None of them is the finish line, because the finish line doesn't exist: even perfect identity tells you nothing about intent. You verify by proof where you can, by origin where you can't, and you hold the line with behaviour for everyone - because a name was never a promise of good behaviour.

I haven't built this yet. When I do, it's signatures first, IP as fallback, behaviour underneath - and I'll write up what broke, because something always does. If you're already running it, tell me what I'm underestimating. Last time that's exactly how this post got written.

Related reading:

Your HTML is fine. The CDN still blocks the bot.

Evgenii Slepinin — Sat, 18 Jul 2026 18:22:55 +0000

This started as a comment @wrencalloway left on my last post. It was sharp enough that it deserved more than a reply — so here's the full version.

You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs.

Except the crawler never fetches your URL. It asks the CDN, and the CDN says 403.

Your content is perfect and unreachable. This is the layer underneath the one everyone talks about — and it's invisible in every tool you'd normally reach for.

Two different kinds of "no"

robots.txt is a note taped to the door. It says "please don't come in." A polite crawler reads it and turns around. A rude one ignores it and walks straight past. Either way, the note never touches your bytes — it's a request, enforced entirely by the visitor's own manners.

A WAF or CDN rule is the door. It answers the request itself, at the edge, before anything reaches your origin: 403 Forbidden, 429 Too Many Requests, or a JavaScript challenge the bot can't solve. The HTML behind that door could be a masterpiece or a blank page. The bot sees neither. It sees the status code.

People spend weeks perfecting the note and never check whether the door is locked.

This isn't hypothetical — here's the data

Cloudflare Radar publishes what actually happens to AI-crawler traffic across its network. From the AI Insights dashboard (7-day view, as of 18 July 2026):

Of all HTTP responses served to AI bots and crawlers:

200 OK — 73.6%
403 Forbidden — 5.2%
429 Too Many Requests — 1.3%
503 — 1.1%

AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026.

Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused.

And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a bot-fight score set too high. The same dashboard's robots.txt tracker shows GPTBot, CCBot and ClaudeBot as the most-disallowed crawlers across the top domains — much of it inherited, not authored.

Most-disallowed AI user agents across top domains. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.

Why you will never notice on your own

Open your site. It loads. Of course it does — your browser isn't on the blocklist. Residential IP, Chrome User-Agent, you pass every check.

Open robots.txt. Allow: /. Looks perfectly welcoming.

View source. Clean HTML, structured data, the works. (You already fixed that part.)

Every tool you'd reach for confirms you're fine — because every one of them comes from you, not from GPTBot. The block is aimed at a class of visitor you are never going to be.

The only place it shows: server logs

The browser lies, because the browser is you. robots.txt lies, because it's advisory. The access log doesn't lie. It has one row per real request, with the visitor's User-Agent and the exact status you returned. It's the only record of what GPTBot actually got.

Grep for the ones that matter and count their status codes:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Google-Extended" access.log \
  | awk '{print $9}' \
  | sort | uniq -c

200 means it got the page. 403 means the edge refused it. 429 means you rate-limited it into silence. The failure mode nobody catches looks like this: robots.txt wide open, logs full of 403s. Everything you can see says yes; the thing that decides says no.

I've read this exact story more than once. Months of content, zero citations in ChatGPT or Perplexity, and an audit that finally finds Cloudflare had been handing GPTBot a 403 the entire time.

The trap inside the fix

The obvious repair — allow anything whose User-Agent contains GPTBot — is wrong, because a User-Agent is a string anyone can type. A price scraper labels itself GPTBot and strolls through your generous new rule.

Cloudflare's own transparency tracking makes the point: it flags which operators can even be verified. As of mid-2026, ByteDance shows as unverified — which is exactly why User-Agent alone can't be trusted.

AI bot transparency tracking. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.

So real verification isn't the name. It's the origin. Two ways:

1. Verify by published IP ranges. OpenAI, Anthropic and the rest publish their crawler IPs — openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and their equivalents. Reverse-DNS the requesting IP, confirm the hostname belongs to the vendor, then forward-resolve it back to the same IP. If either step fails, it's a spoofer wearing the name.

2. Verify by cryptographic signature. This is where it's heading. Web Bot Auth — a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421) — has crawlers sign each request with an Ed25519 key, so identity is proven, not claimed. It's integrated into Cloudflare's Verified Bots program with a dedicated Signed Agents directory, and ChatGPT's agent was in the first signed cohort in 2025. It isn't only Cloudflare, either — AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents.

Note the status honestly: Web Bot Auth is an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.

The point for you: "block the bad bots" and "let the AI in" are the same problem, and the User-Agent solves neither. Verify by origin or by signature — otherwise you block the crawler you want and admit the one you don't.

What to actually do

Check the logs, not the browser. Pull a week of access logs, filter to the AI User-Agents, look at the status codes. That's the whole audit — an afternoon, not a project.

If you see 403/429, find the rule. It's almost always a managed ruleset, a "block AI bots" setting, or an over-eager bot-fight score — not a line you wrote. Allowlist the crawlers you want by verified IP or Web Bot Auth, never by User-Agent alone.

The rule I took away

Your HTML is the last thing in the request's path, not the first. Before a bot reads a single byte of it, the request has to clear DNS, the CDN, the WAF, the rate limiter, the bot-management score. Any one of them can end the request with a status code you never see — on a page that renders flawlessly for you.

"It works in my browser" was always a weak claim. For crawlers it isn't even the right question. The right question is: what status code did the bot get? And the only honest answer is in the logs.

Related reading:

RAG is for finding. Full context is for deciding.

Evgenii Slepinin — Fri, 17 Jul 2026 20:39:39 +0000

The advice in 2026 is settled: chunk your documents, embed them, retrieve top-k, feed those to the model. Don't waste context. Don't waste tokens.

I built a system that deliberately does none of that. Every query gets the expert's entire CV — 69 projects, publications, education, languages. Roughly 34,000 characters, in full, every time.

Here's the reasoning, because it wasn't laziness.

The task decides the architecture

The system finds UN tenders for one specific expert. Every day it crawls 28 sources, normalises and deduplicates thousands of procurement notices, then answers one question per tender: does this person fit?

That looks like retrieval. It isn't. It's exclusion.

Retrieval asks: what in this corpus is most similar to the query?
Exclusion asks: is there anything here that disqualifies this?

Those are opposite operations. The first wants the strongest matches. The second needs the weakest link — and by definition, the weakest link is the thing least similar to everything else.

Why similarity would have failed

A real pattern from the data.

A tender is titled Public Financial Management. The expert has a dozen public-finance projects. Cosine similarity between tender and profile: very high.

# what a RAG pipeline does here
chunks = retrieve(
    query=tender.description,
    corpus=cv_chunks,
    k=3
)
# returns: finance_2019, finance_2021, education_economics
# verdict: strong match

The tender is for an environmental officer. It says so once, on page five, in the responsibilities section.

Or: the role fits perfectly, but requires a national consultant with a mandatory local language. One sentence, buried in eligibility.

The disqualifying detail is never the semantically closest passage. That's exactly what makes it disqualifying — it's the part that doesn't match the pattern. A retrieval system is optimised to not surface it.

What chunking destroys

Split a CV into chunks and you keep the facts but lose the shape.

Retrieve three chunks about public finance and the model sees an expert in public finance. Send all 69 projects and the model sees which countries he's actually worked in, which languages he actually has, what seniority he operates at, and — critically — what he has never done.

Absence is information. And absence cannot be retrieved. There is no chunk that says "this person has no environmental background". You only learn it by reading everything and noticing nothing is there.

RAG can tell you what's in a document. It can't tell you what isn't.

The economics I was told to fear

34,000 characters is roughly 9,000 tokens of context per judgement. Multiplied by hundreds of tenders a day, this is supposed to be the expensive mistake.

It isn't. The AI and the proxies together run on a few dollars a month.

Two reasons:

1. The prefilter does the cheap work first. Stop-words, theme, territory, language, deadline, expired — all in code, no model calls. The AI only ever sees candidates that survived the rules.

# rules run first, model runs last
candidates = [t for t in tenders if passes_rules(t)]
# cheap: no model calls

verdicts = [judge_with_full_cv(t) for t in candidates]
# expensive, but few

2. The CV is byte-identical on every call. Same prefix, over and over — the ideal shape for prompt caching.

The panel tracks cost per run and cost per found tender, so this isn't a guess. I watch the number.

The generalisable lesson: token cost is a real constraint, but it's one people apply from memory rather than measurement. Prices moved. Most RAG pipelines I see were designed for an economics that no longer exists.

What the model returns

Not prose. A structured verdict, logged in full:

{
  "fits": false,
  "short_reason": "national consultant required",
  "long_reason": "Role matches PFM background, but
    eligibility restricts applicants to nationals
    with working Portuguese.",
  "country": "Mozambique",
  "tor_attached": true
}

It judges the substance of the role, not its title. That's the entire point.

When I would use RAG — and do

This is not an argument against RAG. It's an argument against defaulting to it.

RAG is right when:

the corpus is unbounded — a knowledge base that grows forever, docs, support history
latency matters
the task genuinely is retrieval: find the passage, answer from it

I use RAG on other projects for exactly those reasons. Different task, different shape, different answer.

The honest caveat on my side: long context isn't free of problems either. Models attend less reliably to the middle of a long prompt than to its start and end. 34k characters sits comfortably inside what current models handle well. At 300k I'd be having a different conversation, and I'd probably be building a hybrid.

The rule I ended up with

Ask what the model has to do with the document.

Find something → retrieve. The corpus can be enormous, and finding is what embeddings are good at.
Judge something → give it everything. Judgement needs the whole picture, including the parts that aren't relevant, because irrelevance is half the verdict.

Most business tasks people are wiring RAG into are judgement tasks wearing a retrieval costume. Does this candidate fit. Is this contract risky. Should we bid on this. In every one of those, the thing that decides the answer is the detail nobody would have retrieved.

Related reading:

Your page ranks #1 on Google and is invisible to ChatGPT. Both are true.

Evgenii Slepinin — Fri, 17 Jul 2026 19:59:08 +0000

There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.

It's wrong. And it's the single most expensive wrong assumption in technical SEO right now.

The gap

Googlebot renders JavaScript. It has done for years — the page goes into a render queue, headless Chromium executes the scripts, the result gets indexed. Deferred, but it works.

AI crawlers don't render anything.

A server-log study by Vercel and MERJ, published 17 December 2024, found that none of the major AI crawlers execute JavaScript. That includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. A separate analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution.

The detail I find most telling: these crawlers do download JavaScript files. GPTBot in roughly 11.5% of requests, ClaudeBot in roughly 23.84%. They fetch the scripts and never run them.

There's no partial credit here. Either your content is in the initial HTML response, or it isn't.

The one exception is Google Gemini, which rides on Googlebot's Web Rendering Service and inherits its ability to execute JS — along with all its queue delays.

So both things are true at once

Your React site ranks on page one. Google rendered it, indexed it, ranks it.

Then a customer asks ChatGPT the exact question your page answers, and you're not in the response. Nothing is wrong with your rankings. The crawler received an empty shell:

<body>
  <div id="root"></div>
  <script type="module" src="/assets/index-a1b2c3.js"></script>
</body>

You are simultaneously first and invisible.

The trap inside the trap: structured data

This is where it gets specific.

You build a React site. You add react-helmet. You inject JSON-LD through it — Organization, LocalBusiness, FAQPage, the full graph:

<Helmet>
  <script type="application/ld+json">
    {JSON.stringify(schema)}
  </script>
</Helmet>

You open DevTools, expand <head>, and there it is. Beautiful markup. Job done.

It isn't done. DevTools shows you the rendered DOM — the page after JavaScript has run. That is not what a crawler receives.

Open view-source: instead. Not the Elements panel. If your JSON-LD only exists in DevTools and not in view-source, then GPTBot, ClaudeBot and PerplexityBot never see it.

Google will still find it, because Google renders. That's why this failure is so quiet — every tool you'd normally check with says you're fine.

One nuance worth knowing: content doesn't have to be visible prose to count. Data embedded in the initial HTML as inline JSON or server-rendered payload is in the raw response and readable. What's missed is specifically what the browser builds after load.

Why this compounds

Around 92% of ChatGPT agent queries rely on Bing's index — and Bingbot's JavaScript rendering is limited. So a client-side rendered SPA risks being absent from Bing and from every AI crawler that queries it. Two failures, one root cause.

The test takes thirty seconds

curl -s https://you.com/page | grep -c "ld+json"
curl -s https://you.com/page | grep "your headline text"

If grep returns nothing, neither does the crawler.

Do it on your pricing page, your service pages, your best article. Those are the pages with the most to lose.

One layer deeper: the CDN

Everything above assumes the crawler receives the same response you do. It might not.

The curl test above uses your User-Agent and your IP. A CDN or edge middleware can serve differently based on User-Agent, headers, or geography — and AI bots come from datacenter IPs with declared bot UAs, which is exactly the traffic a WAF is tuned to challenge. Your HTML can be perfect and GPTBot still gets a 403, a challenge page, or a stale cache entry before it ever reaches the content.

So test as the bot, not as yourself:

curl -A "GPTBot" -s https://you.com/page | grep -c "ld+json"

And if you're on Cloudflare, the honest check is your server logs: filter by the AI crawlers' User-Agents and read the response codes. 200 means they see you. 403, a redirect, or a challenge means they don't — and no view-source will reveal it, because you're looking from a browser with a human fingerprint.

robots.txt doesn't save you here either: the WAF fires before robots.txt is consulted. Permission and access are different layers, and both fail silently.

(This section came out of a sharp comment from Wren Calloway below — worth reading the thread.)

The fix is not new

Server-side rendering, static generation, or prerendering. Next.js, Nuxt, Angular Universal, Django SSR — the framework matters less than the principle: put the real content in the HTML the server sends.

On our own platform the JSON-LD is rendered directly into the DOM rather than injected through Helmet, and key pages are prerendered. Not because it's elegant — because it's the difference between having schema and having schema anyone can read.

This is the same advice that's been correct for ten years. What changed is the cost of ignoring it. It used to mean waiting on Google's render queue. Now it means being absent from answers entirely, in the fastest-growing discovery channel there is.

Related reading: