Your server logs show a request from GPTBot. Should you trust it?
The honest answer is: not on the strength of that string alone. A user-agent header is a claim the client makes about itself, and any client can claim anything. curl -A "GPTBot" takes about four seconds to type. If your robots.txt policy, your analytics, or your paywall logic branches on the user-agent string, it branches on unverified input.
The good news is that most major AI crawlers publish a way to check. The less good news is that a meaningful minority publish nothing at all, and one whole category is unverifiable by design. This post covers how each verification method works, which crawlers support which, and where the honest answer is still "you cannot tell."
A user-agent string is a claim, not an identity
Three things are worth separating:
- Identification — the crawler tells you who it is (the user-agent string).
- Verification — you independently confirm that claim against something the operator controls.
- Authorization — you decide what that verified identity is allowed to do.
robots.txt operates at layer three but has no access to layer two. It is an advisory protocol: RFC 9309 standardises the syntax and the expectation of compliance, not a mechanism to enforce it. A crawler that ignores your Disallow is not breaking a technical control; it is declining a request. And a crawler that impersonates a well-behaved one inherits whatever privileges you granted the original.
That matters more in 2026 than it did in 2020, because the privileges are now real. Sites allow AI crawlers to get represented in model answers, meter them under pay-per-crawl, or exempt them from bot challenges. Every one of those is a reason for someone to wear the costume.
Three verification methods, strongest first
Reverse DNS with forward confirmation
The classic method, and still the most robust for operators who support it. You take the request's IP, resolve it to a hostname (PTR record), check that the hostname ends in a domain the operator controls, then resolve that hostname back to an IP and confirm it matches the original.
The forward-confirm step is not optional. A PTR record is set by whoever controls the IP block's reverse zone, so without resolving forward again you are trusting an attacker-controlled record.
Operators documenting a reverse-DNS suffix include Google (.googlebot.com / .google.com), Microsoft (.search.msn.com), Amazon (.crawl.amazonbot.amazon), Apple (.applebot.apple.com) and Common Crawl (.crawl.commoncrawl.org).
Published IP ranges
The operator publishes a JSON file of the CIDR blocks its crawler uses; you check membership. OpenAI does this at openai.com/gptbot.json, and it is the most common method in practice — 20 of the 41 crawlers in our registry document IP-range verification, making it the single most widely supported check.
Two operational caveats. The list changes, so fetch and cache it on a schedule rather than hardcoding; and ranges are only as trustworthy as the operator's own infrastructure hygiene — a shared cloud range proves less than a dedicated one.
Web Bot Auth signatures
The direction of travel, and the only method that proves identity cryptographically rather than by network topology. The bot signs its HTTP requests with a private key using HTTP Message Signatures, publishes the public key in a discoverable JWKS directory, and your server verifies the signature. It is progressing through the IETF, and Cloudflare, Akamai and AWS have shipped support on the verifier side.
Here is the honest status from the operator side, which is the part most write-ups skip: checking the primary documentation of all 41 crawlers in our registry, we could not confirm a single one that publishes a signature-agent domain and JWKS URL you can verify against today. The infrastructure exists; the operator-side rollout is not yet documented in a way a site owner can act on. Treat Web Bot Auth as the thing to build toward, not the thing to gate on this quarter.
What the 41 crawlers actually support
Counting verification methods across the registry (a crawler can support more than one):
| Method | Crawlers documenting it |
|---|---|
| Published IP range | 20 |
| Reverse DNS | 11 |
| IP signature | 4 |
| No network-level method at all | 16 |
A crawler can document more than one method, so the first three rows overlap: 25 of 41 support at least one network-level check, and the remaining 16 support none. For those 16 the user-agent string is the only identifier the operator publishes, and there is nothing to check it against. You can allow them, you can block them, but you cannot verify them. (One of the 16, Google-Extended, has not even a user-agent string — see below.)
By purpose, the same 41 break down as: 13 user-triggered fetchers (an agent fetching a page because a human just asked), 10 search-index crawlers, 9 training crawlers, 4 agentic browsers, 3 data providers, and 2 pure opt-out tokens — Google-Extended and Applebot-Extended are not crawlers at all but robots.txt tokens that control how an already-fetched page may be used.
That distinction is worth internalising before you write a single Disallow. Blocking a training crawler costs you nothing in referral traffic. Blocking a user-triggered fetcher means a human asked their assistant about your page and the assistant could not read it.
The category you cannot verify at all
Agentic browsers are a genuinely different problem. They are real browser engines driven by an AI agent on a user's behalf. They carry no stable bot token, because from the network's point of view they are a browser: real Chrome user-agent, real TLS fingerprint, real JavaScript execution.
The registry tracks four, and the churn in that list is itself informative: ChatGPT Atlas (OpenAI) and Comet (Perplexity) are current, while OpenAI's Operator and Google's Project Mariner are already recorded as deprecated — superseded inside eighteen months. Whatever you build here, build it to be replaced.
There is no user-agent string to match, no published IP range, no reverse DNS suffix. Behavioural signals are all that remain, and behavioural signals on a headless-but-real browser are weak.
This is the honest edge of the current toolset, and it is why Web Bot Auth matters: it is the only proposal that would let an agentic browser volunteer a verifiable identity rather than requiring you to infer one.
Implementing the check
Reverse DNS with forward confirmation, in Node with no dependencies:
import { promises as dns } from "node:dns";
// Suffixes an operator has published for its crawler. Never infer these —
// use the domain the operator documents, or you will validate an impostor.
const SUFFIXES = {
googlebot: [".googlebot.com", ".google.com"],
bingbot: [".search.msn.com"],
applebot: [".applebot.apple.com"],
amazonbot: [".crawl.amazonbot.amazon"],
ccbot: [".crawl.commoncrawl.org"],
};
export async function verifyByReverseDns(ip, operator) {
const suffixes = SUFFIXES[operator];
if (!suffixes) return { verified: false, reason: "no published suffix" };
let hostnames;
try {
hostnames = await dns.reverse(ip); // PTR lookup
} catch {
return { verified: false, reason: "no PTR record" };
}
const host = hostnames.find((h) => suffixes.some((s) => h.endsWith(s)));
if (!host) return { verified: false, reason: "PTR outside published domain" };
// Forward-confirm: the PTR record alone is controlled by whoever owns the
// reverse zone, so resolve the hostname back and require the IP to match.
const forward = await dns.resolve(host).catch(() => []);
return forward.includes(ip)
? { verified: true, host }
: { verified: false, reason: "forward lookup did not match" };
}
Two things to get right in production. Do this out of band — a DNS round trip inside the request path is a latency and denial-of-service liability, so verify asynchronously and cache the verdict per IP. And fail closed on your policy, not on the request: an unverified request is not necessarily hostile, it is merely unproven, so downgrade its privileges rather than returning a 403 to what might be a real user.
Why "unclear" is a better answer than a confident guess
Most published AI-crawler lists give every bot a clean yes/no for robots.txt compliance. Compiling this data, that cleanliness turned out to be the least defensible part.
For a good number of crawlers, the operator has simply never published an explicit statement about the agent under that exact name. The available options are then to infer from a sibling bot, to copy what another list asserts, or to record "unclear" and cite what the operator actually says. Only the third survives contact with someone who checks.
So the registry behind this post carries a tri-state — yes, no, unclear — with a primary source URL attached to each individual claim and a last_verified date, plus explicit nulls where a field could not be sourced rather than a plausible-looking guess. It is less tidy. It is the only version that stays true when an operator quietly changes their docs.
If you want the per-crawler data — user-agent tokens, robots.txt tokens, verification methods, published IP-range URLs, opt-out mechanisms and the source URL behind every field — it is at agentswelcome.dev/crawlers, and there is a JSON endpoint at /api/crawlers if you would rather diff it than read it.
FAQ
Does blocking AI crawlers in robots.txt actually stop them?
Not technically. robots.txt is advisory: RFC 9309 defines the syntax and the expectation, not an enforcement mechanism. OpenAI, Anthropic and Perplexity each publish documentation stating their crawlers respect it, and the evidence is that the major operators do. Enforcement, if you need it, is a WAF rule or a verified-bot policy at your edge — not a text file.
Is Google-Extended a crawler I should block?
Google-Extended is not a crawler and has no user-agent string. It is a robots.txt token controlling whether content Google already fetched may be used for Gemini training. Disallowing it does not reduce crawl traffic and does not affect Search indexing. Applebot-Extended works the same way for Apple.
Should I block training crawlers but allow search crawlers?
That is the common policy, and the reasoning holds: a training crawler consumes your content with no referral path back, while a search-index or user-triggered fetcher can surface you in an answer a human is reading right now. Decide per bot type rather than per company — several operators run both kinds under different tokens.
How often does this data change?
Often enough to matter. Operators add tokens, rename agents and update IP ranges without announcement. Any crawler list without a per-claim source URL and a verification date is a snapshot of someone's afternoon, including this one — which is why the numbers above carry the date they were checked: 2026-07-30.
Data from the AGENTS WELCOME crawler registry — 41 crawlers, each field carrying its own primary source and verification date. Checked 2026-07-30.
Top comments (5)
Crawler identity is going to matter more as AI traffic becomes normal infrastructure traffic. User-agent strings are just claims. The useful pattern is verification that survives spoofing: reverse DNS, forward confirmation, published IP ranges, and logs that make the decision auditable.
Agreed on all four, and the last one is the part most write-ups skip.
Two things I'd add from assembling the registry. The asymmetry is bigger than it looks: of the 41 crawlers I checked, 25 publish something you can check against the network — IP ranges, reverse DNS, or a request signature. The other 16 give you a user-agent string and nothing to check it against. That is exactly where auditable logs stop being a nice-to-have, because for those you are not recording a verification result, you are recording a decision you made under uncertainty — and six months later that distinction is the whole story.
The agentic browsers are their own case. Atlas, Comet, Operator and Mariner drive a real browser, so they carry no stable bot user-agent at all — verification there is a request signature or nothing. A UA-matching rule misses them entirely while happily matching anyone who copies the string.
So yes: log the method and the outcome, not just allow or deny.
That asymmetry is the scary part. If a crawler only publishes a user-agent string, the burden shifts entirely to the site owner and every spoofed request looks plausible. A useful registry should probably score crawlers by verifiability, not just list names.
Fair point, and it exposed something the registry was hiding. It now grades every record, and the grading changed the headline number.
The scale is on the artifact, not the claimed method — because a method named in documentation with nothing published behind it leaves the burden exactly where you said it lands. Grading the label gives 25 of 41 with a network-level method. Grading what an operator actually publishes gives 16 of 39:
Three things fell out of building it that I did not expect.
Tier A is empty. Not "sparse" — empty. Not one operator in the registry publishes a signature-agent domain and a JWKS URL you can verify against today, even though Cloudflare, Akamai and AWS all ship the verifier side.
Four crawlers name a method and publish neither artifact. They sit in their own tier rather than being counted as verifiable, because "we use IP ranges" with no range file is a research task handed to you, not a check.
Two records had to come off the scale entirely. Google-Extended and Applebot-Extended are robots.txt policy tokens — nothing ever arrives calling itself that, so there is nothing to verify. Applebot-Extended even carries Applebot's reverse-DNS suffix in its documentation, which would have quietly graded a non-crawler as verifiable.
And the trap that nearly got me:
IP-signatureappears in the data for exactly four crawlers — Atlas, Comet, Operator, Mariner. It reads like cryptography. It is the opposite: those are the agentic browsers, and the field marks the absence of a check. They are tier E, and their unverifiability is structural rather than an operator omission.The tier is derived at load time from the fields it summarises, so it cannot drift from them: agentswelcome.dev/crawlers, or
verifiability.tieron every record in/api/crawlersif you want to diff it.Thanks for the push — "list of names" to "graded by what you can actually check" is a better registry than it was.
That grading change sounds like the right move. A registry should not make weak claims look equal to verifiable ones. Once the artifact carries the grade, readers can distinguish “documented assertion” from “independently checkable behavior.”