The user-agent header is a string the client chose. Anyone can send GPTBot, and plenty do — to bypass rate limits, to test somebody else’s robots.txt handling, or to make scraping look official. To know who actually visited you, you have to verify, and the standard advice to use reverse DNS is right for Google and Bing and wrong for most of the crawlers in this cluster.
Why the user agent alone is not an answer
Every published table of AI crawler user agents, including the one in the robots.txt page, is a list of strings a well-behaved client sends. It is not an identity. Three concrete consequences:
- Your traffic report is inflated by impostors. A self-declared
ClaudeBotthat is really somebody’s scraper is counted as evidence of AI interest in your site, and any conclusion drawn from that count is wrong. - Your block list is evaded trivially. A robots.txt rule against a token stops only clients that identify honestly, which is the population that would have obeyed anyway.
- Real load gets misattributed. If an unverified impostor is hammering you, blaming the named operator sends you to the wrong remedy — you file a complaint instead of writing a rate limit.
Two verification regimes, not one
This is the part the standard write-ups get wrong. There are two mechanisms operators use to let you confirm a request is genuine, and they are not interchangeable.
| Regime | Description |
|---|---|
| Forward-confirmed reverse DNS | The operator commits to reverse-DNS records under a domain it controls. You reverse-look-up the IP, check the hostname ends in that domain, then forward-resolve the hostname and confirm the original IP is in the answer. Google and Bing have documented this for years. |
| Published address ranges | The operator publishes a machine-readable file of CIDR blocks its crawler uses. You fetch it, cache it, and test membership. This is what OpenAI, Anthropic, Perplexity and Apple do for their crawlers. |
Using the wrong one produces a confident wrong answer. A reverse lookup on an OpenAI crawler address will typically return a generic cloud hostname or nothing at all, and a script that treats “reverse DNS did not match” as “impostor” will mark every genuine request as fake. So the parser below carries a per-operator verification method rather than one global one.
The exact URL of each operator’s range file is something to read from that operator’s current crawler documentation rather than from any article, including this one — the paths have moved, and a range file fetched from a stale URL is a verification that silently passes nothing. The script below takes them as configuration for that reason.
Getting a log that can answer the question
Two fields decide whether this exercise is possible at all: the client address and the user-agent string. Plenty of default configurations drop one or both, and a CDN in front of your origin replaces the client address with its own unless you read the forwarded header.
# nginx: combined already includes both.
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
# Behind a CDN, the origin address is the CDN. Use the real client:
log_format aiaudit '$http_cf_connecting_ip $remote_addr [$time_local] '
'"$request" $status $body_bytes_sent "$http_user_agent"';
Caddy writes JSON lines by default, which is easier to parse and is what the script below assumes as one of its two input formats. Whatever you use, check your retention: thirty days is the minimum that makes a crawl-rate question answerable and ninety is better, because the interesting comparison is usually against the same week last quarter.
The parser
Node with no dependencies. It classifies by token, then verifies each distinct address once by the regime that operator uses, caches the result, and reports verified against unverified per agent.
#!/usr/bin/env node
// crawler-audit.mjs — who actually fetched us, and were they real?
// node crawler-audit.mjs access.log
// node crawler-audit.mjs --json access.log (Caddy JSON lines)
//
// Verification is per operator. Reverse DNS is correct for Google and
// Bing and WRONG for the rest: they publish address ranges instead.
// Fill RANGE_SOURCES from each operator's current crawler docs.
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";
import dns from "node:dns/promises";
const RANGE_SOURCES = {
// "openai": "https://openai.com/gptbot.json",
// Add each operator's published range file here, from its own docs.
// Each is expected to be JSON with a list of { ipv4Prefix | ipv6Prefix }.
};
// token -> { operator, method, suffixes? }
const AGENTS = {
"GPTBot": { operator: "openai", method: "range" },
"OAI-SearchBot": { operator: "openai", method: "range" },
"ChatGPT-User": { operator: "openai", method: "range" },
"ClaudeBot": { operator: "anthropic", method: "range" },
"Claude-User": { operator: "anthropic", method: "range" },
"Claude-SearchBot": { operator: "anthropic", method: "range" },
"PerplexityBot": { operator: "perplexity", method: "range" },
"Perplexity-User": { operator: "perplexity", method: "range" },
"Applebot": { operator: "apple", method: "range" },
"Amazonbot": { operator: "amazon", method: "range" },
"meta-externalagent":{ operator: "meta", method: "range" },
"Bytespider": { operator: "bytedance", method: "none" },
"CCBot": { operator: "commoncrawl",method: "none" },
"Googlebot": { operator: "google", method: "rdns",
suffixes: [".googlebot.com", ".google.com"] },
"GoogleOther": { operator: "google", method: "rdns",
suffixes: [".googlebot.com", ".google.com"] },
"Google-CloudVertexBot": { operator: "google", method: "rdns",
suffixes: [".googlebot.com", ".google.com"] },
"bingbot": { operator: "bing", method: "rdns",
suffixes: [".search.msn.com"] },
};
const TOKENS = Object.keys(AGENTS);
function classify(ua) {
if (!ua) return null;
// Longest token first so "Claude-SearchBot" is not caught by "ClaudeBot".
for (const t of TOKENS.slice().sort((a, b) => b.length - a.length)) {
if (ua.toLowerCase().includes(t.toLowerCase())) return t;
}
return null;
}
/* ---- reverse DNS, forward-confirmed ---- */
const rdnsCache = new Map();
async function verifyRdns(ip, suffixes) {
if (rdnsCache.has(ip)) return rdnsCache.get(ip);
let ok = false;
try {
const names = await dns.reverse(ip);
const name = names.find((n) => suffixes.some((s) => n.endsWith(s)));
if (name) {
const v4 = await dns.resolve4(name).catch(() => []);
const v6 = await dns.resolve6(name).catch(() => []);
ok = v4.includes(ip) || v6.includes(ip);
}
} catch { ok = false; }
rdnsCache.set(ip, ok);
return ok;
}
/* ---- published CIDR ranges ---- */
const rangeCache = new Map();
async function ranges(operator) {
if (rangeCache.has(operator)) return rangeCache.get(operator);
const url = RANGE_SOURCES[operator];
if (!url) { rangeCache.set(operator, null); return null; }
try {
const doc = await (await fetch(url)).json();
const flat = JSON.stringify(doc);
const list = [...flat.matchAll(/"((?:\d{1,3}\.){3}\d{1,3}\/\d{1,2})"/g)]
.map((m) => m[1]);
rangeCache.set(operator, list);
return list;
} catch {
rangeCache.set(operator, null);
return null;
}
}
function ipToInt(ip) {
const p = ip.split(".").map(Number);
if (p.length !== 4 || p.some((n) => Number.isNaN(n))) return null;
return ((p[0] << 24) >>> 0) + (p[1] << 16) + (p[2] << 8) + p[3];
}
function inCidr(ip, cidr) {
const [net, bitsRaw] = cidr.split("/");
const bits = Number(bitsRaw);
const a = ipToInt(ip), b = ipToInt(net);
if (a === null || b === null) return false;
const mask = bits === 0 ? 0 : (~0 << (32 - bits)) >>> 0;
return (a & mask) === (b & mask);
}
async function verify(ip, agent) {
const spec = AGENTS[agent];
if (spec.method === "rdns") return verifyRdns(ip, spec.suffixes);
if (spec.method === "range") {
const list = await ranges(spec.operator);
if (!list) return null; // unknown, not false
return list.some((c) => inCidr(ip, c));
}
return null; // no published mechanism
}
/* ---- read the log ---- */
const args = process.argv.slice(2);
const asJson = args.includes("--json");
const file = args.find((a) => !a.startsWith("--"));
if (!file) { console.error("usage: crawler-audit.mjs [--json] <logfile>"); process.exit(1); }
const COMBINED =
/^(\S+) \S+ \S+ \[[^\]]+\] "(?:\S+) (\S+)[^"]*" (\d{3}) (\S+) "[^"]*" "([^"]*)"/;
const stats = new Map(); // agent -> { hits, verified, fake, unknown, ips:Set, paths:Map }
const pending = [];
const rl = createInterface({
input: createReadStream(file),
crlfDelay: Infinity,
});
for await (const line of rl) {
let ip, path, status, ua;
if (asJson) {
let e; try { e = JSON.parse(line); } catch { continue; }
ip = e.request?.remote_ip;
path = e.request?.uri;
status = e.status;
ua = e.request?.headers?.["User-Agent"]?.[0];
} else {
const m = COMBINED.exec(line);
if (!m) continue;
[, ip, path, status, , ua] = m;
}
const agent = classify(ua);
if (!agent || !ip) continue;
if (!stats.has(agent))
stats.set(agent, { hits: 0, verified: 0, fake: 0, unknown: 0,
ips: new Set(), paths: new Map(), statuses: new Map() });
const s = stats.get(agent);
s.hits++;
s.ips.add(ip);
s.paths.set(path, (s.paths.get(path) ?? 0) + 1);
s.statuses.set(String(status), (s.statuses.get(String(status)) ?? 0) + 1);
pending.push([agent, ip]);
}
/* Verify each (agent, ip) pair once. */
const seen = new Set();
for (const [agent, ip] of pending) {
const key = agent + "|" + ip;
if (seen.has(key)) continue;
seen.add(key);
const ok = await verify(ip, agent);
const s = stats.get(agent);
if (ok === true) s.verified++;
else if (ok === false) s.fake++;
else s.unknown++;
}
/* ---- report ---- */
const rows = [...stats.entries()].sort((a, b) => b[1].hits - a[1].hits);
console.log("agent hits IPs ok fake ? top path");
for (const [agent, s] of rows) {
const top = [...s.paths.entries()].sort((a, b) => b[1] - a[1])[0];
console.log(
agent.padEnd(20),
String(s.hits).padStart(6),
String(s.ips.size).padStart(5),
String(s.verified).padStart(3),
String(s.fake).padStart(5),
String(s.unknown).padStart(2),
" " + (top ? top[0].slice(0, 40) : "-"),
);
}
for (const [agent, s] of rows) {
const codes = [...s.statuses.entries()].map(([c, n]) => c + ":" + n).join(" ");
console.log(agent.padEnd(20), codes);
}
The unknown column is the honest part of this script and the reason it is worth running rather than copying a simpler one. An address that cannot be checked is reported as unchecked, not as fake. A tool that collapses those two states will tell you that most of your AI traffic is fraudulent, which is a claim it has no basis for.
Forward-confirmed reverse DNS, in detail
Reverse DNS on its own proves nothing: the owner of an address block controls its PTR records and can put any hostname there. The forward-confirmation step is what makes it evidence.
- Reverse-look-up the address.
dig -x 66.249.66.1 +shortreturns a hostname such ascrawl-66-249-66-1.googlebot.com. - Check the hostname ends in a domain the operator has documented. Ending in
.googlebot.comis the check; containing the word “google” is not, becausegooglebot.com.attacker.examplecontains it too. Match the suffix, with the leading dot. - Forward-resolve that hostname and confirm the original address is in the answer.
dig +short crawl-66-249-66-1.googlebot.com. The attacker controls the PTR for their own address but not the A record under the operator’s domain, so this step is the one that cannot be forged.
# The whole check as a shell one-liner
ip=66.249.66.1
host=$(dig -x "$ip" +short | sed 's/\.$//')
case "$host" in
*.googlebot.com|*.google.com)
dig +short "$host" | grep -qx "$ip" && echo "verified" || echo "FORGED" ;;
*) echo "not a Google hostname: $host" ;;
esac
Matching published address ranges
For the operators that publish ranges, verification is a set membership test and is both faster and more reliable than DNS, with two operational rules.
Cache, but not forever. Fetching the range file on every log line is absurd and fetching it once a year is worse: ranges are added. Once a day is the sane cadence, and the fetch should fail soft — if the file is unreachable, addresses become unknown rather than fake.
Handle IPv6. The script above does the IPv4 arithmetic in full and skips IPv6, which is a real limitation and is stated rather than hidden. A meaningful share of crawler traffic arrives over IPv6, so if your logs contain it, extend the comparison to operate on the 128-bit form before you draw conclusions about the ratio of verified to unknown.
If you are enforcing rather than auditing, the same range files feed a firewall or CDN rule directly, and that is the layer where a decision actually holds — robots.txt is a request, an address rule is a control.
Reading the result
The report has four things worth looking at, in this order.
- The status-code line per agent. A crawler receiving mostly 404s is crawling URLs you removed, and a crawler receiving 429s or 5xx is being rate-limited into an incomplete picture of your site — which will show up later as missing content and no error anywhere.
- The top path. If it is a filter permutation, a search results page or a calendar, you have a crawl trap and the fix is a narrow
Disallowpattern in robots.txt, not a block on the agent. - The fake column. Non-zero against an agent with a published verification mechanism means somebody is spoofing that token at you. Rate-limit by behaviour; do not escalate to the operator, because it is not them.
- User agents against bulk agents. Requests from
ChatGPT-User,Claude-UserorPerplexity-Userare somebody asking a question right now, and the paths they fetch are a direct readout of which of your pages a retrieval stage selected — the observable window described in how assistants choose what to cite.
An honest crawler can still take you down
The failure this cluster most wants you to avoid is not a rogue scraper. It is a fully documented, correctly identified, robots.txt respecting crawler walking a URL space that is larger than you realised you had.
The shape is always the same. Some page filters through the URL — a catalogue with facets, a calendar, a search form that produces linkable results. Every combination is a distinct URL, every one is uncached and expensive to render, and the number of them is the product of the facets rather than the count of your real pages. A crawler doing exactly what it is supposed to do walks that product, renders queue behind each other, and the site becomes slow for everybody including the crawler.
The diagnosis is in the top-path column of the report above, and it arrives before the outage if you are reading it. The fix is a narrow disallow on the query-string pattern rather than a block on the agent, because the agent is not the problem — the unbounded URL space is. That is the reasoning behind the Disallow: /models? rule shown in the robots.txt recipes, and it is worth checking whether your own site has the same shape before something finds it for you.
Top comments (0)