AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up.

Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI.

Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like.

A User-Agent is a claim, not evidence

Writing GPTBot/1.2 into a header costs nothing, so a table grouped by UA is a list of claims.

Behind Cloudflare there are two things to check those claims against:

1. verifiedBotCategory — Cloudflare's reverse-DNS verification result. Empty string means unverified
2. Whether the requested path exists in your sitemap — separates reading an article from crawl bookkeeping such as robots.txt

What survives both filters is "verified bots fetching real pages", and that is the only number worth reporting.

The query (works on a free zone)

Add userAgent and verifiedBotCategory to the dimensions of httpRequestsAdaptiveGroups. clientAsn and botClass require a paid plan; verifiedBotCategory does not.

QUERY_BY_UA_VERIFIED = """
query ($zoneTag: String!, $since: Time!, $until: Time!) {
  viewer {
    zones(filter: { zoneTag: $zoneTag }) {
      httpRequestsAdaptiveGroups(
        limit: 5000
        filter: { datetime_geq: $since, datetime_leq: $until }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { userAgent verifiedBotCategory }
      }
    }
  }
}
"""

Two constraints to plan around:

• One request covers at most one day. Chunk the range in the caller
• Retention is short (roughly a week at daily granularity). You cannot re-derive the past, so write a snapshot to disk on every run

def fetch_rows(token, query, zone_tag, since, until):
    # split the range into one-day windows and concatenate the rows
    all_rows, cursor = [], since
    one_day = dt.timedelta(days=1)
    while cursor < until:
        win_end = min(cursor + one_day, until)
        variables = {
            "zoneTag": zone_tag,
            "since": cursor.isoformat() + "Z",
            "until": win_end.isoformat() + "Z",
        }
        try:
            all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False)))
        except RuntimeError as e:
            print(f"    [skipped] {cursor.date()}-{win_end.date()}: {e}")   # past retention
        cursor = win_end
    return all_rows

One UA string, two rows

That single extra dimension is enough to split claim from reality. Eight days:

Claimed UA	Verified	Unverified	Verified share
ChatGPT-User	211 (AI Assistant)	336	39%
Amazonbot	135 (AI Crawler)	489	22%
ClaudeBot	133 (AI Crawler)	126	51%
OAI-SearchBot	61 (Search Engine Crawler)	132	32%
GPTBot	19 (AI Crawler)	123	13%
meta-externalagent	209 (AI Crawler)	0	100%
Applebot	56 (AI Search)	0	100%
PerplexityBot	0	144	0%
Perplexity-User	0	311	0%

Of 547 requests presenting as ChatGPT-User, 211 came from an address that traced back to OpenAI.

Only two agents came through clean — meta-externalagent and Applebot, 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified.

One of the agents cannot exist

123 requests claimed Google-Extended. Verified share 0%, and 70 of them hit credential-scanning paths.

No inference required. Google's crawler documentation states that Google-Extended has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control.

So every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships gptbot.json.

Classify the path, not just the client

Verification alone isn't enough: a verified bot fetching robots.txt has read nothing.

SCAN_PATTERNS = (
    "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials",
    "config.json", "service_account", "actuator", "api/auth", "phpinfo",
    ".bak", ".yml", ".yaml", ".php", ".sql", "id_rsa", ".npmrc", ".htpasswd",
)
OPS_PREFIXES = ("/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/")
ASSET_PREFIXES = ("/_astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/_image")


def classify_path(path, sitemap_paths):
    # content = a real page was consumed / ops = crawl bookkeeping
    # asset = static file / scan = credential probing / other = path does not exist
    if not path:
        return "other"
    low = path.lower()
    if any(k in low for k in SCAN_PATTERNS):
        return "scan"
    if low.startswith(OPS_PREFIXES):
        return "ops"
    if low.startswith(ASSET_PREFIXES):
        return "asset"
    if not sitemap_paths:
        return "unknown"   # cannot assert existence, so cannot call it content
    return "content" if (path.rstrip("/") or "/") in sitemap_paths else "other"

Using the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of "a page of mine".

Keep the unknown branch too. Fold it into content and your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it.

The result: 468 content fetches against 991 scans

Claimed UA	content	ops	scan	total	verified
ChatGPT-User	216	0	181	547	39%
meta-externalagent	93	9	0	209	100%
Amazonbot	85	1	284	624	22%
Applebot	27	9	0	56	100%
OAI-SearchBot	17	43	74	193	32%
PerplexityBot	13	8	67	144	0%
GPTBot	7	9	72	142	13%
ClaudeBot	6	129	70	259	51%
Google-Extended	0	0	70	123	0%
Perplexity-User	0	0	173	311	0%

content totals 468, scan totals 991. By status code, 403s came to 957 against 705 served with 200.

The ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. "ClaudeBot sent 259 requests" and "six articles were read" are the same data.

Keep the verified share as a column in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot.

---

The week-over-week comparison (content 610 → 468 while scan went 357 → 991), why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones