DEV Community

Davide for Miraqo

Posted on Edited on

Is that really GPTBot? Checking AI crawlers against their official IP lists

This autumn we ran a six-week test with Giovanni, who builds a WordPress plugin that generates llms.txt and similar files for AI engines. The question was simple: do AI crawlers actually read these files? To answer it we had to count AI crawler requests in a server log, and that turned out to be the tricky part.

A user agent is just a string. Anyone can send GPTBot/1.3 in a request header.

Two sites, two pictures

On the store we used for the test, impostors were rare: about 360 requests out of more than 180,000 declared an AI user agent from an address that wasn't on the operator's list. That's 0.2%.

On a second site, smaller and with modest traffic, Giovanni found the opposite. Just over half of the "AI crawler" requests came from verified crawlers. The rest came from two scanners that borrowed crawler names while looking for configuration files and credentials. His plugin counted them too, since it only sees the user agent.

So if you're counting GPTBot or ClaudeBot visits from the user agent alone, the number might be close to right or off by half. You can't tell which without checking the IPs.

The operators publish their IP ranges

OpenAI, Anthropic, Apple and Perplexity all publish JSON lists of the addresses their crawlers use, in the same format ({"prefixes": [{"ipv4Prefix": "..."}]}):

Crawler List
GPTBot https://openai.com/gptbot.json
OAI-SearchBot https://openai.com/searchbot.json
ChatGPT-User https://openai.com/chatgpt-user.json
ClaudeBot, Claude-User, Claude-SearchBot https://claude.com/crawling/bots.json
Applebot https://search.developer.apple.com/applebot.json
PerplexityBot https://www.perplexity.ai/perplexitybot.json
Perplexity-User https://www.perplexity.ai/perplexity-user.json

The script

Standard library only. It reads access logs in the common "combined" format, matches the user agent, checks the client IP against the right list and prints verified vs impostor counts. It also lists any llms.txt, llms-full.txt or ai.txt requests from verified crawlers, since that was our question.

#!/usr/bin/env python3
"""Which "AI crawlers" in your access log are real?
Usage: python3 verify_ai_crawlers.py access.log [more.log ...]   (combined log format)"""
import ipaddress, json, re, sys, urllib.request
from collections import Counter

# User-agent token -> IP list published by the operator. Order matters: first match wins.
LISTS = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "ClaudeBot": "https://claude.com/crawling/bots.json",
    "Claude-User": "https://claude.com/crawling/bots.json",
    "Claude-SearchBot": "https://claude.com/crawling/bots.json",
    "Applebot": "https://search.developer.apple.com/applebot.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}
CONTEXT_FILES = re.compile(r"^/(llms\.txt|llms-full\.txt|ai\.txt)/?(\?|$)")
LINE = re.compile(r'^(\S+) .*?"\S+ (\S+)[^"]*" \d{3} \S+ "[^"]*" "([^"]*)"')

def networks(url, cache={}):
    if url not in cache:
        req = urllib.request.Request(url, headers={"User-Agent": "verify-ai-crawlers"})
        prefixes = json.load(urllib.request.urlopen(req, timeout=20))["prefixes"]
        cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
    return cache[url]

seen, files = Counter(), Counter()
for path in sys.argv[1:]:
    for line in open(path, errors="replace"):
        m = LINE.match(line)
        if not m:
            continue
        ip, url, ua = m.groups()
        bot = next((b for b in LISTS if b.lower() in ua.lower()), None)
        if not bot:
            continue
        try:
            addr = ipaddress.ip_address(ip)
        except ValueError:
            continue
        real = any(addr in n for n in networks(LISTS[bot]) if n.version == addr.version)
        seen[bot, real] += 1
        if real and CONTEXT_FILES.match(url):
            files[bot, url.split("?")[0]] += 1

print(f"{'crawler':18}{'verified':>10}{'impostor':>10}")
for bot in LISTS:
    if seen[bot, True] or seen[bot, False]:
        print(f"{bot:18}{seen[bot, True]:>10}{seen[bot, False]:>10}")
print("\nContext files fetched by verified crawlers:")
for (bot, url), n in sorted(files.items()):
    print(f"  {bot:16} {url:16} {n}")
if not files:
    print("  none")
Enter fullscreen mode Exit fullscreen mode

On our test store, over six weeks, the "Context files" section of the output would have said none. About 118,000 verified AI crawler requests went to category and product pages, and none of them asked for the files written for them.

Things that will trip you up

  • Your log must hold the real client IP. Behind Cloudflare or another proxy, the log may show the proxy's addresses, and every crawler will look like an impostor. Restore the visitor IP first (on Cloudflare, from CF-Connecting-IP).
  • The lists are snapshots. Operators add and drop ranges. A genuine crawler on an address that has since been removed will show up as an impostor, so check old logs soon, or save a copy of the lists alongside them.
  • Caches hide requests. Anything served entirely by a CDN or a page cache never reaches the origin log. Count at the layer that sees every request.
  • Google isn't in the table. There's no separate Google crawler to count for AI features: Google-Extended is a robots.txt token, not a user agent. Googlebot can be verified the same way with Google's own list, googlebot.json.
  • Match order matters. The script takes the first token that appears in the user agent, so keep more specific names before generic ones if you add crawlers.

If verified crawlers never request /llms.txt in your logs, that's your answer for now, whatever anyone says about the file. The full test, with the method, the citation data and its limits, is here: We served llms.txt to AI crawlers for six weeks. None of them asked for it.

This post was written with the help of AI and reviewed and edited by the author.

Top comments (0)