Identifying AI crawlers in your access log: user agent, source, and crawl frequency

Your access log already answers "which AI crawlers hit my site, how hard, and are they who they claim to be". You don't need a new SDK or a tag manager. You need three passes over the same file: classify by user agent, verify by source, measure frequency.

Here are the commands and one Python script that produce all three reports.

What the log gives you

nginx combined format, one line per request:

203.0.113.7 - - [23/Oct/2026:13:45:01 +0000] "GET /blog/wordpress-hosting-cost HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; GPTBot)"

Fields: client IP, timestamp, request line, status, bytes sent, referer, user agent. That's enough. If your log is JSON or a custom format, only step 2 changes.

Step 1 — look at the raw user agents first

awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

The top twenty UAs on your site. AI crawlers almost always declare themselves: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot, meta-externalagent. Treat that as data, not code — keep it in a config file you edit, because operators rename crawlers and new ones show up.

One check before anything else: if you sit behind a CDN or reverse proxy, $remote_addr may be the proxy, not the crawler. Confirm you're logging the real client IP before you trust any per-IP number.

Step 2 — classify

Save this as ai_crawlers.py:

#!/usr/bin/env python3
"""Classify AI crawler traffic in an nginx combined access log."""
import re
import sys
from collections import Counter, defaultdict
from datetime import datetime

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+)(?: [^"]*)?" '
    r'(?P<status>\d{3}) (?P<size>\S+) "(?P<ref>[^"]*)" "(?P<ua>[^"]*)"'
)

BOTS = {
    "gptbot": "GPTBot",
    "oai-searchbot": "OAI-SearchBot",
    "chatgpt-user": "ChatGPT-User",
    "claudebot": "ClaudeBot",
    "perplexitybot": "PerplexityBot",
    "ccbot": "CCBot",
    "bytespider": "Bytespider",
    "amazonbot": "Amazonbot",
    "meta-externalagent": "meta-externalagent",
}

def classify(ua):
    low = (ua or "").lower()
    for token, name in BOTS.items():
        if token in low:
            return name
    if not low or low == "-":
        return "no-ua"
    if re.search(r"bot|crawl|spider|slurp", low):
        return "other-bot"
    return "browser"

def main(path):
    rows = []
    with open(path, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m:
                continue
            d = m.groupdict()
            d["bot"] = classify(d["ua"])
            d["day"] = datetime.strptime(
                d["ts"], "%d/%b/%Y:%H:%M:%S %z"
            ).date().isoformat()
            rows.append(d)

    per_day = defaultdict(Counter)
    ips = defaultdict(Counter)
    paths = defaultdict(Counter)
    status = defaultdict(Counter)

    for r in rows:
        per_day[r["day"]][r["bot"]] += 1
        ips[r["bot"]][r["ip"]] += 1
        paths[r["bot"]][r["path"]] += 1
        status[r["bot"]][r["status"]] += 1

    totals = {b: sum(d[b] for d in per_day.values()) for b in ips}
    bots = sorted(totals, key=lambda b: -totals[b])

    print(f"{'bot':<18} {'hits':>8} {'ips':>6}  top path")
    for b in bots:
        top = paths[b].most_common(1)[0][0] if paths[b] else "-"
        print(f"{b:<18} {totals[b]:>8} {len(ips[b]):>6}  {top}")

    print("\ndaily volume")
    days = sorted(per_day)
    print("day".ljust(12) + "".join(b[:12].rjust(14) for b in bots))
    for day in days:
        print(day.ljust(12) + "".join(str(per_day[day][b]).rjust(14) for b in bots))

    print("\nstatus mix")
    for b in bots:
        mix = ", ".join(f"{s}:{n}" for s, n in status[b].most_common(4))
        print(f"  {b:<18} {mix}")

    print("\ntop ips per bot")
    for b in bots:
        print(f"\n[{b}]")
        for ip, n in ips[b].most_common(5):
            print(f"  {ip:<40} {n}")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "/var/log/nginx/access.log")

python ai_crawlers.py /var/log/nginx/access.log

You get per-crawler hit totals, unique IP counts, the top path, a daily matrix, a status mix, and the busiest IPs. The status mix is the first filter — a bot with mostly 404s is not reading your site, it's walking a list.

Step 3 — verify the source

A user agent is a string the client chooses. Anyone can send GPTBot. So a matching UA means claimed, not verified. Verification means the IP is inside the range the operator publishes:

import ipaddress
import json

# published-ranges.json: {"GPTBot": ["203.0.113.0/24", ...], "ClaudeBot": [...]}
RANGES = json.load(open("published-ranges.json"))
NETS = {bot: [ipaddress.ip_network(c) for c in cidrs]
        for bot, cidrs in RANGES.items()}

def is_verified(ip, claimed_bot):
    nets = NETS.get(claimed_bot)
    if not nets:
        return None  # no published list for this name: unknown, not verified
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in nets)

Most crawler operators publish their ranges as a machine-readable file and document the reverse-DNS suffix they use. Fetch the file on a schedule, store it, diff it — the diff is your early warning that a range moved.

Reverse DNS is the second opinion, and it should only run on hits that failed the range check, not on every line:

import socket

def fcrdns(ip):
    """Forward-confirmed reverse DNS. Returns hostname or None."""
    try:
        host = socket.gethostbyaddr(ip)[0]
        _, _, addrs = socket.gethostbyname_ex(host)
    except (socket.herror, socket.gaierror):
        return None
    return host if ip in addrs else None

Record three outcomes, not two: verified (IP inside the published range), claimed (UA says bot, IP outside every range), unknown (no UA at all). Only the first is evidence that a crawler is actually crawling you.

Step 4 — measure crawl frequency

Hourly volume for one crawler, without touching the script:

grep -i 'ClaudeBot' /var/log/nginx/access.log \
  | awk -F'[][]' '{print $2}' \
  | cut -d: -f1,2 \
  | sort | uniq -c

How to read the numbers:

• Steady hits per day from a handful of IPs — a real crawler on a schedule. Note the rate; that's your baseline.
• One IP, thousands of URLs, mostly 404 — probing, or a sitemap pointing at pages you deleted.
• A burst inside a few minutes — a re-crawl of pages that changed. Check what you deployed that day.
• Many IPs, few hits each — distributed. Verify against ranges before you assume one operator.
• The same path with varying query strings — cache-busting. Your cache may be serving fresh HTML on every request, which invites more crawling.

Pair the frequency with the path list. A crawler that only touches /blog/* is indexing. One that touches /cart, /checkout, or /api/* is doing something you probably want to stop regardless of who it is.

Step 5 — turn the log into a decision

Four buckets, four actions:

Evidence	Action
Verified, and you want it (answer engines that send referrals)	Leave open, make sure the page renders server-side
Verified, and you don't want it (training-only)	robots.txt token, or a block at the proxy
Claimed but not verified	Don't block on UA alone — block on "outside published range"
Unknown, high volume, odd paths	Rate-limit at the edge, then watch

robots.txt is a request. A block at nginx is enforcement. If you need enforcement, do it with a map and keep logging the blocked requests, so six weeks later you can still see what you cut off.

Step 6 — keep it running

• Keep enough history to compare week over week; a single day tells you nothing about a trend.
• Run the script from cron, write JSON, diff against yesterday. A new UA with more than a hundred hits is the thing worth reading.
• Split logs by crawler family so you stop grepping a multi-gigabyte file:

map $http_user_agent $ai_log {
    default          main;
    ~*GPTBot         ai_gptbot;
    ~*ClaudeBot      ai_claude;
    ~*PerplexityBot  ai_perplexity;
    ~*CCBot          ai_ccbot;
}

open_log_file_cache max=100 inactive=60s;
access_log /var/log/nginx/$ai_log.log combined;

The map goes in the http block; the variable in the path is why open_log_file_cache is there.

If you'd rather not maintain the lists

Keeping the token map, the published ranges, and the daily diff in sync is the boring half of this job. If you run WordPress, AumCrawl shows which AI crawlers are hitting the site and lets you allow or block per crawler. The access log stays the source of truth either way — this is the dashboard on top of it, not a replacement for the evidence.