How to confirm an AI crawler is causing your server load, and which bot to rate-limit first

When the load average climbs and someone in the channel says it is the AI crawlers, that is a hypothesis, not a diagnosis. Before you add a single rate limit, three things have to be true:

1. the load spike lines up in time with crawler requests,
2. those requests are hitting work that actually costs CPU or database time,
3. one crawler dominates that cost — the fleet as a whole does not.

Short answer for step 3: the bot with the highest request-time seconds per minute inside the spike window, among the bots you have decided to keep. Not the bot with the most requests. Everything below is how to get those numbers out of a log you already have.

Step 0 — Check that it is CPU at all

nproc
uptime
vmstat 1 5

Read r (run-queue length) and wa (I/O wait). If r is consistently above the number of cores from nproc, requests are queueing. If wa is high while us and sy are low, you are waiting on disk or the database and a web-tier rate limit will not fix it — but write the number down anyway, because you will want it again in step 6.

A load average of 40 on a 4-core box and a load average of 40 on a 64-core box are different incidents. Always report the ratio.

Step 1 — Get $request_time into the access log

The combined log format tells you that a request happened, not what it cost. Add the timing field:

log_format main_ext '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" $request_time';

access_log /var/log/nginx/access.log main_ext;

Reload, and confirm a line ends with a number:

tail -n 3 /var/log/nginx/access.log

If you cannot reload right now, skip the timing column — the script below treats it as optional and you can still rank by request count.

Step 2 — Per-minute, per-crawler totals

Save this as crawl_report.py. Standard library only, no dependencies.

#!/usr/bin/env python3
"""Per-minute, per-crawler request totals from an nginx access log.

Usage:
    python3 crawl_report.py /var/log/nginx/access.log > crawlers.csv

Expects the combined log format; the trailing $request_time field is optional.
"""
import csv
import re
import sys
from collections import defaultdict

MONTHS = {m: f"{i:02d}" for i, m in enumerate(
    "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec".split(), start=1)}

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
    r'"(?P<req>[^"]*)" (?P<status>\d{3}) (?P<size>\S+) '
    r'"[^"]*" "(?P<ua>[^"]*)"(?:\s+(?P<rt>[0-9.]+))?\s*$'
)

# substring found in the User-Agent -> label. Extend as needed.
BOTS = [
    ("gptbot", "GPTBot"),
    ("oai-searchbot", "OAI-SearchBot"),
    ("chatgpt-user", "ChatGPT-User"),
    ("claudebot", "ClaudeBot"),
    ("perplexitybot", "PerplexityBot"),
    ("bytespider", "Bytespider"),
    ("ccbot", "CCBot"),
    ("googlebot", "Googlebot"),
    ("bingbot", "bingbot"),
]


def label(ua):
    low = ua.lower()
    for key, token in BOTS:
        if token.lower() in low:
            return key
    return "other" if ua else "no-ua"


def minute(ts):
    # 24/Oct/2026:13:22:01 +0000 -> 2026-10-24T13:22
    day, mon, rest = ts.split("/", 2)
    year, clock = rest.split(":", 1)
    return f"{year}-{MONTHS[mon]}-{int(day):02d}T{clock[:5]}"


def main(path):
    requests = defaultdict(int)
    seconds = defaultdict(float)
    errors = defaultdict(int)

    with open(path, errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m:
                continue
            key = (minute(m["ts"]), label(m["ua"]))
            requests[key] += 1
            if m["rt"]:
                seconds[key] += float(m["rt"])
            if m["status"].startswith("5"):
                errors[key] += 1

    out = csv.writer(sys.stdout)
    out.writerow(["minute", "bot", "requests", "req_seconds", "errors_5xx"])
    for key in sorted(requests):
        out.writerow([key[0], key[1], requests[key],
                      round(seconds[key], 1), errors[key]])


if __name__ == "__main__":
    main(sys.argv[1])

Run it against a rotated log that covers the incident:

python3 crawl_report.py /var/log/nginx/access.log.1 > crawlers.csv
head -5 crawlers.csv

Step 3 — Put load on the same clock

If your monitoring already samples every minute, export the load series as minute<TAB>load1 and skip the sampler. Otherwise:

( while :; do
    printf '%s ' "$(date -u +%Y-%m-%dT%H:%M)"
    cut -d' ' -f1-3 /proc/loadavg
    sleep 60
  done ) >> /tmp/load.log

Leave it running for a day, then join the two files on the minute column:

awk 'NR==FNR { load[$1] = $2; next }
     FNR == 1 { next }
     { printf "%s %-14s req=%4d req_s=%6.1f load1=%s\n",
            $1, $2, $3, $4, (($1 in load) ? load[$1] : "-") }' \
    /tmp/load.log crawlers.csv | sort

The output is per minute per bot. You want a window where load1 is high and, in the same minutes, one bot's req and req_s jump. A bot that holds a steady 2% of traffic through the spike and through the quiet hours is not your problem, whatever its total is.

The arithmetic sanity check

req_seconds per minute divided by 60 gives the average number of workers that traffic kept busy. Divide the total across all bots and all visitors by 60 as well. If the total sits well below your worker count, nothing in the web tier explains the load — look at cron jobs, backups, or the database. This is the single cheapest way to avoid throttling the wrong thing.

Step 4 — Check what the bot is actually fetching

req_seconds says how much time it consumed. The paths say whether that time went to a static file or to a faceted search query.

# volume, query strings stripped
awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \
  | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20

# variety, query strings kept
awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \
  | sort -u | wc -l

A crawler hitting 400 distinct filtered URLs once each is a different problem from one hitting the same homepage 400 times: the first defeats your cache, the second does not. Also check the status codes — a crawler walking a broken pagination loop can generate thousands of 404s that cost you a database lookup each.

Step 5 — Rank the candidates

Signal	What it tells you	Weight it
req_seconds in the spike window	how much server time it consumed	high
share of total req_seconds during the spike vs. during quiet hours	whether it grew, or was always there	high
distinct URLs fetched	whether it defeats caching	medium
5xx and 404 counts	whether it is stressing failing paths	medium
raw request count	how chatty it is	low — often static files

Throttle one bot first. If the load does not move, you learned something for the price of one config change, and you still have the other candidates unthrottled.

Also: the User-Agent string is self-reported. Before you hard-block anything, resolve the client IP and compare it against the IP ranges the operator publishes. A spoofed UA is common enough that blocking by UA alone is unreliable.

Step 6 — Apply a limit to a single bot and re-measure

Classify by UA, keep one bucket per crawler, and let everyone else through untouched:

# http context
map $http_user_agent $crawler_bucket {
    default               "";
    "~*GPTBot"            "gptbot";
    "~*ClaudeBot"         "claudebot";
    "~*PerplexityBot"     "perplexitybot";
    "~*Bytespider"        "bytespider";
}

limit_req_zone $crawler_bucket zone=crawler_rate:10m rate=30r/m;
limit_req_status 429;
limit_req_log_level notice;

# server context
location / {
    limit_req zone=crawler_rate burst=10 nodelay;
    # ... your existing fastcgi_pass / proxy_pass
}

Requests whose key evaluates to an empty string are not accounted by limit_req, so ordinary browsers fall outside the limiter. Verify that rather than trusting it:

curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.0' https://example.com/

Fire the second one 40 times in a row; you should see 429 appear. Then nginx -t && systemctl reload nginx and re-run the report for the equivalent window tomorrow, with load.log still sampling.

The 30 r/m figure is a starting point, not a recommendation. Pick a rate by looking at what the bot actually consumes and what you can serve, then move it until the load series flattens.

Limits that do not need a config reload

robots.txt Crawl-delay is advisory — it is a request, not an enforcement mechanism, and it is not honoured by every crawler. Use it as a courtesy alongside a real limit, never as the only control. Outright disallow is a business decision, not a technical one: someone has to decide whether that bot's traffic has any value to you before you cut it off.

Step 7 — If you would rather not run the log pipeline

For WordPress sites, AumCrawl is a free plugin that shows which AI crawlers are hitting the site and lets you allow or block them per crawler, which replaces steps 2 and 6 with a UI. The measurement approach above still applies if you want to justify the decision with numbers.

Before you ship the rate limit

• [ ] nproc and the peak load are written down together as a ratio.
• [ ] wa was checked; if the bottleneck is I/O, this article is not the fix.
• [ ] $request_time is in the log format, or you have accepted count-only ranking.
• [ ] The spike window is identified by minute, not by day.
• [ ] req_seconds per minute ÷ 60 is compared against worker count.
• [ ] One bot is throttled, not the whole fleet.
• [ ] A normal browser UA is confirmed unaffected after the reload.
• [ ] The client IP was checked against the operator's published ranges.
• [ ] The re-measurement is scheduled for the same hour on another day.