AI Crawlers Are Eating Your Bandwidth: Block, Slow, or Bill Them

Last month I opened my server metrics expecting the usual flat line and found something else instead: a third of my bandwidth was going to bots. Not Googlebot — I want Googlebot. These were AI crawlers, and they were hitting my site harder than my actual readers.

That was the moment I stopped treating robots.txt as a set-it-and-forget-it file. Here's what I found, and exactly what I did about it, because you will want to do it too — this problem is not specific to my little server.

The bots nobody invited

Something changed in the last couple of years, and it changed quietly. If you run any website with text on it, you are now part of the largest unpaid data extraction operation in history. The Q3 2026 crawler data is honestly wild: nearly half of identified AI crawler traffic declares itself as training-only, and the share keeps creeping up quarter over quarter. Your blog posts, your tutorials, your carefully written error messages that took an afternoon to debug — all of it is raw material.

The usual suspects, so you know who to look for in your logs:

• GPTBot — OpenAI's crawler. Actually one of the more polite ones; honors robots.txt.
• ClaudeBot — Anthropic. Polite-ish, but it crawls aggressively when it decides to.
• CCBot — Common Crawl. The backbone dataset half the internet was trained on.
• Bytespider — ByteDance. Famous for hammering sites at rates that look like a DDoS and infamous for ignoring robots.txt.
• PerplexityBot — comes to answer questions in real time, sends almost no identifiable traffic back to you.
• Amazonbot, cohere-ai, Meta-ExternalAgent — the rest of the dinner party.

Some of these behave. Several don't. And here's the uncomfortable part: robots.txt is a polite request, not a lock. It works exactly as well as a "please do not enter" sign.

Step one: see what you're dealing with

Before changing anything, measure. I grabbed my nginx access log and ran it through GoAccess — we have a full walkthrough in the real-time log analysis article, but the short version is:

goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED

Sort by bandwidth. The AI crawlers will be sitting there in the top ten, often above real referrers. Do this once and you will never unsee it.

Step two: robots.txt, for the ones that listen

Keep robots.txt — some crawlers genuinely honor it, and it's a signal of your intent if you ever need to argue about it. Mine currently looks something like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

Note what I allow: Googlebot, Bing, ClaudeBot, PerplexityBot. I made a deliberate choice to stay in search indexes and AI answer engines — being findable is how this site gets readers. I only block the pure training crawlers. That's my line; yours might be different, and it should be a decision, not an accident.

Step three: the lock — nginx doesn't negotiate

For the bots that ignore the sign on the door, you enforce at the front door. In nginx, a map in the http block, then one if:

# /etc/nginx/conf.d/ai-bots.conf (or inside http {} in nginx.conf)
map $http_user_agent $ai_bot {
    default 0;
    ~*GPTBot        1;
    ~*ClaudeBot     1;
    ~*CCBot         1;
    ~*Bytespider    1;
    ~*PerplexityBot 1;
    ~*Amazonbot     1;
    ~*cohere-ai     1;
}

Then in the server {} block:

if ($ai_bot) {
    return 429;
}

429 Too Many Requests is deliberate. A hard 403 tells them the resource exists but is forbidden; 429 reads as "back off," and some crawler schedulers will actually reduce revisit frequency. Bytespider will not care either way, but it costs you almost nothing to serve a 429 — a blocked request is a few bytes instead of a full page crawl.

One warning: match user agents carefully. ~*ClaudeBot matches any UA containing that string — which is fine, because legitimate Anthropic crawlers identify exactly that way, and nobody else has business using it. Don't write loose patterns like ~*bot or you'll block Googlebot and nuke your search presence.

Step four (optional, and my favorite): send them an invoice

Blocking felt good for about a day. Then I realized the absurdity of the situation: these companies are extracting value from my server, on my electricity, and their end product is a chatbot that tells people not to visit my site. So I went one step further.

AI crawlers that hit my site now get served a 402 Payment Required page — a proper, styled page with a payment link, explaining that training access costs money and here's where to buy a license. It's a statement more than a business model. Not one of them has paid, shockingly. But the 402 sits in my logs like a tiny act of defiance, and every now and then a human finds the page, laughs, and shares it. That has driven more real traffic than any AI crawler ever sent me.

You can do this by pointing the return at a named location with a small HTML page, or just for fun:

if ($ai_bot) {
    return 402;
}

What I'd do differently next time

Start earlier. I ran unprotected for months, and the crawl patterns are sticky — once a training pipeline has your URL in a queue, it retries politely forever. Also, check your CDN/proxy logs if you sit behind Cloudflare: the origin log only shows what got through, and the real picture can be two or three times worse.

And keep watching the logs quarterly. This ecosystem mutates. New bots appear with new user agents, old ones rebrand. My map block has grown three entries since I first wrote it, and I expect it to keep growing.

The bottom line

robots.txt is a suggestion. nginx is a decision. You don't have to fund the AI industry with your server, and you shouldn't — bandwidth you pay for should serve readers, not scrapers. Block the training crawlers, keep the search engines, and if you're feeling theatrical, send the rest an invoice.