The stat that should worry you
Bots now generate 57.5% of HTML traffic on the web. Humans are the minority now. That's not a hot take, that's Cloudflare's own telemetry, and it happened quietly enough that most of us are still writing rate limiters like it's 2019.
Here's the thing that makes it worse: it's not "bots" in the generic sense. It's AI crawlers, specifically, hoovering up training data at a scale that makes old-school SEO scrapers look like a rounding error. Cloudflare alone processes 50 billion AI crawler requests a day. AI crawler traffic is up 757% since 2024.
If you run any kind of git forge, wiki, or content-heavy site, you are already paying for this. You just might not have noticed yet.
What this actually looks like from the inside
Konstantin Ryabitsev runs kernel.org — the infrastructure that serves the Linux kernel source tree to the entire planet. He wrote up what it's like to be on the receiving end right now, and the numbers are absurd:
- ~6 million requests a day hit git.kernel.org
- 14–16 of their 90 CPU cores are constantly busy just rendering individual commit pages for scrapers
- That's roughly 20% of total infrastructure capacity, burned on bots
- Actual human/legitimate traffic? About 2% of the total
Read that last line twice. Two percent legitimate, and they're still keeping the lights on for the other 98.
The part that should make every backend engineer furious: these bots are rendering the Linux kernel's ~1.48 million commits one HTML page at a time, through the web UI, instead of doing what literally any competent tool would do — git clone the repo once and walk the history locally. The scrapers are choosing the most expensive possible way to get the data. Nobody optimized for cost because it isn't their cost. It's yours.
The Anubis arms race
The current best defense is Anubis — an open-source, MIT-licensed reverse proxy from Xe Iaso that makes clients solve a SHA-256 proof-of-work puzzle before they get served. Solve it, get a token, browse for a week. It's now running in front of GNOME's GitLab, FFmpeg, Wine, FreeCAD, and most of the FOSS git-forge world that isn't already behind Cloudflare.
It works. Kind of. On kernel.org, Anubis immediately blocks about two-thirds of incoming bot traffic outright. The other third solves the puzzle and gets through anyway.
And that's the uncomfortable truth about proof-of-work as a defense: it's not actually asymmetric warfare in your favor. A random human on a five-year-old laptop pays the same CPU tax as a scraper running on a rack of GPUs that were purpose-built to be fast at exactly this kind of math. Someone on Hacker News built an iOS extension that solves Anubis difficulty-6 challenges in milliseconds — while a normal browser takes something like 180 seconds on the same difficulty. If your adversary is better resourced than your users, proof-of-work just becomes a tax on the people you didn't mean to tax.
Ryabitsev has had to keep ratcheting the difficulty up — from 4 to 5 and climbing — because the trivial bots get filtered but the well-funded ones keep paying the toll. His read on it: "what we have to offer is worth spending a ton of cycles" for. That's not a bug in their scraper. That's a rational business decision on their end, and it should change how you think about your defenses.
Why IP and user-agent blocking are already dead
The old playbook — block by user-agent string, block by IP range — stopped working the moment scraping got monetized through proxy SDKs baked into smart TV apps and random IoT firmware. Now the requests come from millions of real residential and mobile IPs, each one indistinguishable from an actual person's phone, because in a technical sense, it is one — just running someone else's code in the background without the owner's knowledge. One admin running a small Gitea instance reported eating 40,000 requests a minute for 20 straight minutes from a rotating botnet like this. You can't IP-block your way out of a network that looks exactly like your real users.
What's actually holding the line
Skip the arms race entirely where you can:
-
Point crawlers at the cheap path. If your data is in git, tell scrapers (and humans) to
git cloneinstead of walking a paginated HTML UI. Every rendered page is compute you're donating for free. - Deploy Anubis or an equivalent PoW gate, but tune the difficulty against your actual traffic, not a default. Too low and it's theater. Too high and you're punishing your real users' battery life.
- Tarpit, don't just block. Feed known bad actors garbage — the "iocaine" approach — or serve them content one byte at a time. Wasting their compute is a better ROI than wasting yours on a block-list you'll lose.
- Use robots.txt honeypot paths. Anything that respects robots.txt won't touch them. Anything that hits them gets flagged instantly and you can react from there.
- Assume none of this is permanent. Codeberg watched scrapers learn to solve Anubis challenges within months of deployment. This is not a "set and forget" problem, it's an ongoing adversarial system, and you should budget engineering time for it the same way you budget for dependency updates.
The referral math is also insulting
If the CPU bill wasn't enough, look at what you get back for it. Mistral's crawler fetches 3,389 pages for every referral it sends back to the site it scraped. Anthropic's ClaudeBot sits at 2,237:1. OpenAI's GPTBot is comparatively generous at 217:1 — which is still a terrible deal if you're the one hosting the content.
You're not getting SEO traffic out of this exchange. You're a training-data vendor who never signed a contract and isn't getting paid.
The bottom line
Every open, freely-crawlable resource you run — a git forge, a wiki, a docs site, an API with public endpoints — is now a line item in someone else's training pipeline, whether you agreed to it or not. The kernel.org numbers aren't an outlier, they're a preview. If bots are already the majority of traffic on the open web, "just add rate limiting" stopped being an adequate answer a while ago.
Go check your logs. The 2% number might be closer to your reality than you think.
Top comments (0)