Originally published on LetsLaunch.
Most of what gets written about AI crawlers is about visibility — can ChatGPT
see your page, will it cite you, should you welcome the traffic. None of that
is the problem for a growing list of sites. Their problem is that the traffic
showed up, kept showing up, and the bill came due.
The clearest documented case is Wikimedia. The Wikimedia Foundation's own
engineering blog reported that bandwidth consumption for multimedia downloads
from Wikimedia Commons has surged 50% since January 2024, and it did not
hedge about the cause: the surge was not coming from human readers, but from
"automated, data-hungry scrapers looking to train AI models." TechCrunch's
reporting on the post picked up the same figures. This is not a monitoring
question — we covered that one already —
it's a cost and reliability question, and it deserves its own answer.
Why bots cost more per request, not just more in total
The intuitive model is that bot traffic is a scaled-up version of human
traffic — more requests, same shape, so the cost scales linearly with volume.
Wikimedia's explanation is that this isn't what happened, and the mismatch is
the actual story.
Human traffic clusters around popular pages, the ones a CDN or edge cache
already has warm. Crawler bots, per Wikimedia's own account, tend to "bulk
read" — they work through large numbers of pages indiscriminately, including
the long tail of obscure pages that were rarely requested before and were
never worth caching. Serving those requests means going back to origin
infrastructure instead of an edge cache, every time. A bot doesn't just add
volume; it adds volume in the expensive shape.
The numbers back this up directly. Bots accounted for roughly 65% of the
most resource-intensive traffic Wikimedia serves, despite representing only
around 35% of overall pageviews. Read that gap carefully: the group causing
two-thirds of the expensive work is a third of the requests. If you only
looked at request counts, you would badly underestimate what's driving your
bill.
Wikimedia was blunt about what this does to infrastructure built for a
different kind of spike: its systems, in the Foundation's own words, were
"built to sustain sudden traffic spikes from humans during high-interest
events, but the amount of traffic generated by scraper bots is unprecedented
and presents growing risks and costs." That's a nonprofit that has spent two
decades engineering for viral human traffic, saying the thing it wasn't built
for is a bot.
It isn't only Wikimedia
Wikimedia has the size and the engineering culture to publish a number. Most
sites carrying the same problem don't, which is a reason to take the
anecdotal reports seriously rather than dismiss them for lacking a published
percentage.
The same body of reporting that surfaced Wikimedia's figures also names
several smaller, independent infrastructure projects describing the identical
pattern in their own words: the git-hosting service SourceHut, Diaspora
developer Dennis Schubert, the repair-guide site iFixit, and the
documentation host Read the Docs have all separately and publicly reported
bandwidth spikes and inflated infrastructure costs they attribute to
AI-crawler traffic — in some cases severe enough to affect site reliability
for real users. Unlike Wikimedia's figures, none of these came with a formal
published percentage, so treat them as real, corroborating, but less precise
than Wikimedia's own numbers. The pattern showing up independently across a
git host, a documentation site, a repair-guide site, and an individual
developer's project is still worth something even without a shared
methodology behind it.
Wikimedia's response is measurable too: its 2025/2026 annual plan sets an
explicit goal of cutting crawler-generated request rate by 20% and bandwidth
usage by 30%. That's a nonprofit budgeting engineering time specifically
against bot load, not against human growth — which tells you how large the
line item got.
What this means if you're not running Wikimedia-scale infrastructure
Almost nobody reading this runs a site at Wikimedia's scale, and that's the
point worth making rather than skipping past: the mechanism doesn't require
that scale. A "bulk read" crawler that hits your long tail of rarely-visited
pages, bypassing your cache the same way, produces the same disproportionate
cost on infrastructure a hundredth the size. The absolute dollar figure is
smaller. The ratio — request share versus cost share — is the same shape.
If you're behind a CDN, you already have more leverage here than you might
realize, and most of it costs nothing to turn on.
Start with rate-limiting, not blocking. LetsLaunch runs behind
Cloudflare, the same setup we've described before
for watching crawler traffic — the same dashboard that shows you who's
visiting can also throttle them. Cloudflare's rate-limiting and bot-management
rules can target a specific crawler's request rate directly, which is a much
narrower tool than an outright block. A crawler capped at, say, one request
per second still finishes crawling your site — just slower, and without
hammering your origin in a burst. That's usually the actual goal: less load,
not zero bots.
Know what blocking actually trades away. This is the part worth being
honest about, because it points in the opposite direction from most of what
this blog has argued elsewhere. We've spent several other posts
on the case for staying crawlable, because citation eligibility depends on a
crawler being able to reach your page at all. Blocking a retrieval crawler
outright — OAI-SearchBot, PerplexityBot, Claude-SearchBot — solves a
bandwidth problem by creating a visibility problem. If your actual goal is
citation, rate-limiting is almost always the better trade: a slower crawl
still completes, and you stay in whatever index that crawler feeds. Reserve
an outright block for a crawler that's demonstrably causing reliability
problems for real users right now, not for one you're merely annoyed at
seeing in the logs.
Proof-of-work challenges are a real, separate category. Some smaller
open-source infrastructure projects have adopted tools that put a lightweight
computational challenge in front of a request before serving the page — cheap
enough that a normal browser clears it invisibly, expensive enough that
sustaining a bulk-scraping pattern across thousands of pages gets
computationally costly for whoever is running the scraper. This is a real and
growing defense worth knowing about by mechanism, whether or not you adopt
it — it's a different lever from a robots.txt rule or a firewall block,
because it changes the cost calculus for the crawler operator rather than
just refusing the request.
Match the response to the actual problem. The uncomfortable truth is that
"should I block AI crawlers" doesn't have one answer, because it's two
different questions wearing the same sentence. A site chasing citation
eligibility should not blanket-apply advice written for a bandwidth crisis —
"just block everything AI-labeled" throws away exactly what you're trying to
get. A site with a genuine bandwidth or reliability problem should not feel
obligated to stay maximally open just because citation matters to sites in
general — if the traffic is actively degrading service for real users, that's
a legitimate reason to throttle or block specific offenders, full stop. Check
our crawler cheat sheet for what each bot
actually is before deciding which category you're in.
Is AI crawler traffic actually costing sites money?
Yes, documented directly. The Wikimedia Foundation's own engineering blog
reported a 50% surge in bandwidth for multimedia downloads from Wikimedia
Commons since January 2024, attributed to AI-training scrapers rather than
human readers, with bots responsible for roughly 65% of its most
resource-intensive traffic despite being about 35% of pageviews. Smaller
sites — SourceHut, iFixit, Read the Docs, and independent developer Dennis
Schubert among them — have separately reported the same pattern, without
Wikimedia's level of published detail.
Should I block AI crawlers to save bandwidth?
Only if you have a genuine, demonstrated bandwidth or reliability problem,
and rate-limiting is usually the better first move over an outright block.
Throttling a crawler's request rate reduces load while letting the crawl
still complete, which keeps you eligible for citation in that crawler's
index. A hard block trades your bandwidth problem for a visibility problem —
worth it if the traffic is actually degrading your site for real users, not
worth it if you're just trying to reduce a number in a dashboard.
Why does bot traffic cost more than the same amount of human traffic?
Because of what gets requested, not just how much. Wikimedia's own
explanation is that crawler bots "bulk read" through large numbers of pages,
including rarely-visited ones that were never popular enough to be cached —
so each of those requests goes back to origin infrastructure instead of
being served from a CDN edge cache. Human traffic clusters around popular,
already-cached pages; bot traffic doesn't, which is why a smaller share of
requests can still account for the majority of the most expensive ones to
serve.
Top comments (0)