You opened your Vercel dashboard last Tuesday and did a double-take. The bandwidth graph looks like a ski slope. The bill is three times what you expected. Your site hasn't gone viral — you'd know about that.
What happened? GPTBot, ClaudeBot, meta-externalagent, and a dozen other AI training crawlers paid you a visit. They didn't ask. They didn't check if they were welcome. And they definitely didn't contribute to your MRR.
This isn't a fringe problem. It's happening to developers every week, and most don't realize it until the invoice arrives.
The Scale of the Problem
Let me share some real numbers from Hacker News threads in the past year:
Read the Docs (the open source documentation hosting service) published a post-mortem describing how they got hit "to the tune of thousands of dollars" by AI crawlers. The AI company that caused the most damage did eventually compensate them — but that required them to notice, document, and escalate the issue.
One developer running a self-hosted git instance reported 37,212,377 requests in 60 days, with >99% from bots. Their nginx access log grew to 12 GiB. The constant crawling caused a baseline 30-50% CPU utilization — just from bots, not from actual users.
Another developer with a small browser game (~120 weekly users) had to put their wiki behind a login wall because AI crawlers were "causing massive amounts of spurious traffic" that crashed the game's MariaDB process multiple times.
And an operator of a "very, very low traffic" site reported OpenAI's crawler hitting it "tens of thousands of times a minute non-stop" until they blocked it via Cloudflare.
These are not enterprise companies with armies of ops engineers. These are indie developers, solo founders, and hobbyists — people running on platforms like Vercel, Fly.io, Railway, and Render where bandwidth and serverless invocations cost real money.
How AI Crawlers Burn Your Budget
Understanding the mechanism helps you defend against it.
Why serverless/edge hosting is especially vulnerable
When you run a traditional server, a bot hammering it is annoying but the marginal cost per request approaches zero after you've paid for the instance.
On serverless platforms, every invocation costs money. Vercel charges per function invocation and GB of bandwidth. Fly.io bills per CPU-second. If a bot hits 10,000 pages of your Next.js app with SSR enabled, that's 10,000 cold-ish function calls, not "traffic to a static file."
One commenter nailed the structural problem:
"Combine inconsiderate crawler (AI or otherwise) with a server-side logic that doesn't really need to be there and you have a recipe for a crash, a big hosting bill, or both."
Why AI crawlers bypass caches
Traditional search crawlers eventually learn to respect ETags and If-Modified-Since headers. Many AI crawlers don't. Several developers have observed that some bots actively avoid caches — they add cache-busting query strings or ignore Vary headers. This means your CDN won't save you. Every request hits origin.
User-Agent strings to know
Here are the actual User-Agent strings you'll see in your logs:
# OpenAI
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)
# Anthropic
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
# Meta
meta-externalagent/1.1 +https://developers.facebook.com/docs/sharing/webmasters/crawler)
# Amazon
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
# Apple
Applebot-Extended/0.1
# Common Data
CCBot/2.0 (https://commoncrawl.org/faq/)
There are currently 30+ distinct AI crawler User-Agents in the wild, and new ones appear regularly as every AI lab builds their own data pipeline.
Does robots.txt Actually Work?
This is where it gets complicated.
In theory, you can block all of these with a simple robots.txt addition:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
The major labs (OpenAI, Anthropic, Google, Amazon) claim to respect robots.txt. In practice, the situation is messier:
- Respecting robots.txt doesn't mean not crawling — it means they'll skip the page for training data but may still hit it for real-time search/summarization features.
- Many crawlers are flat-out lying — security researchers have documented crawlers that claim to be GPTBot or ClaudeBot but don't originate from those companies' IP ranges.
- New crawlers appear faster than blocklists update — by the time you've added the top 10 bots to robots.txt, five new ones have appeared.
One developer put it bluntly:
"It's also basically impossible to block most AI crawlers. Most completely ignore robots.txt. Some claim to respect it but I've found the majority that claim to also ignore it. You have to block their User-Agent and then their IPs if they spoof it and use botnets like some China-based AI crawlers."
Practical Mitigation Strategies
Here's what actually works, in order of increasing effort:
1. Add robots.txt entries (5 minutes, partial protection)
Start here. For the well-behaved crawlers (OpenAI, Anthropic, Google), this works. It won't stop everything but it's zero cost.
2. Block known User-Agents at your CDN (15 minutes, good protection)
If you're on Cloudflare (even the free tier), you can block AI crawler User-Agents with a firewall rule:
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "meta-externalagent") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "Amazonbot") or
(http.user_agent contains "Applebot-Extended")
This stops the honest bots before they hit your origin.
Cloudflare also maintains a managed ruleset specifically for AI bots — you can enable it under Security → Bots → "Block AI Scrapers and Crawlers."
3. Rate limit by ASN (30 minutes, strong protection)
Most AI crawlers run from major cloud provider IP ranges. Aggressively rate limiting requests from AWS, GCP, Azure, and Meta's ASNs cuts off the majority of well-funded crawlers. You'll occasionally catch a legitimate developer testing from cloud IPs, but for most personal projects the tradeoff is worth it.
4. Serve static where possible (architectural, maximum protection)
The most bulletproof solution: export your site as static HTML so bot traffic has no marginal cost. If you're using Next.js and don't actually need SSR, the output: 'export' option in next.config.js makes your site immune to per-invocation charges regardless of bot traffic.
First Step: Know What's Already Happening
Before you build defenses, you need to know your exposure. Most developers don't realize how many AI crawlers have free access to their sites right now.
I built a free tool to check this: crawl-check
It checks your site's robots.txt against all major AI crawler User-Agents and tells you which ones are currently blocked and which have unrestricted access. Takes about 10 seconds, no signup required.
Try it on your current domain — you might be surprised.
The Bigger Picture
This isn't going away. The number of AI crawlers is increasing, not decreasing, as every company building an LLM needs fresh training data. The economic incentives are completely misaligned: the crawlers capture value (training data) while you absorb the cost (bandwidth, compute).
The developers most affected are the ones with the least resources to fight back — small indie projects, side projects, documentation sites, personal blogs. The same sites that often have the highest-quality, most useful content.
If you want to stay ahead of this:
- Set up alerts on your hosting platform for unusual bandwidth/invocation spikes
- Audit your
robots.txtregularly as new bots appear - Consider CrawlGuard (currently in waitlist) for ongoing monitoring: crawlguard.user-sentry.com
- Check Cloudflare's block AI bots documentation if you use their CDN
The bots aren't going anywhere. But you can at least make sure they're not running up your bill without your permission.
Found this useful? Check out crawl-check — a free tool to see which AI crawlers have access to your site right now.
Top comments (0)