Block bots, not users: rate limiting an open API

If your API can sit behind a login, rate limiting is easy: meter the account, done.

Ours can't. AI coding agents call it on a first visit, without signing up. Search engines and AI crawlers are supposed to read our pages. And we get scrapers, signup bots and cloud swarms like everyone else. Over four months we blocked the wrong people, let the wrong ones through, and once blocked Googlebot from our own pages.

These are the eight patterns we ended up with. Each has the code shape and the mistake that taught it.

1. Meter the identity, fall back to the IP

The bug: our first limit was 60 requests a minute per IP. One AI platform's shared egress crossed it with ordinary traffic, and every user behind that address was blocked at once. Corporate NAT and mobile carriers do the same thing: thousands of people, one IP.

The pattern: pick exactly one metering key per request. An API key if there is one, the IP only if there isn't.

def meter_key(request) -> str:
    principal = authenticate(request)          # None for anonymous callers
    if principal:
        return f"user:{principal.user_id}"     # never refused because of its IP
    return f"ip:{client_ip(request)}"

Two details that matter:

• A keyed caller is never blocked by an IP rule. Otherwise a single bad neighbour on a shared egress locks out your paying users.
• Make a block last its full window. If crossing the per-minute limit only refuses until the minute rolls over, a bot learns to send exactly the limit. A block of 10 minutes, an hour, or a day (one per window) doesn't reward that.

2. Separate reading from computing

Not all routes cost the same, and not all bots are unwelcome. We want crawlers on our pages: that is how people find us, and how AI answers end up citing us. We don't want them running package scans.

The pattern: classify routes, then decide per class.

FEATURE_ROUTES = {"/v1/query", "/v1/search", "/v1/ai/check-packages", "/v1/sbom/scan"}

async def bot_gate(request, call_next):
    if is_bot(request.headers.get("user-agent", "")):
        if request.url.path in FEATURE_ROUTES and not is_internal(request):
            return JSONResponse({"error": "crawler_not_allowed"}, status_code=403)
        return await call_next(request)        # bots reading pages: never metered
    return await call_next(request)

The bug that is_internal fixes: our page server renders pages by calling our own API, and forwards the visitor's user agent so analytics see the real client. So when Googlebot loaded a page, the page's own API call was refused as "a crawler calling the API". Browsers got 16,129 bytes; Googlebot got a 6,147-byte empty shell. A site: search for our domain returned one page.

The exemption is for calls authenticated with an internal service token, not for anything a user can send. A user's API key doesn't buy it.

Check this in yours: if you block by user agent, grep for every place your own infrastructure forwards one.

3. Classify bots precisely, or your dashboard lies

You can't decide what to block without knowing what you're looking at.

The bug: one SEO crawler wasn't on our bot list, so it was counted as a browser. A four-hour burst of 11,527 requests from a single IP looked like a surge of human visitors.

The pattern: a list of known bots for labelling, plus a catch-all so nothing unlisted falls into "human".

import re

AI_CRAWLERS  = ("gptbot", "claudebot", "perplexitybot", "ccbot")
SEARCH_BOTS  = ("googlebot", "bingbot", "duckduckbot")
CATCH_ALL    = re.compile(r"(?<!cu)bot\b|crawler|spider|scraper|headlesschrome", re.I)

def classify(ua: str) -> str:
    u = ua.lower()
    if any(b in u for b in AI_CRAWLERS):  return "ai-crawler"
    if any(b in u for b in SEARCH_BOTS):  return "search-bot"
    if CATCH_ALL.search(u):               return "other-bot"
    return "browser"

The (?<!cu) is real: a phone brand, CUBOT, puts "cubot" in its browser's user agent, and bot\b would have counted those people as bots.

Split AI crawlers from SEO bots on purpose. An AI company indexing you is a signal worth seeing; most SEO crawlers are noise.

4. Use robots.txt for noise, not for defence

Some crawlers cost you and send nobody back. On one day a single SEO crawler made 11,548 of about 14,400 requests to our site.

robots.txt is the right first response: it's free, and well-behaved crawlers obey it.

User-agent: SomeSeoBot
Disallow: /

User-agent: *
Allow: /
Disallow: /account
Crawl-delay: 1

It's not a security control. It only stops bots that choose to read it, which is why pattern 2 doesn't rely on it. Use it to cut cost, and check the next day's traffic to confirm it worked.

5. Group cloud swarms by who owns the network

The bug: roughly a thousand cloud IPs, each making only a handful of calls. Every one stayed far below its own limit. Together they were most of our anonymous load.

The pattern: for anonymous traffic from datacenter ranges only, add a second meter keyed by the network owner (the organisation or ASN that announces the IP), with a daily ceiling.

def anon_meter_keys(ip: str) -> list[str]:
    keys = [f"ip:{ip}"]
    info = ip_info(ip)                         # your GeoIP/ASN source, cached
    if info.is_hosting:                        # cloud/datacenter, not home or mobile
        keys.append(f"org:{info.org}")         # one budget for the whole provider
    return keys

Keep it narrow: home and mobile connections don't get the org meter, and keyed callers never do. And make it a soft cap that refuses for the rest of the day, without placing a block that outlives the burst.

6. The signup form is your biggest door

You can tune API limits for days while the form that hands out an unlimited key has none. Ours did. What came through:

• Six accounts in six hours, each a real person's Gmail address with dots scattered through it. Gmail ignores dots, so [e-mail] is [e-mail]: six strings, six strangers' inboxes from a harvested list.
• Nineteen accounts over a day, rotating mail providers. Zero API calls.
• Later, 136 sign-ups from 118 IPs. Zero verified.

That last line is the real lesson. The bots weren't using the keys. The harm was verification mail sent to people who never asked. A few spam reports on a young sending domain, and your mail stops arriving for the people who did sign up.

Patterns:

PLUS_TAG = {"gmail.com", "googlemail.com", "outlook.com", "hotmail.com",
            "yahoo.com", "icloud.com", "fastmail.com", "proton.me"}
DOTLESS  = {"gmail.com", "googlemail.com"}

def canonical_email(addr: str) -> str:
    """For uniqueness checks only. Always mail what the person typed."""
    local, _, domain = addr.strip().lower().rpartition("@")
    domain = "gmail.com" if domain == "googlemail.com" else domain
    if domain in PLUS_TAG:
        local = local.split("+", 1)[0]
    if domain in DOTLESS:
        local = local.replace(".", "")        # dots are only meaningless at Gmail
    return f"{local}@{domain}"

• Store the canonical form in its own column with a UNIQUE index. Don't fold dots globally: at Outlook, first.last and firstlast are two different people.
• Give signup the same per-IP budget as any other way to get a key. Ours had one on "request a key by email" and none on the form next to it.
• Cap verification mail service-wide, so no burst can turn your domain into someone's mailer.
• Verify with a button, not a link. Corporate mail scanners open every link before the person does, so a GET link "verifies" accounts nobody looked at. A POST from a page with a button doesn't fire on a scan.
• Add cheap friction to the form, and be honest about it: it raises the cost, but a patient headless browser still passes. The mail cap is what limits the harm.

7. Make every 429 actionable

A bare 429 Too Many Requests makes a human wait and an agent retry, guess, or skip the check. For a security API, "carry on without checking" is the worst possible outcome.

The pattern: say why, until when, and what to do, in fields a program can read.

{
  "error": "rate_limited",
  "axis": "ip",
  "expires_at": "2026-10-13T09:10:00Z",
  "agent_action": "REFUSE",
  "because": "Anonymous hourly budget used up. Proceeding unchecked is not safe.",
  "next_step": "Ask the user for an email address and request a free key.",
  "signup_url": "https://example.com/signup"
}

Plus a Retry-After header. If your callers include AI agents, add one explicit line: do not invent a key, do not proceed without the check.

And fail open on your own outages. If the rate-limit store (Redis, for us) is down, allow the request. Under-blocking for five minutes beats refusing every legitimate user because a cache restarted.

def check(key: str) -> bool:
    try:
        return redis_window_ok(key)
    except RedisError:
        return True     # our outage is not the caller's fault

8. Measure before you block, and after

Most of our bad calls came from numbers that were quietly wrong:

• Don't audit blocked requests as normal traffic. We did, so a bot hammering a block kept the 429 count climbing, and the block looked like it wasn't working.
• Make every report count the same way. Our daily digest and our dashboard disagreed on the day one crawler made 80% of requests.
• Know which limits are live. Our code defaults said one thing; a deployment config overrode them, and production ran at 10 requests a minute per anonymous IP while our docs promised much more.
• Measure what closing a door costs. We once put every endpoint behind a key. Seven days later: one new account, twelve calls. The users we built for, agents evaluating a tool on a first visit, left at the signup page. We reopened two endpoints with a per-IP quota, counted in the unit that actually costs us (packages checked, not requests, after a 1,168-package run died at request 20).

The checklist

• [ ] One metering key per request: account if keyed, IP if not.
• [ ] Blocks last their whole window.
• [ ] Routes split into read and compute; bots allowed on one, refused on the other.
• [ ] Your own servers exempted by an internal token, never by user agent.
• [ ] Bot classification with a catch-all, AI crawlers labelled apart from SEO bots.
• [ ] robots.txt for costly crawlers, verified in the next day's traffic.
• [ ] Anonymous datacenter traffic also metered per network owner.
• [ ] Signup with an IP budget, canonical-email uniqueness, a service-wide mail cap, button verification.
• [ ] Every 429 says why, until when, and what to do next.
• [ ] Fail open when your own limiter is down.
• [ ] Blocked requests counted once, every report counting the same way, documented limits equal to deployed ones.

None of this makes an open API bot-proof. It makes the bots cost more than they're worth, while the people and agents you built it for get through.