I had a /image endpoint capped at 3 requests per minute. Simple express-rate-limit, keyed by IP, standard stuff. I was proud of it, honestly — felt like the responsible thing to do before shipping.
Then I actually looked at where those 3 requests per minute were going.
The assumption I didn't know I was making
Rate limiting by IP protects against one thing: a single user hammering your endpoint. It assumes the thing you're protecting has infinite capacity, and you're just being polite about how fast any one person can use it.
That assumption falls apart the second your endpoint calls a third-party API that has its own rate limit — because that limit isn't per-user. It's per-account. Per-project. Shared across every single person hitting your backend, whether they know each other exist or not.
My /image endpoint calls two different image generation providers depending on the model requested. Both of them enforce rate limits at the account level:
Provider A: 5 images per minute, account-wide
Provider B: 10 images per minute, account-wide
My limiter was capping each IP address at 3 requests per minute. Which sounds conservative, until you do the math: five different users on five different IPs can each stay comfortably under their personal cap of 3/min while collectively sending 15 requests per minute to a provider that only allows 5.
Every one of those users would see my API return success. Then, somewhere between my server and the provider, requests would start silently failing — not because of anything they did, but because of what everyone else was doing at the same time, completely invisibly to them and to me.
Why this is easy to miss
Rate limiting per IP feels correct. It's the default in every tutorial, every middleware library's example code, every "add rate limiting to your API" blog post. It solves a real problem — it's just the wrong problem when the actual constraint sits somewhere else entirely.
The bug doesn't show up in testing. It shows up exactly once you have enough concurrent users for their individual, well-behaved usage to add up to something that isn't well-behaved anymore. Which means the more successful you get, the more likely you are to hit it — right when you can least afford flaky behavior.
What actually fixes it
You need a limiter that reflects the real constraint: a shared counter across every user, not per-IP, checked against the actual provider ceiling before you ever make the call.
javascript
const providerRequestLog = {
providerA: [],
providerB: []
}
const PROVIDER_LIMITS = {
providerA: 5, // per minute, account-wide
providerB: 10
}
function checkCapacity(provider) {
const now = Date.now()
const windowStart = now - 60_000
const log = providerRequestLog[provider]
// drop anything outside the rolling window
while (log.length > 0 && log[0] < windowStart) {
log.shift()
}
if (log.length >= PROVIDER_LIMITS[provider]) {
return { allowed: false }
}
log.push(now)
return { allowed: true }
}
Before calling the provider, you check capacity against the shared log — not against a per-IP bucket. If there's no room left, you reject before spending the call, with a clear Retry-After, instead of finding out from a 429 you didn't see coming from the provider itself.
One caveat worth flagging honestly: this is in-memory, so it only works cleanly on a single process. If you're running multiple instances behind a load balancer, this counter needs to live somewhere shared — Redis, or similar — or each instance ends up enforcing its own version of a limit that's supposed to be global.
The actual lesson
Rate limiting isn't one thing. "Protect against abuse from a single user" and "don't exceed a ceiling shared across all your users" are two different problems that happen to look identical in a middleware config file. Solving the first one doesn't solve the second, and most default setups only ever solve the first.
If your backend depends on any third-party API with account-wide quotas — payment providers, AI providers, anything with a per-project rate limit — it's worth checking whether your rate limiting actually reflects that shared ceiling, or if it's quietly protecting against something that was never the real risk.
Top comments (10)
The "two problems that look identical in a middleware config" framing nails it — and it extends to the response, not just the counter. For a single abuser, rejecting with 429 is correct; they should slow down. But against a shared account ceiling the user you reject did nothing wrong — they just showed up while strangers were busy. There a short bounded queue against the shared counter (admit when a slot frees, surface the wait as Retry-After) beats a hard reject, so well-behaved users stop eating random failures from load they can't see.
One more from the consuming side: if you hold multiple keys/projects with one provider, the ceiling is per-key, not global — a single global bucket quietly under-uses your real quota. We schedule ~30 upstream feed polls against per-key buckets for exactly that reason.
That's a really good point.
I hadn't thought about the UX difference between rejecting an abusive client vs rejecting a well-behaved user because of a shared upstream limit. A short queue with a bounded wait definitely makes more sense in the second case.
And you're absolutely right about multiple API keys too, once you start distributing traffic across projects, a single global bucket becomes unnecessarily conservative.
Really appreciate you adding this
hadn't even considered the per-key vs global distinction. If someone's running multiple projects against the same provider, a single global counter would just be leaving quota on the table. And you're right that a hard reject punishes someone who did nothing wrong except show up at a busy moment, a short queue is a much more honest way to handle that.
Honest caveat to my own queue suggestion, since it's easy to oversell: bound it in time, not just length. A queue that only caps depth will still happily admit someone into a 40-second wait when the provider's backed up — and at that point a fast 429 they can retry themselves is kinder than a slot they'll just abandon. The rule that worked for us: reject up front whenever the projected wait exceeds the Retry-After you'd actually be willing to promise. The queue then absorbs bursts but sheds sustained overload instead of hiding it.
And one trap on the per-key side, because it's the same "ceiling one level up" this whole post is about: some providers enforce a per-key limit and an account-wide aggregate above it. Sharded per-key buckets keep each individual key legal but can still collectively blow the account tier — so the counter sometimes has to be two-level, one bucket per key plus one for the account. Same bug as the article, just wearing a different hat.
This is a good one because it's the failure that only shows up once you're winning. One layer I'd add, from running this in front of payment and AI providers: decide whether you're counting attempts or successes, because they drift. If you increment before the call and it times out, you don't actually know whether the provider counted it. A lot of gateways count the request the moment it lands, even the ones that 5xx back. Decrement on failure and a call that did reach them lets you overshoot; don't decrement and a burst of timeouts makes your limiter think you're full when you're not. We ended up treating the provider's own rate-limit headers (remaining/reset) as the source of truth and using the local counter only to fail fast between calls. Which way did you lean, count on send or count on confirmed response?
to answer directly, I counted on send, not confirmed response. Mostly just because it was the simpler thing to ship at the time, not some deliberate tradeoff I'd thought through. Framed the way you did it, I don't think I can pretend it was neutral. Might lean toward trusting the provider's own rate limit headers going forward instead.
This is exactly why I like posting the messy version of these things instead of waiting until I've "solved" it properly.
Great catch. The fix you landed on has one more layer that bites the same way: that providerRequestLog object lives inside a single Node process, so the moment you run more than one instance behind a load balancer — or the platform autoscales you to three pods — each process keeps its own count and you're back to N× the provider ceiling, the exact bug you just killed re-emerging one level up.
The thing to watch when you move the counter out of process: your in-memory version is race-free for free (synchronous JS on one thread never interleaves the read and the push), but a naive Redis port with GET-then-compare-then-SET loses that — two instances both read 4, both write 5, and you've overshot the limit again. So the shared version wants the increment to be atomic: INCR-then-check with an EXPIRE for the window, or a sorted-set sliding window behind a small Lua script.
And whatever ceiling you compute locally, treat the provider's own 429 as ground truth and back off on it — your counter and theirs always drift a bit (retries in flight, clock skew), and when they disagree the provider wins.
you called out the honest gap in the post before I even had to admit it myself, the in-memory version is one extra instance away from bringing back the exact bug I just described. Kicking myself a little for not mentioning that more clearly. Atomic INCR+EXPIRE makes sense, and yeah, trusting the provider's own 429 over my local count is probably the right call either way.
Rate limiters are policy, not just middleware. IP-based limits are simple until the unit of abuse is an account, tenant, endpoint, workflow, or expensive downstream call. The safest designs usually limit the thing that creates cost or risk, not just the request source.
Rate limiters are a good example of correct code with the wrong product model. A limit by IP or route can satisfy the config and still miss the abuse pattern. The useful question is what scarce resource or user promise the limiter is actually protecting.