I had a /image endpoint capped at 3 requests per minute. Simple express-rate-limit, keyed by IP, standard stuff. I was proud of it, honestly — felt like the responsible thing to do before shipping.
Then I actually looked at where those 3 requests per minute were going.
The assumption I didn't know I was making
Rate limiting by IP protects against one thing: a single user hammering your endpoint. It assumes the thing you're protecting has infinite capacity, and you're just being polite about how fast any one person can use it.
That assumption falls apart the second your endpoint calls a third-party API that has its own rate limit — because that limit isn't per-user. It's per-account. Per-project. Shared across every single person hitting your backend, whether they know each other exist or not.
My /image endpoint calls two different image generation providers depending on the model requested. Both of them enforce rate limits at the account level:
Provider A: 5 images per minute, account-wide
Provider B: 10 images per minute, account-wide
My limiter was capping each IP address at 3 requests per minute. Which sounds conservative, until you do the math: five different users on five different IPs can each stay comfortably under their personal cap of 3/min while collectively sending 15 requests per minute to a provider that only allows 5.
Every one of those users would see my API return success. Then, somewhere between my server and the provider, requests would start silently failing — not because of anything they did, but because of what everyone else was doing at the same time, completely invisibly to them and to me.
Why this is easy to miss
Rate limiting per IP feels correct. It's the default in every tutorial, every middleware library's example code, every "add rate limiting to your API" blog post. It solves a real problem — it's just the wrong problem when the actual constraint sits somewhere else entirely.
The bug doesn't show up in testing. It shows up exactly once you have enough concurrent users for their individual, well-behaved usage to add up to something that isn't well-behaved anymore. Which means the more successful you get, the more likely you are to hit it — right when you can least afford flaky behavior.
What actually fixes it
You need a limiter that reflects the real constraint: a shared counter across every user, not per-IP, checked against the actual provider ceiling before you ever make the call.
javascript
const providerRequestLog = {
providerA: [],
providerB: []
}
const PROVIDER_LIMITS = {
providerA: 5, // per minute, account-wide
providerB: 10
}
function checkCapacity(provider) {
const now = Date.now()
const windowStart = now - 60_000
const log = providerRequestLog[provider]
// drop anything outside the rolling window
while (log.length > 0 && log[0] < windowStart) {
log.shift()
}
if (log.length >= PROVIDER_LIMITS[provider]) {
return { allowed: false }
}
log.push(now)
return { allowed: true }
}
Before calling the provider, you check capacity against the shared log — not against a per-IP bucket. If there's no room left, you reject before spending the call, with a clear Retry-After, instead of finding out from a 429 you didn't see coming from the provider itself.
One caveat worth flagging honestly: this is in-memory, so it only works cleanly on a single process. If you're running multiple instances behind a load balancer, this counter needs to live somewhere shared — Redis, or similar — or each instance ends up enforcing its own version of a limit that's supposed to be global.
The actual lesson
Rate limiting isn't one thing. "Protect against abuse from a single user" and "don't exceed a ceiling shared across all your users" are two different problems that happen to look identical in a middleware config file. Solving the first one doesn't solve the second, and most default setups only ever solve the first.
If your backend depends on any third-party API with account-wide quotas — payment providers, AI providers, anything with a per-project rate limit — it's worth checking whether your rate limiting actually reflects that shared ceiling, or if it's quietly protecting against something that was never the real risk.
Top comments (1)
Great catch. The fix you landed on has one more layer that bites the same way: that providerRequestLog object lives inside a single Node process, so the moment you run more than one instance behind a load balancer — or the platform autoscales you to three pods — each process keeps its own count and you're back to N× the provider ceiling, the exact bug you just killed re-emerging one level up.
The thing to watch when you move the counter out of process: your in-memory version is race-free for free (synchronous JS on one thread never interleaves the read and the push), but a naive Redis port with GET-then-compare-then-SET loses that — two instances both read 4, both write 5, and you've overshot the limit again. So the shared version wants the increment to be atomic: INCR-then-check with an EXPIRE for the window, or a sorted-set sliding window behind a small Lua script.
And whatever ceiling you compute locally, treat the provider's own 429 as ground truth and back off on it — your counter and theirs always drift a bit (retries in flight, clock skew), and when they disagree the provider wins.