I had a /image endpoint capped at 3 requests per minute. Simple express-rate-limit, keyed by IP, standard stuff. I was proud of it, honestly — felt...
For further actions, you may consider blocking this person and/or reporting abuse
The "two problems that look identical in a middleware config" framing nails it — and it extends to the response, not just the counter. For a single abuser, rejecting with 429 is correct; they should slow down. But against a shared account ceiling the user you reject did nothing wrong — they just showed up while strangers were busy. There a short bounded queue against the shared counter (admit when a slot frees, surface the wait as Retry-After) beats a hard reject, so well-behaved users stop eating random failures from load they can't see.
One more from the consuming side: if you hold multiple keys/projects with one provider, the ceiling is per-key, not global — a single global bucket quietly under-uses your real quota. We schedule ~30 upstream feed polls against per-key buckets for exactly that reason.
That's a really good point.
I hadn't thought about the UX difference between rejecting an abusive client vs rejecting a well-behaved user because of a shared upstream limit. A short queue with a bounded wait definitely makes more sense in the second case.
And you're absolutely right about multiple API keys too, once you start distributing traffic across projects, a single global bucket becomes unnecessarily conservative.
Really appreciate you adding this
hadn't even considered the per-key vs global distinction. If someone's running multiple projects against the same provider, a single global counter would just be leaving quota on the table. And you're right that a hard reject punishes someone who did nothing wrong except show up at a busy moment, a short queue is a much more honest way to handle that.
Honest caveat to my own queue suggestion, since it's easy to oversell: bound it in time, not just length. A queue that only caps depth will still happily admit someone into a 40-second wait when the provider's backed up — and at that point a fast 429 they can retry themselves is kinder than a slot they'll just abandon. The rule that worked for us: reject up front whenever the projected wait exceeds the Retry-After you'd actually be willing to promise. The queue then absorbs bursts but sheds sustained overload instead of hiding it.
And one trap on the per-key side, because it's the same "ceiling one level up" this whole post is about: some providers enforce a per-key limit and an account-wide aggregate above it. Sharded per-key buckets keep each individual key legal but can still collectively blow the account tier — so the counter sometimes has to be two-level, one bucket per key plus one for the account. Same bug as the article, just wearing a different hat.
This is a good one because it's the failure that only shows up once you're winning. One layer I'd add, from running this in front of payment and AI providers: decide whether you're counting attempts or successes, because they drift. If you increment before the call and it times out, you don't actually know whether the provider counted it. A lot of gateways count the request the moment it lands, even the ones that 5xx back. Decrement on failure and a call that did reach them lets you overshoot; don't decrement and a burst of timeouts makes your limiter think you're full when you're not. We ended up treating the provider's own rate-limit headers (remaining/reset) as the source of truth and using the local counter only to fail fast between calls. Which way did you lean, count on send or count on confirmed response?
to answer directly, I counted on send, not confirmed response. Mostly just because it was the simpler thing to ship at the time, not some deliberate tradeoff I'd thought through. Framed the way you did it, I don't think I can pretend it was neutral. Might lean toward trusting the provider's own rate limit headers going forward instead.
This is exactly why I like posting the messy version of these things instead of waiting until I've "solved" it properly.
Great catch. The fix you landed on has one more layer that bites the same way: that providerRequestLog object lives inside a single Node process, so the moment you run more than one instance behind a load balancer — or the platform autoscales you to three pods — each process keeps its own count and you're back to N× the provider ceiling, the exact bug you just killed re-emerging one level up.
The thing to watch when you move the counter out of process: your in-memory version is race-free for free (synchronous JS on one thread never interleaves the read and the push), but a naive Redis port with GET-then-compare-then-SET loses that — two instances both read 4, both write 5, and you've overshot the limit again. So the shared version wants the increment to be atomic: INCR-then-check with an EXPIRE for the window, or a sorted-set sliding window behind a small Lua script.
And whatever ceiling you compute locally, treat the provider's own 429 as ground truth and back off on it — your counter and theirs always drift a bit (retries in flight, clock skew), and when they disagree the provider wins.
you called out the honest gap in the post before I even had to admit it myself, the in-memory version is one extra instance away from bringing back the exact bug I just described. Kicking myself a little for not mentioning that more clearly. Atomic INCR+EXPIRE makes sense, and yeah, trusting the provider's own 429 over my local count is probably the right call either way.
Adjacent one, same family of "the provider's counter is not the counter you modelled": on some providers the budget is charged by the
max_tokensyou declare, not by what the model generates.Free tier, 8,000 tokens/minute. A 20-token prompt with
max_tokens: 8192comes back 413 - "Limit 8000, Requested 8271" - before a single token is generated. Same model,max_tokens: 16with a 4,078-token prompt: 200 OK. The request that spends the budget is not the request that uses it.Which sharpens @mickyarun's attempts-vs-successes point: counting on send is still wrong if one send can reserve 8k units. And it isn't per-model the way it first looks - the identical request passed on one probe and 413'd a minute later ("Used 4373, Requested 6278"), so it's a shared rolling window.
Rate limiters are policy, not just middleware. IP-based limits are simple until the unit of abuse is an account, tenant, endpoint, workflow, or expensive downstream call. The safest designs usually limit the thing that creates cost or risk, not just the request source.
Rate limiters are a good example of correct code with the wrong product model. A limit by IP or route can satisfy the config and still miss the abuse pattern. The useful question is what scarce resource or user promise the limiter is actually protecting.