One of our integration partners pushed a firmware update to their fleet of devices, and every device in that fleet decided to sync at once. Our API request volume from that single partner went from roughly 40 requests a second to just over 1,600 in the space of about ninety seconds. Nothing was malicious about it — it was a retry storm caused by a config change on their side, the kind of thing that happens to every API provider eventually. What made it interesting for us wasn't the spike itself, it was watching which parts of our rate limiting held and which parts didn't.
We'd had rate limiting in production for over a year, but it was a fixed-window counter: X requests per client per 60-second window, reset on the minute. Fixed windows have a well-known flaw that we'd read about and shrugged off as theoretical — you can get up to 2x your limit in a short burst if it straddles the window boundary, because a client can spend its full quota in the last second of one window and its full quota again in the first second of the next. What we hadn't accounted for was what happens when a client isn't trying to burst but genuinely has 40x normal traffic sustained for over a minute. The fixed window doesn't smooth that out at all — it just rejects everything past the limit with a hard 429, uniformly, the instant the counter fills. For a partner with real, if misconfigured, traffic, that meant near-total request failure for the worst two minutes of the incident, plus a wall of retries hitting us right as we were trying to recover.
Why token buckets fit this problem better
A token bucket holds a maximum number of tokens, refills at a steady rate, and every request costs one token. The bucket depth (max tokens) controls how much burst you tolerate; the refill rate controls your actual sustained throughput limit. We sized ours so normal traffic never touches the ceiling, short bursts (the kind real clients produce constantly — a page load firing five API calls at once) drain the bucket but don't empty it, and sustained overload drains it and then throttles smoothly at the refill rate instead of hard-cutting at a window boundary. The partner's spike still got rate limited — that's the point — but it degraded gracefully into a steady trickle of allowed requests instead of alternating between "everything succeeds" and "everything 429s."
Per-client buckets, not one global bucket
Our first implementation used a single bucket per API key, which was right, but we'd initially discussed a simpler global bucket per endpoint to save on Redis calls. That would have meant one partner's spike could throttle every other partner hitting the same endpoint. We kept per-key buckets and ate the extra Redis round-trip; a shared cluster with pipelining made this a non-issue in practice, and it meant the incident was fully contained to the one partner who caused it. Nobody else even saw elevated error rates.
429s need a Retry-After header, or your rate limiting fights your clients' retry logic instead of cooperating with it. We were already setting this, but the incident showed us our retry-after values were too conservative — we were telling clients to back off longer than the bucket actually needed to refill, which meant well-behaved clients were retrying later than necessary and (ironically) sometimes bunching up their retries into new bursts. Tightening the header to match the real refill math smoothed out a secondary wave of thundering-herd retries we hadn't anticipated.
We also learned we needed a way to talk to the partner directly, not just rate limit them. Rate limiting protected us, but the partner's own dashboards were showing them a wall of failed syncs with no obvious explanation. We added a lightweight webhook we can fire at a partner's registered ops contact when they're being sustained-rate-limited for more than a few minutes, with the current bucket state and refill rate included. It turned what used to be a confused support ticket into a five-minute conversation where their engineer just fixed the retry config on their end.
None of this stopped the spike from happening — that's not really rate limiting's job. What it did was turn a 40x traffic surge from an incident that degraded the platform for everyone into one that was contained, visible, and self-correcting within a couple of minutes. That containment is really the whole value proposition of rate limiting done right: not preventing bad traffic, but making sure it can't take down the good traffic sitting next to it.
We treat this kind of isolation as a baseline requirement now for anything at Edilec that serves multiple external partners on shared infrastructure — you can read more about how we think about API reliability at edilec.com.
Top comments (0)