Our checkout started throwing intermittent 429s from the gateway during a flash sale. Client retry logic used fixed exponential backoff: 1s, 2s, 4s, 8s, capped at 30s. Looked reasonable on paper.
What we missed: the gateway's rate limiter tracks consecutive throttled requests per merchant key, and every response carried a Retry-After header we weren't reading. Our backoff was shorter than their cooldown window on most attempts, so each retry landed inside the penalty period and got throttled again. Three consecutive 429s bumped the merchant-level cooldown from 1s to 64s. By retry six we were locked out for just over 4 minutes, during peak traffic, with orders queuing behind it.
Fix was almost embarrassingly simple: parse Retry-After, sleep for that exact duration plus a small jitter, and reset our own backoff counter to zero afterward instead of letting it keep climbing on top of theirs. Lockout window dropped to under 10 seconds on the next load test.
The part that still bugs me: nothing in the gateway docs said the cooldown was cumulative per consecutive throttle. We reverse-engineered it from logs after the fact.
Anyone else had a rate limiter punish you harder because your own retry logic looked like abuse from the other side?
Top comments (2)
Yes, and the payments version is worse than a lockout, because 429 is the friendly failure.
The cooldown punished you. What it did not do is take the money twice. A gateway that accepts the retry instead of throttling it gives you a second authorisation on the same order, and nothing in your logs looks like an incident until the customer sees two pending amounts. So I would read your four minutes as the good outcome: the rate limiter was the only thing standing between a retry storm and duplicate charges, and it was doing its job loudly.
The line I would pull out is resetting your own counter rather than letting it climb on top of theirs. Two backoff algorithms composed do not add, they multiply, and the fix is not tuning either one. It is deciding that only one side owns the clock. Once Retry-After is present, yours should not exist.
And the part that still bugs you is a permanent condition rather than a documentation gap. An undocumented cooldown is a number the vendor can change on a Tuesday, so the value you reverse-engineered from logs has a shelf life. Worth writing it down with the date you measured it and re-measuring on a schedule, because a stale number that looks authoritative is harder to debug than no number at all.
Did the cooldown turn out to be per merchant key only, or does it also track the source IP? That changes whether adding a second worker helps you or doubles your problem.
Fair point on the 429 being the friendly failure. The dangerous case is a timeout or a 5xx, where the first authorisation may have gone through and a retry creates a second one. That's why every order attempt carries an idempotency key, so a retry on the same order can't turn into a second charge. The 429 incident was loud, but it was never going to double-charge anyone.
"Only one side owns the clock" is a better way to say it than we managed in the post. Once Retry-After is present, our own backoff now stays out of it completely.
Good call on dating the number. The 1s → 64s escalation is now written down with the date we measured it, and it's on the list to re-check, since nothing in the docs guarantees it.
On your question: as far as we can tell from the logs, it's per merchant key only. Spreading retries across workers didn't reset the cooldown, so a second worker would just burn through the throttle budget twice as fast. We haven't tested it with a dedicated IP-change experiment, though, so take that as observed, not confirmed.