Our checkout started throwing intermittent 429s from the gateway during a flash sale. Client retry logic used fixed exponential backoff: 1s, 2s, 4s, 8s, capped at 30s. Looked reasonable on paper.
What we missed: the gateway's rate limiter tracks consecutive throttled requests per merchant key, and every response carried a Retry-After header we weren't reading. Our backoff was shorter than their cooldown window on most attempts, so each retry landed inside the penalty period and got throttled again. Three consecutive 429s bumped the merchant-level cooldown from 1s to 64s. By retry six we were locked out for just over 4 minutes, during peak traffic, with orders queuing behind it.
Fix was almost embarrassingly simple: parse Retry-After, sleep for that exact duration plus a small jitter, and reset our own backoff counter to zero afterward instead of letting it keep climbing on top of theirs. Lockout window dropped to under 10 seconds on the next load test.
The part that still bugs me: nothing in the gateway docs said the cooldown was cumulative per consecutive throttle. We reverse-engineered it from logs after the fact.
Anyone else had a rate limiter punish you harder because your own retry logic looked like abuse from the other side?
Top comments (0)