Rate-Limiting Our Cross-Border Checkout API With a Token Bucket
Black Friday taught us an expensive lesson. Our checkout endpoint, which normally hums along, got hammered by a mix of real buyers and retry scripts. Some of those retries were our own fault — a client that re-posted on a timeout — and each one triggered a fresh round of inventory checks and payment intent calls. We needed a limiter that could tell a burst from an abuse. Taocarts is the platform behind this checkout, and this is how we tuned its front door.
Why a fixed window was not enough
Our first attempt was the obvious one: a fixed counter per user, reset every minute. It was easy, and it was wrong in two ways.
First, fixed windows have an edge problem. A client can send N requests at the end of one window and N more at the start of the next — 2N in a few seconds, right at the boundary. Second, a hard cap of N per minute punishes legitimate bursts. A buyer refreshing the page a dozen times while comparing shipping options is not an attacker.
What we wanted was: allow short bursts, but keep the average rate under control. That is exactly what a token bucket gives you.
The model
Picture a bucket per key (we use tenant id plus endpoint). The bucket holds tokens, up to a capacity C. Every request consumes one token. Tokens refill continuously at rate R per second.
- A burst is fine as long as the bucket has tokens — that is what capacity C buys you.
- Sustained abuse drains the bucket faster than R refills it, and requests start getting rejected.
- After the abuse stops, the bucket refills and normal traffic resumes without a manual reset.
Two parameters do all the work: capacity C controls how big a burst we tolerate; refill rate R controls the long-term average.
Implementation sketch
We run Redis, so the bucket lives there. The clever part is that we do not run a background timer to refill tokens. We compute lazily: when a request arrives, we work out how many tokens would have accrued since the last request, based on elapsed time.
def allow(key, capacity=20, refill_rate=0.5):
now = redis.time()
tokens, last = redis.hmget(key, "tokens", "last")
if tokens is None:
tokens, last = capacity, now
elapsed = max(0, now - last)
tokens = min(capacity, float(tokens) + elapsed * refill_rate)
if tokens < 1:
redis.hset(key, mapping={"tokens": tokens, "last": now})
return False
redis.hset(key, mapping={"tokens": tokens - 1, "last": now})
return True
This is lazy refill: no cron, no per-bucket thread. The bucket only "exists" when someone touches it, and stale buckets cost nothing beyond a hash key that we let expire via TTL.
The part that mattered: atomicity
The naive version above has a race. Two concurrent requests can both read the same token count and both decide they are allowed. Under load, that doubles your effective rate.
We fixed it with a Lua script executed inside Redis. Reading the hash, computing refill, decrementing, and writing back all happen in one atomic operation on the Redis single thread. No locks on the application side, no distributed coordination — just one script that runs to completion.
This was not optional. Under normal traffic the race is rare; under a flash sale it is constant. A limiter that leaks under the exact load it is meant to handle is worse than no limiter, because you trust it.
What the numbers looked like
We set capacity to 20 and refill to 0.5 tokens per second — roughly 30 requests per minute sustained, with room for a 20-request burst. Before the limiter, a single misbehaving client could push several hundred checkout calls a minute and drag down response latency for everyone. After, checkout p99 latency stayed flat during the sale, and the number of duplicate payment intents dropped sharply.
We also return Retry-After on rejection, computed from how long until the next token is available. Well-behaved clients back off politely instead of hammering harder.
Lessons worth keeping
- Fixed windows leak at boundaries; token buckets smooth bursts without punishing them.
- Lazy refill keeps the implementation cheap — no timers, no per-key worker.
- Make the check-and-decrement atomic. Do the race-test under load, not in a unit test.
- Tune capacity and refill separately: one controls bursts, the other controls the average.
- Return
Retry-Afterso clients cooperate rather than collide.
Rate limiting is not glamorous work, but it is the kind of unglamorous plumbing that decides whether checkout survives its busiest hour. We would rather spend an afternoon on a token bucket than an all-nighter on incident calls.
Tags: #webdev #programming #ecommerce #automation
Top comments (0)