A rate limiter answers one question on every request: has this client made too many calls, and should I reject this one? It sounds trivial until you have twenty API servers behind a load balancer and none of them agree on the count.
Why you need one
Without a limit, a single misbehaving client can starve everyone else. A retry loop with no backoff, a scraper, or a credential-stuffing attack will happily send thousands of requests per second. Rate limiting protects your capacity, enforces fair use across tenants, and is often a billing boundary (free tier gets 100 requests per minute, paid gets 10,000).
The two algorithms worth knowing
Token bucket models a bucket that holds up to N tokens and refills at a fixed rate, say 10 tokens per second. Each request removes one token. If the bucket is empty, the request is rejected. The nice property is that it allows bursts: a client that has been quiet accumulates tokens up to the bucket capacity, then can spend them quickly. You only need to store two numbers per client, the current token count and the timestamp of the last refill, and compute the refill lazily on each request.
Sliding window log keeps a timestamp for every request in the current window and counts how many fall inside it. It is exact but expensive, because a client sending 10,000 requests per minute means storing 10,000 timestamps.
Sliding window counter is the pragmatic middle ground. You keep a counter per fixed window (say per minute) and approximate the sliding window by weighting the previous window. If you are 25 percent into the current minute, the effective count is the current minute's count plus 75 percent of the previous minute's count. This smooths out the boundary problem that plagues fixed windows, where a client can send a full quota at 0:59 and another full quota at 1:00, doubling the intended rate for a moment.
The trade-off
Token bucket is cheap and burst-friendly, which is what most public APIs actually want. Sliding window counter gives you a smoother, more predictable limit at slightly higher storage cost. Fixed window is the simplest to reason about but has the boundary-doubling flaw, so avoid it for anything security-sensitive.
Making it distributed
The hard part is not the algorithm, it is shared state. If each server keeps its own bucket in memory, a client hitting three servers gets three times the limit. You need a single source of truth, and the usual choice is Redis because it is fast and gives you atomic operations.
The naive approach reads the counter, checks it, and writes the new value. That is a race: two servers read the same count, both decide the request is allowed, both write. The fix is to make the read-modify-write atomic. A Redis Lua script runs on the server as a single unit, so you can refill the bucket, check for a token, decrement, and set a TTL without any other command interleaving. That single round trip is your entire rate limit decision.
Two more details matter in production. First, set an expiry on each key so idle clients do not leak memory; a bucket that has not been touched in an hour can be dropped and rebuilt from empty. Second, decide your failure mode. If Redis is unreachable, do you fail open (allow the request) or fail closed (reject it)? Most systems fail open for availability, accepting that a Redis outage temporarily removes the limit, but a payment or auth endpoint might fail closed.
How the real systems do it
Stripe runs rate limiting in Redis and famously uses token bucket for most limits, with a separate concurrency limiter for expensive endpoints. Cloudflare pushes limiting to the edge so an abusive client is rejected close to its own network, before the request ever reaches an origin. The common thread is that the decision lives in one fast, shared, atomic place, and the response includes headers like Retry-After and X-RateLimit-Remaining so well-behaved clients can slow themselves down.
If you get one thing right, make the counter update atomic. Everything else is tuning.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-rate-limiter
Top comments (1)
Great breakdown of a classic distributed systems challenge. Rate limiting looks simple at first, but the real complexity appears when you need to balance fairness, burst handling, accuracy, and scalability across multiple instances.
I like the comparison between token bucket and sliding window because it highlights that there is no universal winner — the right choice depends on the product requirements. Token buckets are great for absorbing short bursts, while sliding windows provide stronger control over request patterns.
The distributed aspect is where things get really interesting: synchronization, shared state, latency, and consistency all become important design decisions. A well-designed rate limiter is not only about protecting infrastructure; it also improves reliability and user experience. Great system design article!