Design a Rate Limiter: The Four-Point Answer, and the Trap

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

"Design a rate limiter."

It is one of the most common system design interview questions, and it has an unusually reliable failure mode. The candidate hears it as an algorithms question, names one — token bucket, leaky bucket, fixed window, sliding window log — sketches the mechanism, and stops.

Everything they said was correct. It also answered the easy half.

Naming the algorithm tells the interviewer you have read the same article everyone else read. What they are actually listening for starts one question later: where does the count live when you are running more than one server? The algorithm is a function. The counter is state, and state is where system design lives.

Here is the four-point answer.

1. Count against the API key or the user id, not the IP

Before any mechanism, decide what you are counting against. This is a choice with a wrong answer, not a definition.

The tempting answer is the IP address, because it is always there and it needs no authentication. The problem is that an IP address is not a person. One IP is routinely one office, one university campus, one coffee shop, one mobile carrier's NAT pool — hundreds or thousands of unrelated people sharing a single address. Rate limit on it and every one of them shares a single budget. The first heavy user of the morning exhausts the quota for the entire building.

Meanwhile, the abuser you built this for is the one party in the story for whom addresses are cheap and disposable. They rotate through a proxy pool and never touch your ceiling.

So you punish the innocent and miss the abuser — a limiter that is somehow both too strict and too loose, which is a strange thing to ship.

	IP address	API key / user id
What it identifies	A network path, at this moment	An account — the thing you actually meter
Who shares one	An office, a campus, a carrier NAT pool — thousands of strangers	One customer, by construction
Cost to rotate	Cents, via any proxy pool	A new signup, and you can see it happen
Available before auth	Yes	No
Stable across networks	No — wifi to cellular changes it	Yes
Failure mode	Throttles the innocent, misses the abuser	Needs an identity to exist first

Count against the identity: the API key, the user id, the tenant. That is stable across networks, it is the thing your product actually meters, and it is the thing a customer would recognise on an invoice.

There is one honest exception. Anonymous, unauthenticated traffic — a public signup endpoint, a login form, an unauthenticated search — has no identity to count against yet. There, the IP is the best signal available and an IP limit is the right call. Say that out loud in the interview; knowing when the weaker key is the correct key is worth more than rejecting it outright.

2. Enforce it at the gateway, not inside every service

Put the limiter at the front door — the API gateway, the edge proxy, the ingress — and not inside each service behind it.

Two reasons, and they are different in kind.

The operational one: one front door is one place where limits are configured, one place where they are changed, one place to look when a customer says they are being throttled. Implement the check in each service and you have N implementations of the same rule, which will drift, and a limit change becomes N deploys.

The economic one is stronger. A request rejected at the door never gets past the door. It does not open a database connection, does not enqueue a job, does not hit your billing code, does not consume a worker slot. That is the entire point of rate limiting under load: the traffic you refuse must be cheap to refuse. A limiter that sits deep in the stack — say, as a decorator on a service method — has already paid for connection setup, deserialisation and authentication before it decides the answer is no. Under the exact conditions you built it for, it costs you the most.

There is a second-order version of the same mistake: a request that fails the check in service C has already done real work in A and B.

3. The counter is shared state

This is the point most answers miss, and it is where the question is actually decided.

Say you keep the count in a dictionary in the API process. It is fast, it needs no extra infrastructure, and it works perfectly in development, where you run one process.

Then you run three.

You wrote "100 requests per minute" in the spec. What you have built is 100 requests per minute per instance. Three instances is 300. Ten is 1000. And the number is not even fixed — a load balancer spreading a user's requests across the fleet means each instance sees roughly a third of that user's traffic and never trips, so in practice the ceiling is higher still and depends on how the balancer happens to route.

Then the autoscaler joins in. Now the effective limit moves with your traffic: busy hour, more instances, higher real limit — precisely the opposite of what a limiter is for. The system is most permissive at the moment it should be strictest, and nothing in your monitoring will say so, because every individual instance is dutifully enforcing 100.

The fix is not a cleverer algorithm. It is one store, one atomic increment.

One store: a single place — Redis is the usual choice — that every instance talks to, so there is exactly one number per key per window.

One atomic increment: not read, then decide, then write. Those are three steps, and two requests arriving together can both read 99, both decide they are allowed, and both write 100. The check and the claim have to be a single indivisible operation. Whether that is INCR with an expiry, a conditional decrement, or a small server-side script is an implementation detail; the invariant is what earns the point.

That store is now on the hot path of every single request, so say the consequences out loud too: it needs to be fast, it needs to be close, and you need an answer for what happens when it is unreachable. Fail open and you have no limiter during exactly the incident that produced the outage. Fail closed and a limiter outage is a full outage. Choosing deliberately, and being able to say which way you chose and why, is the senior answer.

4. Answer with a 429, a Retry-After, and the quota left

The last point is the contract, and it is the one candidates skip because it feels like trivia.

Reject with 429 Too Many Requests. Include Retry-After with the number of seconds until the window resets. Include the remaining quota — X-RateLimit-Remaining, or whatever your API already uses — on the successful responses too, so a well-behaved client can see the wall coming before it hits it.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Remaining: 0

The reason this matters is not politeness. It is that the client now has a number to wait on instead of a guess.

A limiter that returns a bare 500, or a 403, or a 200 with an error body, teaches every client integrating against you to invent its own retry timing — and the timings they invent are uniformly worse than the one you already know, because you are the only party who knows when the window resets. You are handing out a fact you have and they do not. Withholding it costs you the retries.

The same question, in front of a model

If you have ever been handed an LLM API key, you have already met all four points wearing different names — and one extra twist that makes this the sharpest live example of the pattern.

Look at how a model provider states its limits. Not "100 requests per minute" but RPM and TPM — requests per minute and tokens per minute, enforced together. That second dimension breaks an assumption the classic answer quietly makes: that every request costs the same. Here they do not. One request is forty tokens; the next drags a 100,000-token document behind it.

So the count is no longer +1. It is +cost, and you do not know the cost until you have counted the prompt — and you do not know the full cost until the model has finished generating, because output tokens count too. Real limiters handle this by reserving an estimate up front and reconciling on completion: increment by prompt tokens plus a max-output allowance, then correct the counter down when the stream ends. That is still one store and one atomic operation. It is just an operation that adds a number rather than one.

The other three points transfer unchanged, and get more important rather than less:

• What you count against — the tenant, not the IP, because one of your customers is a chat product with ten thousand users behind one egress address.
• Where you enforce it — at the gateway in front of the model, because a rejected inference request must never reach the GPU. This is the economic argument at its most extreme: a request admitted by mistake does not cost you a database connection, it occupies a slot on a machine that costs dollars an hour, for seconds at a time.
• The reply — 429 with a Retry-After. Every major provider sends one, and every serious client library reads it.

And if you are serving your own model rather than calling someone else's, the scarce resource shifts again: what actually runs out is not requests per minute but concurrent sequences in the batch and blocks in the KV cache. A limiter in front of an inference server is really admission control for GPU memory, and the honest version of it counts what the GPU is short of rather than what is easy to count.

Which is the same lesson as point 3, one level up: the interesting part was never the algorithm. It was working out what the number means and where it is kept.

The trap

Everyone answers "token bucket". The interviewer is waiting to hear where the count lives.

That is the whole question, compressed. The algorithm is a small, well-documented function that you could look up in a minute. The counter is distributed mutable state on the hot path of every request — which is a system design problem, which is why this is a system design interview.

If you have time for one more sentence after the four points, make it this one: the algorithm decides whether to allow a request, and the storage decides whether that decision is true.

The verdict — the whole answer in four lines

• key → count per user or API key, not per IP; IP only for anonymous traffic
• gateway → one front door, so a rejected request costs nothing
• counter → one shared store, one atomic increment, because in-process counts multiply by your instance count
• 429 → plus Retry-After and the remaining quota, so the client waits on a number instead of a guess

Name an algorithm if you like — token bucket is a fine choice and takes five seconds to justify. Then spend the rest of your answer on the four points above, because that is the part the interviewer cannot look up on your behalf.

References and further reading

On the reply contract — point 4

• IETF, RFC 6585 §4 — Additional HTTP Status Codes: 429 Too Many Requests (IETF, 2012) — the normative definition of the status code: the server indicates the user has sent too many requests in a given amount of time, and the response may include a Retry-After. Worth quoting in an interview because the spec explicitly declines to define how the server counts, which is exactly the storage decision this article argues you are being tested on: rfc-editor.org/rfc/rfc6585#section-4
• IETF, RFC 9110 §10.2.3 — HTTP Semantics: the Retry-After header field (IETF, 2022) — the support for "a number to wait on, not a guess": Retry-After carries either a delay in seconds or an HTTP-date after which the client may retry, and nothing else: rfc-editor.org/rfc/rfc9110#field.retry-after
• IETF, draft-ietf-httpapi-ratelimit-headers — RateLimit header fields for HTTP (IETF, in progress) — the ongoing standardisation of the "quota remaining" half of point 4, and the reason the X-RateLimit-* headers above are a de-facto convention rather than a standard. Note that it is a draft, not a published RFC: datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers

On the shared counter — point 3

• Redis documentation, INCR — "Pattern: Rate limiter" — the concrete mechanism behind "one store, one atomic increment", from the store this design would actually use: a counter per key per window, incremented atomically and expired with the window. It also documents the exact race the wording above is chosen to avoid — the gap between the INCR and the EXPIRE: redis.io/docs/latest/commands/incr

On what to count and where to enforce it — points 1 and 2

• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering, chapter 21, "Handling Overload" (O'Reilly / Google, 2016) — Google's practice of metering per customer against a quota rather than per source address, and the argument that a rejected request is not free: it still costs CPU and memory, so it must be refused as early and as cheaply as possible. That is the economic case for the front door.
• Paul Tarjan, Scaling your API with rate limiters (Stripe Engineering blog, 2017) — a production account that lines up with points 1, 2 and 4: limits applied per API key at the edge rather than inside application code, backed by a shared Redis counter, returning 429 with enough information for the caller to act: stripe.com/blog/rate-limiters

If a reference you would expect is missing, say so in the comments and I will add it.

---

Watch the reel: Design a rate limiter — the four-point answer, and the trap