Design a rate limiter, answered properly

Design a rate limiter, answered properly
This is the most common system design question I was asked, and the one where the follow ups do all the work. Here is the full walkthrough, including the parts I got wrong first.

Written after being asked this four times in one year. Updated October 2026. 9 min read

Design a rate limiter came up in four separate processes in one year. The first time I described token bucket in ninety seconds and then sat there, having used up what I knew. The remaining forty minutes were where the round actually happened.

The algorithm is the easy part and everyone gets there. The round is decided after that.
Clarify before you draw
Four questions, about ninety seconds.

What are we protecting?
A public API, a login endpoint, an expensive operation. The answer changes the limits.
Limited by what?
Per user, per API key, per IP address, or globally. Each has different problems.
What scale?
Hundreds of requests a second or hundreds of thousands changes the storage answer.
What should happen at the limit?
Reject, queue, or degrade. Rejecting is usually right but it is worth asking.
The algorithms, and when each fits
Pick one and defend it.

Allows
bursts
Memory cheap
Exact
Simple to
explain
Token bucket
Limited
Sliding window log
n/a
n/a
Limited
Fixed window
Limited
n/a
Leaky bucket
n/a
Limited
Limited
Clearly fits
Sometimes
Not the tool for this
Token bucket is the usual default. Say why you picked it rather than just naming it.
Token bucket holds a number of tokens that refill at a fixed rate. Each request takes one. An empty bucket means rejection. It allows a short burst up to the bucket size, which matches how real clients behave, and it is cheap to store as two numbers per key.

Fixed window is simpler and has one notorious flaw worth raising yourself: a client can send a full allowance at the end of one window and another full allowance at the start of the next, producing double the intended rate across that boundary.

Making it work across many servers
The first real follow up.

In process counters stop working the moment you run more than one instance, because each server only sees its own share of traffic. Move the counters to a shared fast store keyed by the identity and the window.

The subtlety to raise unprompted is atomicity. Reading the count, deciding, and writing it back as three separate steps lets two servers both allow the final request. Use an atomic increment or a small script that performs the check and update as one operation.

What you return at the limit
The part that affects real clients.

A good rejection
429 status code
A header with the limit
A header with remaining quota
A retry after hint
A bad rejection
A generic 500
No indication of when to retry
Silently dropping the request
A different shape from your normal errors
Without a retry hint, a well intentioned client retries immediately and your rate limiter becomes a generator of additional load. This is the detail that separates someone who has designed one from someone who has read about one.

When the counter store is down
The follow up that decides the round.

You have two options and both are defensible. Failing open keeps traffic flowing while losing protection, which suits a general API where availability matters most. Failing closed protects the service while rejecting legitimate users, which suits a login endpoint where the limiter is a security control.

The answer interviewers want is not one of the two. It is that you know there is a choice, you know what each one costs, and you can say which you would pick for this specific endpoint.

Extras worth mentioning if time allows
Briefly, not exhaustively.

Different limits per tier
Free and paid users rarely share a limit.
Local plus shared
A small in process cache reduces round trips at high volume.
Observability
Which metric tells you the limiter is wrongly configured rather than working.
Testing it
How you would verify the limit without hammering production.
How I practised this question
Design it, then remove something.

After my second time being asked, I started designing it on paper and then deleting a component: the shared store, one of the servers, the retry header. I used Phantom Code AI[https://www.phantomcodeai.com/] in its slower reasoning mode to attack my design rather than produce one, specifically asking what breaks when two servers race for the last token. That race condition was consistently the weak point in my own version.

Since the same tool covers coding, SQL and behavioural rounds, the evening habit carried the rest of the loop rather than being a single question exercise.

Common questions
Short answers.

Which algorithm should I pick?
How does it work across servers?
What should the API return?
What if the store is down?
Everyone gets the algorithm. Few get the failure modes.
This question survives in every loop because the first ninety seconds are easy and the next twenty minutes are not. Burst behaviour, races, retry hints and store outages are where it is decided.

Sketch it tonight, then delete the shared store and write what happens next.

Written after the fourth time, finally enjoying it

How many times have you been asked this one?

Four, for me, and the follow ups were different every time.