System Design Interview: Your 100-Request Limit Just Allowed 200

One boundary example, a small Python model, and a better way to explain what your rate limiter actually promises.

You set the limit to 100 requests per minute, check that request 101 gets rejected, and feel pretty good about the implementation.

Then someone sends 100 requests just before the minute ends and another 100 just after the next minute begins. Your service accepts all 200, even though only a fraction of a second has passed.

Nobody hacked the counter. It followed the rule you implemented, which may not be the rule you meant.

I like this system design exercise because the interesting part arrives before Redis, sharding, or a beautifully drawn API gateway. We have to decide what per minute means, and that tiny phrase changes almost everything that follows.

Make the timing awkward before making the architecture impressive

Use an original practice scenario with one authenticated client and one endpoint. Every admitted request costs one unit, and rejected attempts do not consume this particular quota. Other abuse controls may count attempts differently.

Suppose the implementation uses fixed windows from 0 to 60 seconds, then 60 to 120 seconds. Each window has its own counter, capped at 100.

At 59.9 seconds, the client sends 100 requests. At 60.1 seconds, it sends another 100. Both windows remain within their individual limits, but the service has admitted 200 requests across 0.2 seconds.

That behavior is not automatically a defective fixed-window implementation. It is a mismatch if the product promise was at most 100 admissions during any rolling 60-second interval.

For our exercise, choose that rolling promise explicitly. At decision time t, count admissions in the interval (t − 60 seconds, t]. An admission exactly 60 seconds old has expired. A request arriving now belongs to the interval if we admit it.

Honestly, writing down that boundary is more useful than listing five algorithm names and hoping one sounds senior enough.

Keep a little history instead of resetting the calendar

A sliding log stores the timestamps of admitted requests. Before deciding on a new request, remove expired entries, count what remains, and admit only if there is room. Record the new timestamp only after that decision.

In our 59.9-second example, the first 100 timestamps are still recent at 60.1 seconds. The second burst is rejected. Those entries expire at 119.9 seconds under the boundary rule we chose.

Redis’s rate-limiting walkthrough explains the different algorithms and their trade-offs. I prefer using a concrete timeline like this before selecting one, because an implementation can be correct for one policy and unsuitable for another.

Here is a small reference model for one quota key. It uses integer milliseconds and caller-supplied, nondecreasing decision times so the boundary is easy to test without waiting for a real clock.

from collections import deque
class RollingLimit:
    def __init__(self, limit=100, window_ms=60_000):
        if limit <= 0 or window_ms <= 0:
            raise ValueError('limits must be positive')
        self.limit = limit
        self.window_ms = window_ms
        self.admitted = deque()
        self.last_time = None
    def allow(self, now_ms):
        if self.last_time is not None and now_ms < self.last_time:
            raise ValueError('time must not move backward')
        self.last_time = now_ms
        cutoff = now_ms - self.window_ms
        while self.admitted and self.admitted[0] <= cutoff:
            self.admitted.popleft()
        if len(self.admitted) >= self.limit:
            return False
        self.admitted.append(now_ms)
        return True

The less-than-or-equal comparison is doing real work. Changing it to a strict less-than would keep a request exactly one window old and enforce a different boundary.

This model assumes sequential calls in one process. It is not thread safe, distributed, persistent, or a production middleware. Its job is to make the admission rule executable before we add those responsibilities.

Each admitted timestamp enters and leaves the deque once, giving amortized constant time per call over a sequence of calls. One call can still remove up to the limit’s worth of expired timestamps. Stored history is bounded by the admission limit for this one key.

Test the edge without sleeping for a minute

You can run these checks immediately after the class. They separate the fixed-window boundary burst from the exact point when old admissions expire.

limiter = RollingLimit()
assert sum(limiter.allow(59_900) for _ in range(100)) == 100
assert sum(limiter.allow(60_100) for _ in range(100)) == 0
assert not limiter.allow(119_899)
assert limiter.allow(119_900)
edge = RollingLimit(limit=1)
assert edge.allow(0)
assert not edge.allow(59_999)
assert edge.allow(60_000)

Also try multiple requests with the same timestamp. Time equality must not make distinct requests disappear. In this deque they remain separate entries; if you later use a sorted set, a timestamp alone is not a safe unique member identifier for all requests arriving at that time.

Keep the test clock under your control. Sleeping introduces scheduling noise while teaching very little about the contract you are trying to verify.

Three servers quietly change the meaning of your limit

Now create three independent copies of the model behind a load balancer. If the same client reaches all three, each copy can admit 100 requests. You have implemented three local allowances, which can add up to 300.

Our original promise applied to the client’s total traffic, so local correctness is no longer enough.

One design is to route decisions for a quota key to authoritative shared state. The sequence that removes expired entries, checks the count, and records an admission must execute atomically. Separate network calls can let two requests both observe the final free slot and both take it.

A short Redis script is one implementation option. Redis documents atomic script execution, but a script is not a complete availability strategy. Keep its work bounded, and decide what state loss or failover means for the quota.

In a sorted-set version, timestamps can be scores while distinct admission identifiers are members. Partition by the quota key so the state needed for one decision stays together. A particularly busy key can still become a hot spot; adding shards does not split an indivisible decision for free.

Client identity needs care too. A browser-provided string should not let someone choose a new quota whenever they want. Tie this example’s key to the authenticated account and endpoint policy. In a multi-tenant service, include the tenant boundary rather than accidentally sharing somebody else’s allowance.

Decide what happens when the limiter cannot decide

If the shared store times out, there is no magical response that preserves every desirable property.

Failing open admits traffic while the quota cannot be checked, which sacrifices strict enforcement. Failing closed protects that boundary by refusing traffic, which also affects legitimate clients. A reduced local allowance is another explicit compromise, but it is not equivalent to the original global promise.

For this practice scenario, assume the endpoint triggers expensive work and strict admission control matters more than serving through a limiter outage. We will refuse new work when the limiter is unavailable, and distinguish that infrastructure failure from a known quota rejection in the response and metrics. Different product priorities can justify another decision.

Use a consistent time authority for distributed decisions and define behavior for clock corrections. The monotonic sequence we supplied to the Python model does not appear automatically when several machines contribute timestamps.

A rolling quota still allows a burst, and that may be the next question

Our model admits 100 requests at the same instant when the history is empty. It prevents the 200-request boundary trick, but it does not spread those first 100 requests evenly over a minute.

That distinction matters if each request launches expensive work. A separate concurrency cap limits how much work is active at once; a rate quota controls admissions over time. Neither one automatically replaces the other.

If the desired policy is a controlled burst plus a sustained refill rate, discuss a token bucket instead. If precise rolling counts are unnecessary, a simpler counter may be appropriate. Choose the promise first and explain the cost of keeping it, rather than calling one algorithm universally best.

To rehearse the full conversation, use this rate-limiter design prompt and change one assumption at a time. Start with one client, then add several servers, then make the shared store unavailable. Keep your original promise visible while explaining which part survives each change.

Before closing your notebook, write the interval boundary, the quota key, the atomic decision, and the failure policy in ordinary sentences. If any of them remains vague, you have found the next thing to practice.

I care much more about that explanation than whether the diagram contains the trendiest database. A counter can be perfectly implemented and still enforce the wrong agreement.

~ Emily