System Design: Rate Limiter
A capstone system design walkthrough — designing a distributed rate limiting system end to end — covering the core domain model, the major rate-limiting algorithms and their trade-offs (token bucket, leaky bucket, fixed window, sliding window), enforcing limits consistently across many nodes, the storage layer's own latency and availability demands, tiered and multi-dimensional limits, graceful degradation under limiter failure, and the specific low-latency, high-consistency-under-concurrency demands that make a rate limiter a small system with an outsized number of subtle correctness traps.
Table of Contents
- Introduction
- Why a Rate Limiter Is a Different Kind of Hard
- The Core Domain Model
- Rate Limiting Algorithms
- The Counter Store: Where State Actually Lives
- Enforcing Limits Consistently Across Many Nodes
- Multi-Dimensional and Tiered Limits
- Where the Limiter Sits: Placement in the Request Path
- Response Contract: Telling Clients What Happened
- Graceful Degradation When the Limiter Itself Is Unhealthy
- Distributed Clock Skew and Window Boundary Effects
- Configuration Management and Dynamic Limit Updates
- Data Security and Abuse Considerations
- Consistency, Availability, and the CAP Trade-off for a Rate Limiter
- Scaling the System
- Observability for a Rate Limiter
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
A rate limiter takes the general system design vocabulary covered in this series' System Design guide — counters, sliding windows, distributed coordination, low-latency storage — and applies it to a component that is small in scope but sits directly in the critical path of every single request it protects, which means its own latency and availability become part of the latency and availability of everything behind it. This guide walks through designing such a system end to end, drawing directly on this series' Redis, Distributed Systems, and Resilience guides, each of which turns out to be load-bearing infrastructure for a rate limiter that's actually correct and fast under real concurrent load, rather than optional architectural polish.
Client → Edge/Gateway → Rate Limiter (check + increment, per-key) → [Allow] → Backend Service
↓ (fast lookup) → [Deny] → 429 response
Counter Store (Redis/similar)
1. Why a Rate Limiter Is a Different Kind of Hard
It sits on the critical path of every request it protects, with a very tight latency budget
Most systems covered in this series can afford some latency because the work they do is substantial enough to justify it. A rate limiter does comparatively little work — check a counter, maybe increment it, return a decision — and that decision needs to add single-digit milliseconds at most to every request it touches, because it's evaluated far more often than almost anything else in the request path. This is why the counter store's own latency (Section 4) gets as much design attention in this guide as the limiting algorithm itself.
The core operation is a read-modify-write under genuinely high concurrency, by definition
A rate limiter's entire job is counting concurrent requests from the SAME key
in a SHORT window — which means the read-modify-write race condition this
series' Database guide warns about in general is not a rare edge case here,
it's the expected, constant operating condition for any popular key.
Unlike most systems where concurrent writes to the same row are an occasional hot-key problem (per this series' High-Volume Transaction Processing guide's Section 7), a rate limiter's most important keys — the ones actually worth limiting — are, by construction, the ones seeing the most concurrent traffic; an implementation that isn't atomic under concurrency will systematically under-count exactly the traffic it most needs to catch.
Being wrong has two very different failure modes, and neither is free
A critical, freeing realization for the design that follows: a rate limiter, in the overwhelming majority of real-world designs, does not need to be perfectly, globally precise to be useful — it needs to fail in the direction the system prefers when forced to choose. Undercounting (allowing slightly more traffic than the configured limit) risks the backend it protects; overcounting (rejecting legitimate traffic) risks user experience and trust. Deciding which failure mode is more acceptable, for which limit, is a real design decision (Section 9's "fail open vs. fail closed") rather than something correctness alone resolves — this mirrors the availability-vs-consistency framing covered throughout this series' System Design guide, applied here to a component whose entire purpose is enforcing a limit.
2. The Core Domain Model
Modeled deliberately simply, like this series' URL Shortener guide's domain
public record RateLimitKey(string Value); // e.g. "user:123:api:/orders" or "ip:203.0.113.4"
public record RateLimitRule(string Name, int Limit, TimeSpan Window, RateLimitAlgorithm Algorithm);
public record RateLimitDecision(bool Allowed, int Remaining, TimeSpan RetryAfter);
public interface IRateLimiter
{
Task<RateLimitDecision> CheckAsync(RateLimitKey key, RateLimitRule rule);
}
As with this series' URL Shortener guide's own domain modeling choice, a rate limiter doesn't warrant a heavyweight DDD aggregate — its core operation is a single, well-defined check against a rule, and modeling it as a small, composable interface (per this series' Interface Segregation discussion) keeps the algorithm (Section 3), the storage backend (Section 4), and the rule configuration (Section 11) independently swappable.
Rules as configuration, not code
public record RateLimitRule(string Name, int Limit, TimeSpan Window, RateLimitAlgorithm Algorithm, RateLimitScope Scope);
// Scope determines the KEY: PerUser, PerIp, PerApiKey, Global, or a composite of several
Per this series' Configuration Management guide, keeping rules as data rather than hardcoded logic is what makes Section 6's multi-dimensional limits and Section 11's dynamic updates possible without a redeploy — a limiter whose rules are compiled into its code can't respond to a sudden abuse pattern (per this series' URL Shortener guide's Section 10) nearly as quickly as one whose rules live in a fast-to-update configuration store.
3. Rate Limiting Algorithms
Fixed window counter — the simplest, with a real boundary-burst flaw
// Increment a counter keyed by (identity, current_window_start), expire after the window
var windowKey = $"{key}:{CurrentWindowStart(rule.Window)}";
var count = await _store.IncrementAsync(windowKey, expiry: rule.Window);
return new RateLimitDecision(count <= rule.Limit, Math.Max(0, rule.Limit - count), TimeUntilNextWindow(rule.Window));
As covered in this series' Rate Limiting Algorithms guide, a fixed window is trivial to implement and reason about, but has a well-known flaw: a client can send its full limit right at the end of one window and its full limit again right at the start of the next, achieving up to double the intended rate in a short burst straddling the boundary — a real correctness gap worth knowing about even when the simplicity is otherwise attractive.
Sliding window log and sliding window counter — closing the boundary-burst gap
Sliding window LOG: store a timestamp per request, count entries within the
trailing window — precise, but memory cost scales with request volume per key.
Sliding window COUNTER: approximate the sliding window by weighting the
previous fixed window's count proportionally to how much of it overlaps
the current trailing window — nearly as accurate, far cheaper to store.
Per this series' Rate Limiting Algorithms guide's comparison, the sliding window counter is the practical middle ground most production systems reach for: it closes the fixed window's boundary-burst flaw to within an acceptable approximation, without the sliding log's per-request storage cost — worth knowing the log variant exists for cases needing exact precision, but the counter variant is the common default.
Token bucket — the standard choice when bursts should be permitted deliberately, up to a cap
public async Task<RateLimitDecision> CheckTokenBucketAsync(RateLimitKey key, RateLimitRule rule)
{
var bucket = await _store.GetOrCreateBucketAsync(key, capacity: rule.BurstCapacity, refillRate: rule.Limit / rule.Window.TotalSeconds);
bucket.Refill(DateTimeOffset.UtcNow); // add tokens accumulated since last check, capped at capacity
if (bucket.Tokens >= 1) { bucket.Tokens -= 1; return RateLimitDecision.Allow(bucket.Tokens); }
return RateLimitDecision.Deny(TimeUntilNextToken(bucket));
}
As covered in this series' Token Bucket discussion, this algorithm is the right choice when a system wants to permit legitimate bursty behavior (a client that's mostly quiet but occasionally sends a quick flurry of requests) up to a configured burst capacity, while still enforcing a steady-state average rate over time — distinct from the window-based algorithms above, which cap total requests within a fixed interval regardless of how "bursty" that traffic's shape actually is.
Leaky bucket — for smoothing bursty traffic into a steady outbound rate
Requests enter a queue (the "bucket"); they're processed ("leak out") at a
fixed rate regardless of how bursty the input was — per this series' Queueing
Theory discussion, this smooths traffic reaching a downstream system rather
than just rejecting excess, which matters when the goal is protecting a
fragile downstream dependency from burst load, not just capping a client's rate.
Per this series' Leaky Bucket discussion, this variant is less common at the API-gateway layer (where rejecting excess with a clear signal, per Section 8, is usually preferred over silently queueing and delaying) and more common as an internal traffic-shaping mechanism protecting a downstream service that genuinely can't handle bursts even briefly, regardless of the long-run average rate being acceptable.
4. The Counter Store: Where State Actually Lives
Why an in-memory, single-process counter doesn't survive contact with more than one node
❌ A counter held in application memory only limits requests landing on THAT
specific process — with multiple app instances behind a load balancer
(the normal case), the effective limit becomes (configured limit × instance
count), silently far looser than intended.
As covered in this series' Distributed Systems guide, any rate limiter deployed across more than one instance needs shared, external state for its counters — this is precisely why Redis (or a similarly fast, atomic-operation-supporting store) is the standard choice covered in this series' Redis guide, rather than each instance tracking its own local count.
Redis as the default choice, and why its atomic operations matter specifically here
// INCR is atomic in Redis — no read-modify-write race, per this series' Redis guide
var count = await _redis.StringIncrementAsync(windowKey);
if (count == 1) await _redis.KeyExpireAsync(windowKey, rule.Window); // set TTL only on first increment
Per this series' Redis guide, Redis's single-threaded execution model makes INCR genuinely atomic without any application-level locking — directly solving Section 1's read-modify-write race concern, which is precisely why Redis (or an equivalent store with the same atomicity guarantee) is the default backing store for nearly every production rate limiter, rather than a general-purpose relational database whose transactions would add latency this system's budget can't absorb.
Lua scripting for atomic multi-step operations (token bucket, sliding window counter)
-- Executed atomically as a single Redis operation, per this series' Redis Scripting guide —
-- avoids the race between "read bucket state" and "write updated bucket state" as two separate round trips
local tokens = tonumber(redis.call('GET', KEYS[1]) or capacity)
local refilled = math.min(capacity, tokens + elapsed * refill_rate)
if refilled >= 1 then
redis.call('SET', KEYS[1], refilled - 1, 'EX', ttl)
return 1
end
return 0
As covered in this series' Redis Scripting guide, algorithms needing more than a single atomic increment (Section 3's token bucket, in particular) should execute their read-modify-write logic as a single Lua script on the Redis server itself — this closes the same race a naive "GET, compute, SET" sequence from application code would reintroduce, since Redis executes the entire script atomically without another client's operation interleaving.
5. Enforcing Limits Consistently Across Many Nodes
Centralized counter store as the default, straightforward answer
Every rate limiter instance, regardless of which app node it's colocated with,
reads and writes the SAME shared Redis instance (or cluster) for a given
key — this is the simplest way to get globally consistent counting across
a horizontally scaled deployment, and the right default absent a specific
reason not to use it.
Given Section 4's atomicity discussion, routing every limiter check through one shared, atomic-operation-capable store is the straightforward way to achieve consistent global counting — the trade-off, covered next, is that this introduces a network hop and a shared dependency into every request's critical path.
Local approximate counting as a latency and load-reduction optimization, at the cost of precision
Per this series' Approximate Algorithms discussion: each node maintains a
LOCAL counter and only periodically syncs/reconciles with the shared store
(or divides the global limit evenly across known node counts) — trading
some precision (the effective limit can drift somewhat above the configured
one) for eliminating a network round trip on every single request.
As covered in this series' Distributed Rate Limiting discussion, some high-throughput systems deliberately accept a looser, approximate limit in exchange for not paying a network round trip to a shared store on every request — dividing the global limit across a known set of nodes, or using local counting with periodic reconciliation, is a real, valid choice when Section 1's latency budget is tighter than a centralized store's round-trip time allows, and the limit itself doesn't need to be enforced with perfect precision.
Consistent hashing to route a given key's checks to the same store shard
Per this series' Consistent Hashing guide (echoed from the High-Volume
Transaction Processing guide's Section 5): sharding the counter store by
key ensures a given identity's requests are always checked against the
SAME shard, avoiding the cross-shard coordination a poorly-partitioned
store would otherwise require for every check.
Sharding the counter store's keyspace using consistent hashing — the same technique this series' High-Volume Transaction Processing guide applies to account balances — keeps a given rate-limited identity's checks landing on one shard consistently, which matters for the same reason it matters there: cross-shard coordination on every single check would reintroduce exactly the latency and complexity a rate limiter's tight budget (Section 1) can't afford.
6. Multi-Dimensional and Tiered Limits
Why "one limit per API" is rarely sufficient in practice
A real API typically needs several SIMULTANEOUS limits: per-user, per-IP
(catching abuse from a single source spanning multiple accounts), per-API-key,
and sometimes a GLOBAL ceiling protecting a specific downstream dependency
regardless of which client is calling it.
As covered in this series' API Gateway guide's rate limiting discussion, a request often needs to be checked against several rules simultaneously, with the request failing if it exceeds any of them — this is why Section 2's RateLimitRule includes an explicit Scope, letting the same limiter evaluate a request against a per-user rule, a per-IP rule, and a global rule as three independent checks rather than trying to encode all of that into one composite key.
Tiered limits reflecting a subscription or trust level
var rule = tierRegistry.GetRuleFor(client.Tier); // e.g. Free: 100/hr, Pro: 10,000/hr, Enterprise: custom
Per this series' SaaS Multi-Tenancy guide's tiering discussion, mapping a client's subscription or trust tier to a specific rule (rather than one universal limit for every caller) is standard practice for any API-as-a-product system — and per Section 11, this mapping needs to be updatable without a redeploy, since tier changes (an upgrade, a temporary limit increase during a promotion) happen on a business timeline, not an engineering release cycle.
Layering static and dynamic (risk-based) limits
A static, tier-based limit is the baseline — but per this series' Fraud
Detection and Surveillance system design guides' risk-scoring discussion,
some systems layer a DYNAMIC adjustment on top (temporarily tightening
limits for a client showing early signs of abuse, before a hard block
is warranted) — a genuinely more sophisticated policy layer, not a
replacement for the static baseline.
Worth noting as an extension, not a requirement: some rate limiters incorporate a dynamic, risk-score-adjusted layer on top of static tiered limits, echoing this series' Surveillance and Fraud Detection guides' pattern of statistical anomaly detection feeding into an otherwise rules-based system — a reasonable evolution once the static baseline described above is solid, not a starting requirement.
7. Where the Limiter Sits: Placement in the Request Path
At the edge/API gateway — the most common and generally preferred placement
Per this series' API Gateway guide: enforcing limits at the gateway, before
a request ever reaches application services, protects EVERYTHING behind
it uniformly and avoids every individual service needing to implement its
own limiting logic redundantly.
As covered in this series' API Gateway guide, placing the rate limiter at the edge is the most common architecture — it centralizes the limiting logic, protects all downstream services uniformly, and rejects excess traffic as early and cheaply as possible, before that traffic has consumed any of the more expensive compute further into the system.
Per-service, defense-in-depth limiting for services with their own specific capacity constraints
A gateway-level limit protects the SYSTEM broadly; an individual service
with its own specific, tighter capacity constraint (a database connection
pool ceiling, say) may still want its OWN, service-specific limit as a
defense-in-depth measure, per this series' Resilience guide's bulkhead discussion.
Per this series' Resilience guide's layered-defense principle, gateway-level limiting doesn't replace a service's own internal protections — a particularly resource-constrained downstream service can still benefit from its own, tighter local limit, treating the gateway's limit as the first, broad line of defense and its own as a more specific, service-aware backstop.
Client-side rate limiting as a complementary, not a substitute, layer
A well-behaved client implementing its OWN request pacing (respecting
Section 8's Retry-After header, backing off proactively) reduces load on
the server-side limiter — but server-side enforcement remains mandatory
regardless, since a rate limiter can never rely on a client's good behavior
as its actual security boundary.
Encouraging (and documenting, per Section 8) client-side pacing is a genuine best practice that reduces unnecessary rejected-request overhead, but per this series' Security guide's general "never trust the client" principle, it can only ever be a complementary optimization — the server-side limit is the actual enforcement boundary, full stop.
8. Response Contract: Telling Clients What Happened
Standard headers so well-behaved clients can self-regulate
HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1735689600
Retry-After: 42
As covered in this series' API Design guide's convention discussion, returning standard rate-limit headers on every response (not just rejected ones) lets well-behaved clients see how close they are to a limit and pace themselves proactively — this is a genuinely low-cost addition that measurably reduces the volume of requests that need to be outright rejected, since clients that can see the data will often self-throttle before hitting the wall.
A clear, actionable 429 response body, not just the status code
{ "error": "rate_limit_exceeded", "limit": 1000, "window_seconds": 3600, "retry_after_seconds": 42 }
Per this series' API Design guide's error-response discussion, a machine-readable body accompanying the 429 status gives client implementations exactly what they need to implement correct backoff automatically, rather than requiring a developer to read documentation to discover what the numeric limit actually was.
Deciding whether to reveal exact remaining counts, given Section 12's abuse considerations
Exposing precise remaining-request counts is usually the right default for
cooperative clients — but for a limit specifically defending against
ADVERSARIAL clients (a login-attempt limiter, say), revealing exact
thresholds can help an attacker calibrate around them, per this series'
OWASP Top 10 guide — worth a deliberate, per-rule decision, not a blanket policy.
Worth flagging as a genuine, rule-specific trade-off rather than a universal default: for rules meant to slow down or expose adversarial behavior (repeated failed login attempts, credential-stuffing patterns), returning precise counts and thresholds can hand an attacker exactly the information needed to stay just under the limit — a coarser or deliberately vague response is often the better choice for that specific class of rule, even while precise headers remain the right default for ordinary API-usage limits.
9. Graceful Degradation When the Limiter Itself Is Unhealthy
Fail open vs. fail closed — the central decision this guide's Section 1 sets up
public async Task<RateLimitDecision> CheckWithFallbackAsync(RateLimitKey key, RateLimitRule rule)
{
try
{
return await _primaryLimiter.CheckAsync(key, rule);
}
catch (StoreUnavailableException)
{
return rule.FailurePolicy == FailurePolicy.FailOpen
? RateLimitDecision.Allow(unknown: true) // per this series' Resilience guide's fail-open discussion
: RateLimitDecision.Deny(TimeSpan.FromSeconds(5)); // conservative, protects the backend
}
}
As covered in this series' Resilience guide's circuit breaker discussion, what happens when the counter store (Section 4) itself is unreachable is a genuine, rule-specific policy decision — fail open (allow the request) protects user experience but risks the very backend the limiter exists to protect; fail closed (reject the request) protects the backend but turns a rate limiter outage into a full outage of everything behind it. Per Section 1's framing, most systems reasonably choose fail-open for general API limits (a brief period of unlimited traffic is usually more tolerable than blocking all legitimate users) and fail-closed for limits specifically protecting a fragile, easily-overwhelmed downstream dependency — again, a decision made per rule, not once for the whole system.
Circuit breaking around the counter store itself
Per this series' Resilience guide: repeated failures talking to the counter
store trip a circuit breaker, switching to the configured fallback policy
IMMEDIATELY rather than letting every request pay a full timeout waiting
to discover the store is down — protecting Section 1's latency budget even
during a store outage.
Wrapping calls to the counter store in a circuit breaker (per this series' Resilience guide) ensures that once the store is known to be unhealthy, subsequent requests fail fast into the configured fallback policy rather than each one individually waiting out a connection timeout — critical given Section 1's tight latency budget, since a slow failure mode here is nearly as damaging as an incorrect one.
10. Distributed Clock Skew and Window Boundary Effects
Why relying on each node's local clock for window boundaries is a subtle correctness risk
Fixed and sliding window algorithms (Section 3) both depend on agreeing what
"now" is — meaningful clock skew between application nodes (or between an
app node and the counter store) can shift window boundaries slightly
differently depending on which node computed them.
As covered in this series' Distributed Systems guide's clock synchronization discussion, meaningful clock drift between nodes computing window boundaries independently can cause the same logical moment to fall into different windows depending on which node is asking — this is a subtle, rarely-catastrophic-but-real correctness gap worth designing around rather than assuming away.
Letting the counter store's clock be the single source of truth for time
-- Use Redis's own TIME command inside the Lua script, rather than trusting
-- the calling application node's local clock, per this series' Distributed
-- Systems guide's "single source of truth for time" principle
local now = redis.call('TIME')
Per this series' Distributed Systems guide, having every window-boundary calculation defer to the counter store's own clock (rather than each calling node's local clock) sidesteps inter-node clock skew entirely — every check agrees on "now" because they're all asking the same single source, which is a small design choice that closes a real, if narrow, correctness gap cheaply.
11. Configuration Management and Dynamic Limit Updates
Rules need to change faster than a deployment cycle allows
A sudden abuse pattern (per this series' URL Shortener guide's Section 10),
a customer upgrading their tier mid-cycle, or a downstream dependency
needing emergency protection during an incident ALL require limit changes
on a timescale of minutes, not a full deployment pipeline's timescale.
As covered in this series' Configuration Management guide, storing rules in a fast-to-update configuration store (a dedicated config service, or the same Redis instance backing the counters themselves) — rather than compiling limits into application code — is what makes Section 6's tiered limits and this section's emergency adjustments operationally realistic rather than requiring an emergency deployment under pressure.
Propagating configuration changes to every limiter instance consistently
Per this series' Configuration Management guide's propagation discussion: a
pub/sub mechanism (or short-TTL config caching with periodic refresh) keeps
every limiter instance's view of the current rules converged within a
bounded, known window, rather than some nodes enforcing a STALE rule
indefinitely after a change.
Given that a rate limiter typically runs as many concurrent instances (Section 5), a rule change needs a propagation mechanism — per this series' Configuration Management guide, a pub/sub invalidation signal or a short, bounded config TTL — ensures every instance converges on the new rule promptly, rather than some fraction of traffic continuing to be checked against an outdated limit indefinitely.
12. Data Security and Abuse Considerations
The rate limiter is itself a component adversaries will specifically probe
Per this series' OWASP Top 10 guide: an attacker aware they're rate-limited
will often probe FOR the exact limit (Section 8's disclosure trade-off),
attempt to bypass the limiter's KEY derivation (rotating IPs, spoofing
headers the limiter trusts for identity), or target the counter store
itself if it's reachable.
As covered in this series' OWASP Top 10 and API Security guides, a rate limiter is a security control, and adversaries treat it accordingly — key derivation (Section 2) needs to be based on genuinely hard-to-spoof identity signals (an authenticated user or API key, not a client-supplied, easily rotated header) for any rule meant to actually constrain a determined adversary, distinct from rules meant only to smooth ordinary traffic patterns.
Protecting the counter store from becoming its own attack surface
The counter store (Section 4) should be on a private network, not directly
reachable from outside the system, per this series' Network Security
guide — an attacker with direct access to the store could manipulate
counters directly, bypassing the limiter's enforcement logic entirely.
Per this series' Network Security and Secret Management guides, the counter store deserves the same access-control discipline as any other internal, sensitive infrastructure component — direct external reachability would let an attacker bypass the limiter's logic entirely by manipulating stored counts directly, rather than going through the checks this whole system exists to enforce.
Avoiding the limiter itself becoming a source of information leakage
Distinguishing a "rate limited" response from an "unauthenticated" or
"resource not found" response too precisely can leak information about
which identities or resources exist, per this series' OWASP Top 10 guide's
information disclosure discussion — worth a deliberate check for any rule
applied to sensitive or enumerable identifiers.
Worth a brief, deliberate check per this series' OWASP Top 10 guide: a rate limiter's response for a valid-but-limited identity versus an invalid one shouldn't inadvertently reveal which identities are valid (a subtly different response timing or error message for "this user exists and is rate-limited" versus "this user doesn't exist" is exactly the kind of narrow information-disclosure gap worth closing deliberately for any limit keyed on potentially-sensitive or enumerable identifiers.
13. Consistency, Availability, and the CAP Trade-off for a Rate Limiter
Why "approximately correct, fast, and available" usually beats "exactly correct, slow, and fragile" here
As covered in this series' System Design guide's CAP theorem discussion, a rate limiter is one of the clearer cases in this series' collection where near-perfect precision isn't actually the goal — per Section 1's framing, a limiter that's occasionally off by a few requests in either direction, but stays fast and available under real concurrent load, is almost always the better system than one that's exactly precise but adds meaningful latency or becomes a single point of failure.
Where the trade-off shifts — genuinely security-critical limits
A limit protecting against credential stuffing or brute-force login attempts
(Section 12) has a narrower tolerance for undercounting than an ordinary
API-usage limit — here, the trade-off deliberately shifts toward stronger
consistency, per Section 9's per-rule fail-closed policy, even at some cost
to Section 1's latency and availability goals.
This is the same per-rule reasoning Section 9 already introduced for fail-open/fail-closed policy, extended to the consistency trade-off itself — most limits can comfortably favor availability and approximate counting, but a narrow set of genuinely security-critical rules should deliberately accept more latency or stricter enforcement in exchange for closing the gap an approximate counter would otherwise leave for an adversary to exploit.
14. Scaling the System
Applying this series' System Design guide's building blocks, with limiter-specific emphasis
Sharding the counter store (per this series' Database Sharding and Redis
Cluster guides): by rate-limit key (Section 5's consistent hashing),
parallelizing counter throughput across shards the same way this series'
High-Volume Transaction Processing guide shards by account
Read-through local caching of NON-authoritative decisions (per this series'
Caching guide): a very short-TTL local cache of "definitely still allowed"
results can shave a network round trip off the common case, while any
cache miss or expiry falls back to the authoritative check
Connection pooling and pipelining (per this series' Redis guide): batching
or pipelining multiple checks where a single request needs several
simultaneous rule evaluations (Section 6) reduces round-trip overhead
Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against Section 1's latency budget and Section 13's precision-vs-speed trade-off before being applied — a caching layer that would be an unambiguous win elsewhere needs a specifically short TTL here, since a stale "allowed" decision cached too long could let a client blow past its limit for the duration of that staleness.
Horizontal scaling of the limiter service itself, decoupled from the counter store's own scaling
Per this series' Microservices guide: the STATELESS limiter service (the
component evaluating rules and calling the counter store) scales
independently and trivially — all the genuinely hard scaling work is in
the counter store (Section 4), which is why that store's own architecture
gets the greater share of this guide's scaling attention.
Because the limiter service itself holds no state (Section 4 pushed all of it into the counter store), scaling the service layer is comparatively simple horizontal scaling per this series' Microservices guide — the real scaling challenge, and the one worth the most design attention, is entirely in the shared counter store underneath it.
15. Observability for a Rate Limiter
Every guide in this series' observability trio, applied with critical-path-specific stakes
Structured logs (per this series' Structured Logging guide): rejected
requests with their key and rule, sampled at high volume given how
frequently this component is invoked relative to almost anything else
in the request path
Distributed tracing (per this series' Distributed Tracing guide): the
limiter's own check should appear as a clearly labeled, fast span in every
traced request — essential for spotting when the limiter itself becomes
a disproportionate share of a request's total latency
Metrics (per this series' Prometheus/Grafana guide): check latency (p50/p99),
allow/deny rate per rule, counter store error rate, fail-open/fail-closed
fallback activation count — the aggregate health signals an on-call
engineer watches continuously
Every technique from this series' observability guides applies directly, with one critical-path-specific addition worth stating explicitly: the limiter's own p99 latency deserves the same scrutiny this series' System Design guide gives to a system's slowest, most user-visible operation — because unlike most internal components, this one runs on literally every protected request, so even a small latency regression here has an outsized, multiplicative effect on overall system latency.
Alerting on limiter-health symptoms, distinct from the traffic patterns it's reporting on
# Per this series' Prometheus/Grafana guide's symptom-based alerting principle
rate(rate_limiter_fallback_activations_total[5m]) > 0
Any activation of Section 9's fallback policy is worth alerting on immediately, regardless of which direction (fail-open or fail-closed) it fell — it means the counter store itself is degraded, which is a meaningfully different, more urgent signal than an ordinary spike in legitimately rejected traffic, and the two are worth distinguishing clearly in dashboards so on-call response targets the actual problem (a struggling store) rather than mistaking it for a traffic spike.
16. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| In-memory, per-instance counters behind a load balancer | The effective limit silently becomes (configured limit × instance count) | Shared, external counter store (Redis or similar) that every instance reads and writes |
| Naive "GET, compute, SET" logic against the counter store | Reintroduces the exact read-modify-write race the store's atomicity was meant to prevent | Atomic single operations (INCR) or server-side Lua scripts for multi-step algorithms |
| Fixed window counters for anything sensitive to burst abuse | A client can send up to double the intended rate by straddling a window boundary | Sliding window counter (or log, if exact precision is required) |
| One universal limit for every caller and endpoint | Doesn't reflect real differences in trust level, subscription tier, or downstream fragility | Multi-dimensional, tiered limits evaluated per rule and per scope |
| No defined policy for counter-store unavailability | The limiter's own outage silently becomes either a full system outage or a wide-open bypass, by accident rather than decision | An explicit, per-rule fail-open/fail-closed policy, backed by a circuit breaker |
| Trusting client-supplied headers for rate-limit key identity | Trivially bypassed by an adversary who simply changes the header value | Key derivation from genuinely hard-to-spoof identity (authenticated user, API key) for security-sensitive rules |
| Relying on each node's local clock for window boundary calculations | Clock skew between nodes can shift window boundaries inconsistently | Defer window-boundary time to the counter store's own clock as the single source of truth |
| Compiling limits into application code | A sudden abuse pattern or tier change requires a full deployment to address | Rules stored as fast-to-update configuration, propagated to all instances via pub/sub or short TTL |
Quick Reference Table
| Concept | Purpose |
|---|---|
| Sliding window counter | The practical default algorithm, closing the fixed window's boundary-burst flaw affordably |
| Token bucket | The right choice when deliberate, capped bursts should be permitted on top of a steady rate |
| Atomic operations / Lua scripting on the counter store | Closes the read-modify-write race that concurrent traffic on hot keys makes routine, not rare |
| Consistent hashing across counter store shards | Keeps a given identity's checks on one shard, avoiding cross-shard coordination per check |
| Multi-dimensional, tiered rules | Reflects real differences in trust, subscription, and downstream fragility instead of one universal limit |
| Fail-open / fail-closed policy per rule | Makes the limiter's own outage behavior a deliberate decision, not an accident of implementation |
| Standard rate-limit headers + actionable 429 body | Lets well-behaved clients self-regulate, reducing unnecessary rejected-request volume |
| Fast, propagated configuration for rules | Lets limits respond to abuse or business changes on a minutes timescale, not a deployment timescale |
Conclusion
A rate limiter takes every general system design technique covered throughout this series and applies it to a component that's small in scope but disproportionately consequential, because it sits on the critical path of everything it protects and must stay correct under exactly the kind of high, concurrent contention on hot keys that most systems only occasionally have to worry about. The design that actually holds up rests on a small number of deliberate choices: an algorithm chosen to match the actual traffic shape being limited, not just the simplest one to implement; a shared, atomic-operation-capable counter store that closes the read-modify-write race concurrent traffic on popular keys makes routine; multi-dimensional, tiered rules that reflect real differences in trust and fragility rather than one blanket limit; and an explicit, per-rule policy for what happens when the limiter's own infrastructure degrades, since an accidental answer to that question is either a silent security bypass or a self-inflicted outage.
Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — Redis's atomic operations and scripting as the practical backbone, Distributed Systems' clock-skew and consistent-hashing discipline applied at a smaller scale than usual, Resilience's circuit breakers and fail-open/fail-closed policy, and the full observability trio watching a component whose own latency multiplies across every request it touches. A rate limiter is, in that sense, less a distinct discipline from everything else in this series than a compact, high-leverage proving ground for exactly the kind of concurrency and trade-off awareness the rest of this series argues matters everywhere else too.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the fixed-window-boundary-burst incident that turned out to matter far more than a synthetic load test ever revealed.
Top comments (0)