Why Distributed Rate Limiting Needs a Shared Store, Not Local Counters

#webdev #backend #api #scalability

A team implements rate limiting, tests it locally against a single server instance, watches it correctly reject the eleventh request in a ten-per-minute limit, ships it to production feeling confident the logic is solid, and discovers weeks later that a client is somehow sending forty requests per minute without ever getting rejected once. The rate limiter isn't broken, and nothing in the logs points to an obvious defect. It's working exactly as implemented. The problem is that "implemented" meant a counter stored in each server's local memory, and production has four servers behind a load balancer, each with its own independent view of how many requests that client has sent.

The Local Counter Trap

An in-memory counter per server is the simplest possible rate limiter to build, and it's genuinely correct for a single-instance deployment. The moment you scale horizontally, though, that same counter becomes silently wrong. If a client's requests get distributed across four servers by a load balancer, each server only sees roughly a quarter of that client's actual traffic. A limit of ten requests per minute, enforced independently by four servers each unaware of what the others are counting, effectively becomes a limit of forty requests per minute for any client whose traffic happens to spread evenly across all four instances.

This failure mode is particularly dangerous because it's invisible in normal testing, and it doesn't announce itself with an error, a crash, or anything that shows up in a typical alerting dashboard. A local development environment running a single instance enforces the limit perfectly. Staging environments with a single instance also enforce it perfectly. The gap only appears once you're running multiple instances in production, at exactly the traffic volume where the rate limiter's actual job, protecting shared infrastructure, matters most.

Why a Shared Store Fixes This

The fix is centralizing the count in a store every server instance can read from and write to, rather than each instance keeping its own local count. Redis is the most common choice for this, since it's fast enough to check on every request without adding meaningful latency, and its atomic increment operations let multiple servers safely update the same counter concurrently without a race condition producing an incorrect count.

With a shared store, all four servers in the earlier example check and update the same counter for a given client, so the client's actual limit is enforced correctly regardless of which server happens to handle any individual request. The load balancer's distribution behavior becomes irrelevant to the correctness of the rate limit, which is exactly the property you want, since you generally don't want your rate limiting correctness to depend on assumptions about how traffic happens to get routed.

The Latency Tradeoff

Checking a shared store on every request adds a network round-trip that a purely local counter doesn't have. For most applications this cost is small, a Redis lookup is typically sub-millisecond when the store is properly co-located with the application servers, but it's not free, and it's worth measuring rather than assuming it's negligible for latency-sensitive endpoints. Some systems mitigate this with a local cache layer that periodically syncs with the shared store, trading perfect real-time accuracy for reduced latency, which is often an acceptable tradeoff since rate limiting rarely needs to be exact to the single request.

Handling the Shared Store's Own Failure Modes

Centralizing rate limit state in a shared store introduces a new failure mode that a local counter never had: what happens if the shared store itself is unavailable. A rate limiter that fails closed (rejecting all requests when it can't reach its store) turns a Redis outage into a full API outage, which is usually a worse outcome than the rate limiter briefly not enforcing limits at all. Most production systems fail open in this specific case, allowing requests through without enforcement during a shared-store outage, and treating the shared store's own availability as a separate reliability concern with its own monitoring and redundancy rather than a single point of failure for the whole API.

This Is a Specific Case of a General Distributed Systems Problem

Rate limiting across multiple servers is really just one instance of a much broader pattern: any counter, lock, or piece of coordinated state that needs a consistent view across multiple independent processes needs a shared source of truth, not independent local state that happens to look correct in isolation. The same underlying lesson shows up in distributed locking, deduplication, and session state, anywhere multiple servers need to agree on a fact rather than each maintaining their own potentially divergent copy of it.

Choosing the right rate limiting algorithm on top of a correctly shared counter, token bucket versus sliding window versus fixed window, and sizing limits so legitimate burst traffic doesn't get needlessly rejected, is covered in more depth in How to Design a Rate Limiter That Doesn't Punish Legitimate Bursts. That piece assumes correctly shared state as a starting point; this piece is about how to actually get there once you've scaled past a single server instance.

Testing for This Specific Failure Mode

Because this bug only appears at multi-instance scale, and never shows up in a single-instance test no matter how thorough, it's worth deliberately testing for it rather than assuming local testing already caught it. Spin up multiple instances of your API behind a load balancer in a staging environment, hammer it with traffic from a single simulated client, and confirm the aggregate request count across all instances actually respects the configured limit, not just the count any individual instance sees. This is one of the few classes of bug where correct behavior in a single-instance test tells you almost nothing about correct behavior in the production topology that actually matters.

A Related Trap: Deploys That Reset State

Even a correctly shared counter can get quietly reset in a way that undermines the limit's intent if the shared store itself gets restarted or redeployed on the same cadence as your application servers. A rolling deploy that happens to also cycle the Redis instance backing your rate limiter resets every client's counted usage to zero, which briefly grants every client a fresh full quota right as a deploy completes. This is rarely catastrophic on its own, but it's worth being aware of if you notice a recurring pattern of unusually permissive behavior clustered around deploy windows specifically, since that's a strong signal the rate limit state and the deploy cycle are coupled in a way that wasn't intended.