Rate limiting sounds simple. Count requests. Block the extra ones. But when you actually build it, things get complicated fast. This article explains the main ideas, the algorithms, and the real problems that appear in production.
What Is a Rate Limiter?
A rate limiter watches incoming requests and decides: allow this one, or block it?
Every rate limiter needs three things:
- Limit: the maximum number of requests allowed. Example: 100 requests.
- Window: the time period for that limit. Example: per minute.
- Identifier: how the system knows who is sending the request. Example: the user's ID, API key, or IP address.
So a complete rule looks like: "100 requests per minute per user ID."
The identifier matters more than most people think. IP address seems like a natural choice, but it breaks down quickly. A company office with 500 employees may share one IP address. A university network does the same. If you use IP as your identifier, you can accidentally block hundreds of innocent users because one person on that network sent too many requests. API keys and user IDs are much better options when you have them.
The Algorithms
There are five common approaches. Each one works differently and fits different situations.
1. Token Bucket
Think of a bucket that holds tokens. Each user has their own bucket.
- The bucket has a maximum size, for example 10 tokens.
- Tokens are added at a fixed rate, for example 1 token per second.
- Each request uses 1 token.
- If the bucket is empty, the request is blocked.
The bucket size and the refill rate are two different settings. The refill rate controls how many requests the user can make over time. The bucket size controls how many they can make at once, in a burst.
A user with a bucket of 10 and a refill rate of 1 per second can send 10 requests in one second, then 1 request every second after that.
Bursts are not a problem with this algorithm. They are the point. Use token bucket when your users sometimes send several requests at once and that is normal behavior.
2. Leaky Bucket
Imagine a bucket with a small hole at the bottom.
- Requests come in from the top (like water being poured in).
- Requests are processed at a fixed rate through the hole at the bottom.
- If too many requests arrive at once, the bucket overflows and those requests are dropped.
This always processes requests at the same rate. No matter how many requests arrive, the output is smooth and steady.
The downside: if a user sends no requests for a while, that idle time is wasted. When they come back, they get no extra allowance for the quiet period. Token bucket saves unused capacity; leaky bucket does not.
Also, requests wait in the queue instead of getting rejected immediately. For web APIs, a fast rejection is usually better than a long wait.
3. Fixed Window Counter
Time is divided into fixed blocks. For example, one block per hour.
- Each block has a request limit: 5 requests per minute.
- The system counts requests in the current block.
- If the count reaches the limit, new requests are blocked.
- When the block ends, the counter resets to zero.
This is the easiest algorithm to build and understand.
The known problem: a user can send double their limit in a short time by timing it at the boundary between two blocks. Send 5 requests at 11:59:59, then 5 more at 12:00:01. That is 10 requests in 2 seconds, even though the limit is 5 per minute. This is not a rare edge case. Users who want to send as many requests as possible figure this out.
Use fixed window when your system can handle that occasional double-burst and you want the simplest possible implementation.
4. Sliding Window Log
Instead of resetting a counter every hour, this algorithm keeps a record of the exact time of every request.
- When a new request arrives, delete all old records outside the window.
- Count the remaining records.
- If the count is at the limit, block the request.
This is the most accurate approach. The count is always exact. No boundary problem like fixed window.
The cost is memory. Every request from every user needs a stored timestamp. At high traffic, this adds up fast. Use this when accuracy is critical and traffic is not too high.
5. Sliding Window Counter
This is a lighter version of the sliding window log. Instead of storing every timestamp, it keeps just two numbers: the count from the previous window and the count from the current window.
It estimates the real rolling count with this formula:
rolling_count = previous_count × (1 - elapsed_fraction) + current_count
For example, if 30% of the current hour has passed, it assumes 70% of the previous hour's requests are still inside the rolling window. This is an estimate, not an exact count. The error is usually small.
This is a good middle ground. More accurate than fixed window, much cheaper in memory than sliding window log. It is a good default for most public APIs.
What Happens When a Request Is Blocked
When a request exceeds the limit, there are three ways to respond.
Blocking: reject the request immediately with HTTP 429 Too Many Requests. Include a Retry-After header so the client knows when to try again. This is the simplest and most common approach.
Throttling: instead of rejecting, slow the request down. Make it wait before processing. The request eventually succeeds, just with added delay. The downside is that the server still holds the connection open while waiting.
Shaping: allow the request but give it lower priority. High-traffic users go to the back of the line. Normal users are served first. CDNs use this for users who have exceeded their bandwidth limit.
For most APIs, just use blocking. It is fast, clear, and easy for clients to handle.
The Distributed Problem
Everything above works fine on one server. Most production systems run on multiple servers at the same time.
If each server keeps its own counter in memory, each server applies the limit independently. With 10 servers and a limit of 100 requests per minute, a user can actually send 1000 requests per minute by spreading their traffic across all 10 servers.
The fix is to use a shared counter that all servers read and write together. Redis is the standard tool for this. Every request goes to Redis to check and update the counter. Redis handles the counting atomically, meaning two servers cannot read the same number at the same time and both allow a request that should have been blocked.
The cost is one extra network call on every request. At high traffic, that adds latency.
There is also a question worth deciding before you go to production: what happens if the rate limiter itself crashes? Two options:
- Fail open: all requests pass through while the limiter is down. Users are not affected, but your infrastructure is unprotected.
- Fail closed: all requests are blocked while the limiter is down. Your infrastructure is safe, but real users cannot access your API.
Neither option is wrong. It depends on what you are protecting. Decide this early.
Where to Put the Rate Limiter
A rate limiter can sit in different places in your system.
API gateway: sits in front of your entire application. Best for blocking abuse before it ever reaches your servers. One configuration applies to everything.
Application middleware: lives inside your service. Needed when different endpoints have different limits, or when the limit depends on something the gateway does not know about, like a user's subscription plan.
For most systems: use the gateway for general protection, use application-level limits where the rules are more specific.
Which Algorithm Should You Use
| Algorithm | Best for | Main cost |
|---|---|---|
| Token Bucket | Bursty but legitimate traffic | Irregular load on downstream services |
| Leaky Bucket | Smooth, predictable output rate | Wastes idle capacity, slow rejection |
| Fixed Window | Simplicity matters, bursts are acceptable | Boundary exploit allows 2x burst |
| Sliding Window Log | Maximum accuracy, low traffic | High memory usage |
| Sliding Window Counter | Public APIs, balanced accuracy and cost | Slightly approximate count |
The algorithm matters, but the identifier and placement matter just as much. A good algorithm on the wrong identifier still lets abuse through.
Top comments (0)