What Is Rate Limiting?
Imagine a busy café where hundreds of customers arrive at the same time.
The café has limited staff and equipment, so it cannot handle unlimited
orders at once. The manager sets a rule to control how many orders can
be placed within a certain time.
Software systems use the same idea. Rate limiting controls how many
requests a user, application, or IP address can make within a specific
time period.
For example, an API might allow each user to make 100 requests per
minute. If a user exceeds the limit, the system can reject extra
requests, usually with HTTP status code 429 Too Many Requests.
How Does Rate Limiting Work?
Think of the café manager as the rate limiter.
- A customer sends an order. In software, this is an API request.
- The manager checks how many orders that customer has already placed during the time window.
- If the customer is within the limit, the order is accepted.
- If the customer has exceeded the limit, the request is rejected or delayed according to the policy.
In a real application, the rate limiter might run in an API gateway,
reverse proxy, application middleware, or dedicated service. It usually
tracks usage by a key such as a user ID, API key, IP address, or
endpoint.
Why Do We Need Rate Limiting?
1. To Protect Servers from Overload
Too many requests can exhaust CPU, memory, database connections, or
other resources. Rate limiting helps control incoming traffic so the
system has a better chance of staying responsive.
2. To Prevent Abuse
A login API without limits could be targeted by someone repeatedly
trying passwords or requesting OTPs. Rate limiting makes excessive
attempts harder. It should be combined with authentication, monitoring,
and abuse detection.
3. To Ensure Fair Usage
Imagine one café customer placing hundreds of orders while everyone else
waits. Per-user limits help prevent one client from consuming all
available capacity.
4. To Control Costs
Some operations are expensive, such as AI generation, sending SMS
messages, or calling paid third-party APIs. Rate limits can help control
usage and reduce unexpected costs.
Where Is Rate Limiting Used?
- Login and OTP APIs: Limit repeated login attempts or OTP requests.
- Public APIs: Keep usage within published quotas.
- Search endpoints: Prevent excessive or automated searches.
- Payment APIs: Protect services from excessive requests while preserving correct payment handling.
- AI APIs: Limit requests or tokens to manage capacity and cost.
- Social media platforms: Limit actions such as posting, commenting, or sending messages.
The limit should fit the operation. A password-reset endpoint may need a
stricter limit than a public read-only endpoint.
Common Rate Limiting Algorithms
1. Fixed Window Counter
Imagine the café resets each customer's order counter at the start of
every minute. A customer can place up to 100 orders during that minute.
The system counts requests in fixed time windows, such as 10:00--10:01
and 10:01--10:02.
Advantages: Simple to implement and efficient.
Disadvantage: Traffic can spike around a window boundary. A user
could make 100 requests just before the minute ends and another 100
immediately after it begins.
2. Sliding Window Log
The system records request timestamps. For each new request, it counts
how many requests occurred in the previous time period, such as the last
60 seconds.
Advantages: Accurate rolling-window enforcement.
Disadvantages: Storing many timestamps can use significant memory
and processing, especially for high-volume clients.
3. Sliding Window Counter
This approach combines counts from the current and previous fixed
windows, weighting the previous window according to how much of it
overlaps the rolling time period.
Advantages: A practical balance between accuracy and memory use.
Disadvantages: It estimates the rolling count rather than tracking
every request timestamp exactly.
4. Token Bucket
Imagine each customer has a bucket containing tokens. Every request
costs one token, and tokens refill at a steady rate. If a token is
available, the request is allowed. If the bucket is empty, the request
is rejected or delayed.
A bucket with a capacity of 10 tokens might allow a burst of 10
requests, while refilling at 2 tokens per second.
Advantages: Allows controlled bursts while limiting the average
request rate.
Disadvantages: Requires choosing the right bucket capacity and
refill rate.
5. Leaky Bucket
Imagine orders entering a container and leaving at a steady pace. If
orders arrive faster than they can be processed, they wait in the
container. If it fills up, new orders are rejected. Depending on
implementation, a leaky bucket may queue requests or reject excess
requests.
Advantages: Smooths incoming traffic and helps control output rate.
Disadvantages: A queue can add latency, and excess requests still
need a defined policy.
Which Algorithm Should You Choose?
Requirement Possible choice
Simple quota with low implementation complexity Fixed Window
Accurate rolling time limit Sliding Window Log
Efficient rolling-window approximation Sliding Window Counter
Allow short bursts while controlling average rate Token Bucket
Smooth traffic into a steady rate Leaky Bucket
There is no single best algorithm for every system. Choose based on
whether bursts should be allowed, how accurate the limit must be,
available storage, and the traffic pattern.
How to Implement Rate Limiting in a Real System
Step 1: Define the Limit
Decide what is being limited and the allowed rate. For example, 100
requests per minute per authenticated user for a search endpoint.
Step 2: Choose the Rate-Limit Key
Common keys include user ID, API key, IP address, or a combination of
user and endpoint. IP-only limits can affect multiple legitimate users
behind the same network, while user-only limits may not protect an
unauthenticated endpoint.
Step 3: Choose Where the Limiter Runs
An API gateway or reverse proxy can apply limits before requests reach
application servers. Application middleware can enforce rules close to
the endpoint. A shared service or data store can coordinate limits
across multiple application instances.
Step 4: Store and Update Usage Safely
If the application runs on multiple servers, a local in-memory counter
on each server may allow a user to exceed the intended global limit. A
shared store such as Redis can help coordinate counters, but updates
must be atomic and the design should account for latency and store
failures.
Step 5: Decide What Happens When the Limit Is Reached
Common policies include rejecting the request, delaying it, or placing
eligible work in a queue. For HTTP APIs, return 429 Too Many Requests.
If useful, include a Retry-After header to indicate when the client
can try again.
Step 6: Monitor and Tune
Track rejected requests, traffic patterns, latency, and usage by
endpoint or client. Adjust limits based on real capacity and legitimate
user behavior.
Example: An API with a Limit of 5 Requests per Minute
Suppose a user can make five search requests per minute.
- Requests 1--5: allowed.
- Request 6 during the same applicable window: rejected with
429 Too Many Requests. - After enough time has passed for the algorithm's window or tokens to reset or refill, requests can be allowed again.
The exact reset behavior depends on the chosen algorithm. A token bucket
refills gradually rather than necessarily resetting all at once.
Important Design Considerations
Rate Limiting vs. Load Shedding
Rate limiting enforces usage rules, often per user or client. Load
shedding rejects selected work when a system is overloaded to help
protect overall health. A production system may use both.
Rate Limiting vs. Throttling
These terms are sometimes used interchangeably. Rate limiting defines or
enforces a request rate; throttling can also mean slowing down or
delaying requests rather than immediately rejecting them.
Distributed Systems
When requests are served by several application instances, the system
needs a consistent way to apply the limit. A shared data store or
gateway can help, but it introduces its own availability and latency
trade-offs.
Client Behavior
Clients should handle 429 responses sensibly. They can wait before
retrying, respect Retry-After when provided, and use exponential
backoff with jitter where appropriate. Retrying immediately can make
overload worse.
Rate Limits Are Not a Complete Security Solution
Rate limiting helps reduce abuse, but it does not replace
authentication, authorization, fraud detection, input validation, or
monitoring.
Quick Interview Question
Question: An API allows 100 requests per minute. A user sends 100
requests in the first 10 seconds and then sends one more request during
that minute. Should the 101st request be allowed?
Answer: Under a fixed-window limit of 100 requests for that minute,
the 101st request should be rejected. However, the answer can differ
with a token bucket or another policy, so state the algorithm and
time-window behavior.
Follow-up: How would you enforce the limit when the application runs
on 10 servers?
A strong answer mentions a shared rate-limit store or gateway, atomic
updates, a clear key such as user ID or API key, and a plan for what
happens if the shared store is unavailable.
Key Takeaway
Rate limiting is like a café manager controlling how many orders each
customer can place in a given time. It protects system capacity,
supports fair usage, reduces abuse, and controls costs. The right design
depends on the required limit, whether bursts are allowed, and how
requests are distributed across servers.
Top comments (0)