Almost every production API needs rate limiting eventually — to stop abuse, protect against traffic spikes, and keep one noisy client from degrading service for everyone else. Most tutorials tell you to add rate limiting, far fewer explain the actual tradeoffs between the different algorithms you could use. Picking the wrong one for your situation means either blocking legitimate traffic unnecessarily or failing to actually protect your system under load.
Here's a practical walkthrough of the main approaches, with working code for each.
Fixed Window Counter
The simplest approach: count requests in fixed time windows (e.g., per minute), reset the counter when the window ends.
import time
class FixedWindowLimiter:
def __init__(self, max_requests, window_seconds):
self.max_requests = max_requests
self.window_seconds = window_seconds
self.requests = {} # key -> (window_start, count)
def allow(self, key):
now = time.time()
window_start = int(now // self.window_seconds) * self.window_seconds
if key not in self.requests or self.requests[key][0] != window_start:
self.requests[key] = [window_start, 0]
self.requests[key][1] += 1
return self.requests[key][1] <= self.max_requests
The problem: This has a well-known edge case — the "boundary burst" issue. If a client sends max requests at the very end of one window, and max requests again at the very start of the next window, they've effectively sent 2x their limit in a short burst spanning the boundary, and the algorithm never notices.
When it's still fine: For rough, low-stakes limiting where occasional boundary bursts don't matter much, fixed window is simple and cheap. Don't reach for something more complex than your actual requirement needs.
Sliding Window Log
Keeps a timestamp for every request in a rolling window, and counts how many fall within the last N seconds from now — not from a fixed boundary.
from collections import deque
import time
class SlidingWindowLogLimiter:
def __init__(self, max_requests, window_seconds):
self.max_requests = max_requests
self.window_seconds = window_seconds
self.logs = {} # key -> deque of timestamps
def allow(self, key):
now = time.time()
if key not in self.logs:
self.logs[key] = deque()
log = self.logs[key]
# Drop timestamps outside the current window
while log and log[0] <= now - self.window_seconds:
log.popleft()
if len(log) < self.max_requests:
log.append(now)
return True
return False
The tradeoff: This is accurate — no boundary burst problem — but it costs memory proportional to the number of requests per client per window, which can get expensive at high request volumes or with many distinct clients.
Sliding Window Counter (The Practical Middle Ground)
Approximates the sliding window log's accuracy without storing every timestamp, by blending the current and previous fixed-window counts, weighted by how far into the current window you are.
import time
class SlidingWindowCounterLimiter:
def __init__(self, max_requests, window_seconds):
self.max_requests = max_requests
self.window_seconds = window_seconds
self.windows = {} # key -> {current_start, current_count, previous_count}
def allow(self, key):
now = time.time()
current_start = int(now // self.window_seconds) * self.window_seconds
state = self.windows.get(key)
if state is None or state['current_start'] != current_start:
previous_count = state['current_count'] if state and state['current_start'] == current_start - self.window_seconds else 0
state = {'current_start': current_start, 'current_count': 0, 'previous_count': previous_count}
self.windows[key] = state
elapsed_in_current = now - current_start
weight = 1 - (elapsed_in_current / self.window_seconds)
estimated_count = state['previous_count'] * weight + state['current_count']
if estimated_count < self.max_requests:
state['current_count'] += 1
return True
return False
Why this is often the best default: It's memory-efficient (just two counters per client, not a log of every request), and it smooths out the boundary burst problem well enough for the vast majority of real use cases. This is what most production rate limiters (including the one built into many API gateways) actually use under the hood.
Token Bucket
A different mental model entirely: each client has a "bucket" that holds tokens, refilled at a steady rate. Each request consumes one token; if the bucket is empty, the request is rejected. This naturally allows short bursts (up to the bucket's capacity) while enforcing a steady average rate over time.
import time
class TokenBucketLimiter:
def __init__(self, capacity, refill_rate_per_second):
self.capacity = capacity
self.refill_rate = refill_rate_per_second
self.buckets = {} # key -> (tokens, last_refill_time)
def allow(self, key):
now = time.time()
tokens, last_refill = self.buckets.get(key, (self.capacity, now))
elapsed = now - last_refill
tokens = min(self.capacity, tokens + elapsed * self.refill_rate)
if tokens >= 1:
tokens -= 1
self.buckets[key] = (tokens, now)
return True
self.buckets[key] = (tokens, now)
return False
Why teams choose this: Token bucket is the natural fit when you genuinely want to allow bursts — a client that's been idle should be able to send a quick flurry of requests, as long as their average rate over time stays within budget. This maps well onto real usage patterns (a user rapidly clicking through a UI after being idle, for instance) better than a strict per-window count does.
Leaky Bucket (The Opposite Philosophy)
Where token bucket allows bursts, leaky bucket smooths them out — requests go into a queue (the "bucket"), and are processed at a constant, fixed rate, regardless of how bursty the incoming traffic is.
This is less commonly implemented as a simple in-memory limiter and more often realized through an actual request queue with a fixed-rate worker pulling from it — useful when you need to protect a downstream system that genuinely can't handle bursts at all, even brief ones (a legacy system with a hard concurrency limit, for example).
Choosing Between Them
| Situation | Best fit |
|---|---|
| Simple, low-stakes limiting, don't want to over-engineer | Fixed window counter |
| Need accuracy and boundary bursts genuinely matter | Sliding window log (if volume is manageable) or sliding window counter (if not) |
| Want to allow natural bursts for idle-then-active clients | Token bucket |
| Protecting a downstream system that can't handle any burst at all | Leaky bucket / queue-based smoothing |
| Building a general-purpose API rate limiter with good default behavior | Sliding window counter |
A Practical Note on Distributed Systems
Every example above assumes a single in-memory store. The moment you're running multiple API server instances behind a load balancer, in-memory counters per instance won't give you an accurate global rate limit — a client could get max_requests allowed on each instance. For distributed rate limiting, the same algorithms apply, but the counters need to live in a shared store (Redis is the common choice), typically implemented with atomic operations (INCR + EXPIRE, or a Lua script for the more complex sliding window variants) to avoid race conditions between instances.
Final Thoughts
There's no universally "correct" rate limiting algorithm — the right choice depends on whether you're optimizing for memory efficiency, accuracy, or burst tolerance, and what your downstream system can actually handle. Understanding the tradeoffs behind each approach, rather than reaching for whatever the first tutorial you find implements, is what makes the difference between rate limiting that actually protects your system and rate limiting that just adds complexity without solving the real problem.
Prism Infoways builds and audits backend systems and APIs, including rate limiting and abuse-protection strategies, for growing products. More at prisminfoways.com.
Top comments (0)