Cross-post. Original: stellarbytecapital.com/blog/api-rate-limiting
Rate limiting looks trivial — "just count requests and block over N" — and quietly turns into a distributed-systems problem the moment you have more than one server, bursty traffic, or clients you actually care about. Done well, it protects your API from abuse and absorbs spikes without punishing legitimate callers. Done naively, it drops good requests, lets bad ones through, and lies to clients about when they can retry.
What you're actually protecting against
Be clear which goal you mean — the design differs:
- Overload protection — keep a spike from taking down the service.
- Fair use — stop one noisy client starving the rest.
- Abuse prevention — blunt brute-force, scraping, credential-stuffing.
- Cost control — cap expensive endpoints (an LLM call, a report).
The core algorithms
Fixed window. Count per calendar window ("100/min", reset on the minute). Simple, but a client can send 100 at 12:00:59 and 100 at 12:01:00 — 200 in one second. The boundary is a burst loophole.
Sliding window. Counts over a rolling window instead of a fixed one; a common variant weights the previous window's count as time advances, smoothing the edges. More accurate, slightly more state.
Token bucket (usually the best default). A bucket holds up to N tokens and refills at a steady rate; each request spends one. It allows controlled bursts — a quiet client accumulates tokens and can spend them in a spike — while capping the sustained average. Matches how well-behaved clients actually behave. The related leaky bucket enforces a perfectly smooth output rate for downstreams that can't tolerate bursts.
Fixed window is the one everyone writes first and regrets. Token bucket is the one they migrate to.
The hard part: distributed rate limiting
A counter in each server's memory works until you have two servers — now "100/min" becomes "100/min per instance", and ten instances give a client 1,000.
- Centralize the counter (typically Redis) using atomic operations / a Lua script so check-and-increment can't race.
- Mind the race. "Read, compare, increment" across instances double-counts under load. Make it atomic on the shared store, not in app code.
- Trade accuracy for latency where you can. Perfectly global limits add a network hop per request; approximate local limits with periodic sync trade slight overshoot for speed.
Scope: what are you limiting per?
Rarely a single global limit. Limit per API key/user for fairness and billing; per IP for anonymous abuse (careful behind NATs/proxies); per endpoint so an expensive route gets a tighter budget. Often several at once, and a request must pass all. This is exactly how exchanges budget their APIs — the client side is covered in exchange API integration.
Respond honestly — the client-facing contract
-
Return
429 Too Many Requests— not a generic 400 or 503. -
Send
Retry-Afterso the client knows when to try again instead of hammering. -
Expose limit headers (
X-RateLimit-Limit/-Remaining/-Reset) so clients can self-pace before hitting the wall.
A client that can see its remaining budget rarely trips the limit at all.
What to avoid
- Fixed windows for anything that matters — the boundary burst lets through 2× at the worst moment.
- Per-instance in-memory counters behind a load balancer — your real limit is silently N× what you configured.
- Non-atomic check-then-increment — it races and undercounts under exactly the load you're limiting for.
-
Silent drops or wrong status codes — clients can't back off without
429+Retry-After. - One global limit for everything — no per-key fairness, no per-endpoint protection.
Match the tool to the goal: token bucket for realistic bursty clients, a shared atomic counter once you're horizontal, layered per-key and per-endpoint scopes, and an honest 429 contract so clients cooperate.
We're Xingyao Byte — building payment platforms, quant trading systems, secure AI-execution layers, and reliable backends. Remote, async-first → stellarbytecapital.com
Top comments (0)