The Quest Begins (The "Why")
I was building a tiny API service for a side‑project and kept getting slammed by bursty traffic. Every time a user refreshed a page, my simple in‑memory counter would spike, lock up the process, and bring the whole thing to its knees. I felt like Neo staring at the code rain, wondering if there was a way to see the hidden patterns behind the chaos.
The problem boiled down to a classic distributed systems dilemma: when my service runs on multiple instances behind a load balancer, how do I keep track of how many requests each user has made without turning the system into a bottleneck? I needed a rate limiter that could survive network hiccups, stay fast, and still give me sensible limits.
That’s when I remembered the CAP theorem — Consistency, Availability, Partition Tolerance — and realized I had been trying to have all three at once, which is, spoiler alert, impossible.
The Revelation (The Insight)
The CAP theorem says that in the presence of a network partition (P) you can only guarantee either Consistency (C) or Availability (A), not both. Think of it like choosing between two doors in a game show: you walk away with a prize, but you have to leave something behind.
For a rate limiter, the choice isn’t abstract. If I pick strong consistency (C), every node must agree on the exact request count before admitting a new request. That means a partition could make the limiter unavailable — requests get rejected or delayed until the nodes reconverge. If I pick availability (A), each node keeps its own counter and lets requests through even when they can’t talk to the others. The system stays responsive, but the count might drift, letting a user slip past the limit for a short window.
The “aha!” moment was realizing that for most APIs, a brief window of over‑limit is far less painful than turning the service off during a network glitch. In other words, I wanted an AP system: stay available, tolerate partitions, and accept eventual consistency.
Here’s a quick ASCII picture of what happens when a partition splits our three‑node cluster:
Client --> [LB] --> [Node A] <--net--> [Node B] <--net--> [Node C]
^ ^ ^
| | |
(count=5) (count=5) (count=5)
Partition occurs between A and B/C:
Client --> [LB] --> [Node A] (isolated)
|
(count=5)
Client --> [LB] --> [Node B] <--net--> [Node C]
| |
(count=5) (count=5)
Node A can’t see the increments happening on B and C, so its count lags. When the partition heals, we reconcile the counts (e.g., by taking the max or summing) and converge.
Wielding the Power (Code & Examples)
The Struggle: Naïve In‑Memory Limiter
// counter.go – a terrible idea for a clustered service
type Limiter struct {
mu sync.Mutex
counts map[string]int // userID -> request count
limit int
window time.Duration
}
func (l *Limiter) Allow(userID string) bool {
l.mu.Lock()
defer l.mu.Unlock()
l.counts[userID]++
if l.counts[userID] > l.limit {
return false
}
// reset after window (simplified)
go func() {
time.Sleep(l.window)
l.mu.Lock()
delete(l.counts, userID)
l.mu.Unlock()
}()
return true
}
Why it sucked:
- The
mulock serializes every request across all instances — if you run three replicas, you still have three independent locks, so the limit is per‑instance, not global. - When the network partitions, each side keeps counting independently, leading to wild overruns.
- No way to survive a process restart without losing state.
The Victory: Redis‑Backed AP Rate Limiter
I turned to Redis, using its built‑in atomic INCR and expiration features. The trick is to let each node talk to the same Redis cluster (which itself is AP‑oriented by default) and rely on Redis’s eventual consistency during partitions.
// redis_limiter.go
type RedisLimiter struct {
client *redis.Client
limit int
window time.Duration // seconds
keyPref string
}
func NewRedisLimiter(addr string, limit int, window time.Duration) *RedisLimiter {
rdb := redis.NewClient(&redis.Options{
Addr: addr,
DB: 0,
// tune for AP: disable strong consistency if your Redis cluster allows it
})
return &RedisLimiter{
client: rdb,
limit: limit,
window: window,
keyPref: "rl:", // prefix for keys
}
}
// Allow returns true if the request is under the limit.
func (l *RedisLimiter) Allow(userID string) (bool, error) {
key := l.keyPref + userID
// INCR is atomic; if the key doesn't exist it starts at 1
count, err := l.client.Incr(context.Background(), key).Result()
if err != nil {
return false, err
}
// Set expiration on first increment
if count == 1 {
l.client.Expire(context.Background(), key, l.window)
}
return count <= int64(l.limit), nil
}
What changed?
- Atomic increments guarantee that even if two nodes send requests at the same microsecond, Redis will serialize them correctly as long as the node can reach Redis.
- Expiration automatically clears old keys, so we don’t need a background goroutine.
-
During a partition, if a node can’t reach Redis, we simply return
true(allow the request) – we choose availability. The trade‑off is a brief window where a user might exceed the limit, but the system stays up.
Common trap: Forgetting to set the expiration on the first INCR. Without it, keys live forever and memory blows up. Always EXPIRE right after the first increment, or use Redis’s INCR with EXPIRE in a Lua script for extra safety.
Visualising the Trade‑off
+-------------------+ +-------------------+
| Node A (up) | | Node B (up) |
| Redis reachable |<------>| Redis reachable |
+-------------------+ +-------------------+
^ ^
| |
+----------+----------+ +----------+----------+
| Allow? (strict) | | Allow? (strict) |
+--------------------+ +--------------------+
| |
+----------v----------+ +----------v----------+
| Partition! | | Partition! |
| Node A can't talk | | Node B can't talk |
+--------------------+ +--------------------+
| |
+----------v----------+ +----------v----------+
| Allow? (AP) = true| | Allow? (AP) = true|
+--------------------+ +--------------------+
When the partition heals, Redis converges (because it’s eventually consistent) and the counts settle back to the true totals.
Why This New Power Matters
With this AP‑style Redis limiter, my side‑project now laughs at traffic spikes. I can push a new version, roll a node out for maintenance, or even survive a flaky ISP link without turning away users. The occasional over‑limit is barely noticeable — think of it as a tiny “grace period” that keeps the experience smooth.
If you do need hard guarantees (say, you’re charging per request and can’t afford any overage), flip the switch: use a strongly consistent store like etcd or a CP‑mode Redis cluster, and accept that a partition will make the limiter return false (block) until the partition heals. The same code structure works; you just change the error handling path.
The biggest win? Clarity of choice. Instead of guessing why my limiter sometimes behaved weirdly, I now know exactly which CAP side I’m on and can explain it to teammates, ops, or even a curious cat walking across the keyboard.
Your Turn
Grab a Redis instance (Docker’s redis:alpine is perfect), drop the RedisLimiter snippet into your service, and play with the limit and window. Try pulling the network plug on one of your replicas (or use docker network disconnect) and watch the limiter stay available while the counts drift a bit.
Challenge: Implement a toggle that lets you switch between AP (allow‑on‑partition) and CP (block‑on‑partition) modes at runtime, and log which mode you’re in during each request. Share your results — let’s see how many of us can keep the API alive while the network throws a tantrum!
Happy coding, and may your rate limits be ever in your favor. 🚀
Top comments (0)