Rate Limiting Like a Boss: Lessons from **The Matrix**

The Quest Begins (The "Why")

I still remember the night our API started choking under a sudden traffic spike. Users were seeing 503 errors, the monitoring dashboard lit up like a Christmas tree, and I felt like Neo dodging bullets—except the bullets were request bursts and I had no slow‑motion reflexes. We had a simple in‑memory counter on each service instance, and as soon as we added a third node the limits went haywire: each node thought it was under the threshold while the aggregate traffic blew past our quota. It was the classic “split‑brain” problem, and we needed a shared source of truth that could keep up with millions of requests per second without becoming a bottleneck.

The dragon we were slaying? Designing a distributed rate limiter that’s both accurate and fast enough for real‑world traffic. Spoiler: the answer lived in Redis, and the real magic was a tiny Lua script that turned a simple key‑value store into a bullet‑proof gatekeeper.

The Revelation (The Insight)

The breakthrough came when I stopped thinking about “counters” and started thinking about time windows. A rate limiter isn’t just a number that goes up and down; it’s a view of how many requests have arrived in the last N seconds. If we can efficiently query that view atomically, we get both correctness and performance.

Here’s the insight that made everything click:

Store timestamps of each request in a Redis sorted set, trim the set to the window, and check its cardinality—all inside a single Lua script.

Why does this work?

• Sorted sets give us O(log N) insertion and automatic ordering by timestamp.
• The ZREMRANGEBYSCORE command lets us drop old entries in one shot.
• ZCARD returns the current count.
• Wrapping these three calls in a Lua script guarantees they run atomically—no race conditions between checking and updating.

It felt like discovering the One Ring’s power: a small, unassuming artifact that, when used correctly, controls massive influence.

Trade‑offs we weighed

Option	Pros	Cons
Per‑instance in‑memory counter	Zero network latency, simplest code	Inaccurate under horizontal scaling; needs sticky sessions or over‑provisioning
Redis INCR with EXPIRE	Easy to understand, low latency	Counter resets on expiry leads to bursty traffic at window edges; no sliding window
Redis sorted set + Lua (chosen)	Accurate sliding window, atomic, works with any number of nodes	Slightly higher CPU on Redis (still negligible for typical QPS), requires scripting discipline
External service (e.g., Envoy rate‑limit)	Offloads logic, integrates with mesh	Adds another hop, operational overhead, licensing concerns

The sorted‑set approach gave us the best of both worlds: correctness comparable to a centralized lock‑free counter and latency low enough to sit in the critical path of our API gateway (sub‑millisecond p99 on a modest Redis cluster).

Wielding the Power (Code & Examples)

The Struggle – Naïve per‑instance limiter

// WARNING: This is the "before" code – don't use it in prod!
type memLimiter struct {
    mu      sync.Mutex
    count   int
    window  time.Duration
    reset   time.Time
}

func (l *memLimiter) Allow() bool {
    l.mu.Lock()
    defer l.mu.Unlock()
    now := time.Now()
    if now.Sub(l.reset) > l.window {
        l.count = 0
        l.reset = now
    }
    if l.count >= l.max {
        return false // reject
    }
    l.count++
    return true
}

Problem: Each replica has its own count. With three nodes and a limit of 100 req/s, we could actually allow 300 req/s before anyone notices.

The Victory – Distributed sliding‑window limiter with Redis

First, the Lua script (saved as rate_limiter.lua):

-- KEYS[1] = redis key for this identifier (e.g. "rate:user:123")
-- ARGV[1] = window size in seconds
-- ARGV[2] = maximum requests allowed
-- ARGV[3] = current unix time as float

local now = tonumber(ARGV[3])
local window = tonumber(ARGV[1])
local max = tonumber(ARGV[2])

-- Remove timestamps older than now - window
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window)

-- Count current requests
local current = redis.call('ZCARD', KEYS[1])

if current >= tonumber(max) then
    return 0 -- not allowed
end

-- Add this request
redis.call('ZADD', KEYS[1], now, now .. ':' .. math.random(1,10000))
-- Set expiry so the key auto‑cleans if idle
redis.call('EXPIRE', KEYS[1], window)
return 1 -- allowed

Now the Go wrapper:

type redisLimiter struct {
    client *redis.Client
    script *redis.Script
    window time.Duration
    max    int64
}

func newRedisLimiter(client *redis.Client, window time.Duration, max int64) *redisLimiter {
    src, _ := os.ReadFile("rate_limiter.lua")
    return &redisLimiter{
        client: client,
        script: redis.NewScript(string(src)),
        window: window,
        max:    max,
    }
}

func (r *redisLimiter) Allow(ctx context.Context, key string) (bool, error) {
    now := float64(time.Now().UnixNano()) / 1e9
    res, err := r.script.Run(ctx, r.client, []string{key},
        r.window.Seconds(), r.max, now).Result()
    if err != nil {
        return false, err
    }
    allowed, _ := res.(int64)
    return allowed == 1, nil
}

What we avoided (the traps):

1. Calling ZADD then ZCARD separately – without the Lua wrapper a race condition could let two concurrent requests both see a count below the limit and both be added, briefly exceeding the quota.
2. Forgetting to set an expiry – the sorted set would grow forever with stale timestamps, eating memory. The EXPIRE call (or a periodic cleanup) keeps the key size bounded by the window.
3. Using millisecond precision as the score – while fine, we must ensure the random suffix (now':'rand) makes each entry unique; otherwise ZADD would ignore duplicates and under‑count.

Quick benchmark (local Redis, 50k req/s)

Implementation	99th‑pct latency	CPU usage (Redis)	Accuracy
Per‑mem counter	0.2 ms	0 % (local)	±30 % under scale
Redis INCR+EXPIRE	0.7 ms	2 %	±5 % (burst at edges)
Sorted‑set + Lua	0.9 ms	3 %	Exact sliding window

The extra ~0.7 ms is a tiny price for guaranteed correctness—especially when you consider the cost of a single 503 error in terms of lost trust and potential SLA penalties.

Why This New Power Matters

Armed with this limiter, our service now laughs at traffic spikes. We can safely expose APIs to third‑parties, enforce per‑user quotas, and even implement adaptive back‑off without fear of accidental over‑grant. The same pattern works for any Redis‑backed store: API keys, IP addresses, session IDs—just change the key.

More importantly, the mindset shift from “counting” to “querying a time‑bounded set” opened doors to other problems: sliding‑window metrics, incremental rollouts, and even simple leaderboard implementations (just store scores instead of timestamps).

If you ever find yourself wrestling with distributed state, ask yourself: Can I model this as a sorted set with a temporal dimension? More often than not, the answer will be a resounding yes, and you’ll have a new spell in your toolkit.

Your Turn – A Small Quest

Grab a Redis instance (Docker’s redis:alpine works fine) and try implementing a per‑IP sliding‑window limiter for a toy HTTP handler. Play with different window sizes and limits, then hammer it with wrk or hey. Notice how the latency stays flat while the count stays honest.

When you get it working, drop a comment below with your favorite tweak or a surprising edge case you discovered. I’m excited to see what you build—happy limiting! 🚀