Leveling Up: Building a Real-Time Notification System at Scale

The Quest Begins (The "Why")

I still remember the day our chat app started feeling like a crowded subway at rush hour. Users were complaining that notifications arrived minutes late—or worse, not at all. We had a flood of events: messages, reactions, mentions, and every single one needed to be fanned out to thousands of connected clients in near‑real time.

Our first instinct was to blast every event straight into a Pub/Sub topic and let each subscriber push it out via WebSocket. Sounds simple, right? The reality hit us when we looked at the metrics: the broker was choking, CPU spikes were through the roof, and our downstream services were dropping connections like they were hot potatoes.

We were trying to slay a dragon with a toothpick. The dragon? Thundering herd—a sudden surge of identical notifications that overwhelmed our workers and left users staring at stale screens. We needed a way to smooth the burst, protect our backend, and still guarantee that no important update got lost.

The Revelation (The Insight)

After a few sleepless nights and a lot of coffee, the breakthrough came from a surprisingly humble concept: rate limiting. Not the kind that blocks users, but a smart rate limiter that shapes the outflow of notifications so our workers see a predictable, manageable traffic pattern.

The key insight? Treat the notification pipeline like a leaky bucket. Instead of letting every event rush in, we let them drip out at a steady rate, while allowing short bursts for genuine spikes (think a sudden wave of reactions during a live event). By placing a token‑bucket limiter right before the worker pool, we decouple the irregular ingestion rate from the steady processing capacity.

Why does this beat a simple fixed‑window counter or a naive “drop‑if‑over‑limit” approach?

• Fixed windows cause the dreaded “burst at the edge of the window” problem—you can still get a huge spike right after the window resets.
• Dropping events outright risks losing important notifications (imagine missing a friend’s reply because you hit a limit).
• A token bucket lets us borrow capacity from idle periods and save it for bursts, all while guaranteeing a long‑term average rate.

It felt like when Neo dodges bullets in The Matrix: we weren’t stopping the bullets; we were redirecting them so they never hit us.

ASCII diagram of the flow

+----------------+       +------------------+       +---------------------+
|   Event Ingest | ---> | Token Bucket     | ---> |   Worker Pool (N)   |
|   (Kafka/Pub)  |       | (Rate Limiter)   |       |   (process & push)  |
+----------------+       +------------------+       +---------------------+
        ^                          |                         |
        |                          v                         v
   Bursty stream          Smooth outflow               Steady fan‑out
                           (tokens per sec)          to WebSocket gateways

Wielding the Power (Code & Examples)

The “before” – a naive rate limiter that almost broke us

# naive_fixed_window.py
import time
from collections import defaultdict

class FixedWindowLimiter:
    def __init__(self, max_events, window_sec):
        self.max_events = max_events
        self.window = window_sec
        self.counters = defaultdict(int)   # key -> count
        self.reset_times = defaultdict(float)

    def allow(self, key):
        now = time.time()
        if now - self.reset_times[key] > self.window:
            # reset window
            self.counters[key] = 0
            self.reset_times[key] = now

        if self.counters[key] < self.max_events:
            self.counters[key] += 1
            return True
        return False

What went wrong?

• At t = window + ε we reset the counter, allowing another max_events burst instantly → double‑batch problem.
• No borrowing: if we were idle for half a window, we couldn’t use that saved capacity later.

The “after” – a token bucket limiter using Redis (the battle‑tested spell)

# token_bucket_redis.py
import time
import redis

class TokenBucket:
    """
    Redis‑backed token bucket.
    refill_rate: tokens added per second
    capacity:   max tokens the bucket can hold
    """
    lua_script = """
        local key = KEYS[1]
        local now = tonumber(ARGV[1])
        local rate = tonumber(ARGV[2])
        local capacity = tonumber(ARGV[3])
        local requested = tonumber(ARGV[4])

        local last_ts = redis.call('HGET', key, 'last_ts')
        local tokens = redis.call('HGET', key, 'tokens')
        if last_ts == false then
            last_ts = now
            tokens = capacity
        else
            last_ts = tonumber(last_ts)
            tokens = math.min(capacity, tonumber(tokens) + (now - last_ts) * rate)
        end

        local allowed = 0
        if tokens >= requested then
            tokens = tokens - requested
            allowed = 1
        end

        redis.call('HMSET', key, 'tokens', tokens, 'last_ts', now)
        redis.call('EXPIRE', key, 3600)   # optional TTL
        return allowed
    """

    def __init__(self, redis_client, key, refill_rate, capacity):
        self.r = redis_client
        self.key = key
        self.rate = refill_rate
        self.capacity = capacity
        self.sha = self.r.script_load(self.lua_script)

    def allow(self, requested=1):
        now = time.time()
        result = self.r.evalsha(self.sha, 1, self.key, now, self.rate, self.capacity, requested)
        return bool(result)

How to plug it into the pipeline

# worker_dispatcher.py
r = redis.Redis(host='redis', port=6379, db=0)
bucket = TokenBucket(r, key='notif_limiter', refill_rate=500, capacity=1500)  # 500 msg/s, burst up to 1500

def handle_event(event):
    if bucket.allow():
        dispatch_to_workers(event)   # goes to the pool in the diagram
    else:
        # Optional: park the event in a delayed queue for retry
        delayed_queue.put(event)

Traps to avoid (the “boss levels”)

Trap	Why it’s deadly	How we dodge it
Using INCR + EXPIRE per request	Two round‑trips + race conditions → inaccurate counts under load	Use a single Lua script (atomic) as shown above
Setting capacity too low	Legitimate bursts get throttled, causing delayed notifications	Size capacity for the maximum expected burst (e.g., 3× sustained rate)
Forgetting to update last_ts on every call	Tokens never refill → bucket drains permanently	The Lua script always writes back the current timestamp
Ignoring Redis latency	Adds jitter to the allowed rate	Keep Redis close (same AZ) or use a clustered setup with read replicas for the bucket

Why This New Power Matters

With the token bucket in place, our notification system went from “panic mode” to “steady cruise.”

• Throughput became predictable. Workers now see a smooth 500 msg/s stream, autoscaling based on a stable metric instead of reacting to spiky CPU usage.
• Bursts are absorbed. When a live stream drops a thousand reactions in a second, the bucket lets us process them over the next few seconds without dropping any.
• Fairness across channels. Each channel (or user‑id) gets its own bucket key, so a hot topic can’t starve out quieter conversations.
• Operational simplicity. One Redis key per limiter, a tiny Lua script, and clear observability (we just HGET the token count to see how “full” the bucket is).

The best part? The same pattern can be reused elsewhere—rate‑limiting API calls, throttling background jobs, or even shielding a database from thundering‑herd reads.

Your Turn

If you’ve ever felt like you were trying to drink from a firehose, give the token bucket a whirl. Grab a Redis instance, drop the Lua script in, and watch your system level up from chaotic to heroic.

Challenge: Implement a per‑user token bucket for a “typing indicator” feature, where each user can send at most 5 indicators per second but can burst to 20 when they’re really excited. Share your results—or your funny failure stories—in the comments!

Happy coding, and may your notifications always arrive just in time! 🚀