Building a Real-Time Notification System at Scale: Lessons from The Matrix

The Quest Begins (The "Why")

Picture this: it’s 2 a.m., our notification service is buzzing like a beehive, and the pager is screaming because a flash‑sale just went live. Users are expecting instant push alerts, but our system is choking under a sudden surge of traffic. Every request hits a PostgreSQL table to check if a user has exceeded their per‑second limit, and the DB starts to queue up like a line at a coffee shop during rush hour. Latency spikes, some notifications drop, and the support tickets start piling up like unread emails after a weekend.

We’d tried a few quick fixes—capping the number of workers, adding more DB read replicas—but none of them tackled the root cause: we were making a stateful check on every single request, and that check was the bottleneck. It felt like we were trying to stop a flood with a sponge.

That night, after way too much coffee, I realized we needed a rate limiter that could make decisions locally and fast, while still being globally consistent enough to prevent abuse. The goal wasn’t to block every request; it was to smooth out the traffic so our downstream services could keep up without breaking a sweat.

The Revelation (The Insight)

The breakthrough came when I stopped thinking of rate limiting as a “gate” and started seeing it as a token bucket that lives close to the caller.

Why token bucket?

• It allows bursts (perfect for notification spikes) while still enforcing an average rate.
• It can be implemented with a few atomic operations in Redis, meaning the decision is made in a single round‑trip.
• Unlike a fixed‑window counter, it doesn’t suffer from the “reset‑at‑edge” problem where a burst of traffic at the end of one window and the start of the next can double‑dip the limit.

The critical insight was simple: store only two numbers per user—remaining tokens and the timestamp of the last refill—and update them atomically with a Lua script. Because the script runs inside Redis, we get the consistency of a transaction without the overhead of a lock or a separate coordination service.

Here’s how it looks in practice: each incoming request asks Redis, “Give me one token if you have it; otherwise tell me to wait.” If a token is granted, we proceed to send the notification; if not, we either drop the request or return a gentle back‑off signal to the client.

Trade‑offs we weighed

Approach	Pros	Cons
Fixed window counter (DB)	Simple to reason about	Burst‑friendly? No. Edge‑case spikes can double the limit.
Leaky bucket (in‑process)	No external dependency	Hard to share state across instances; requires sticky sessions or complex sharding.
Token bucket (Redis Lua)	Handles bursts, O(1) decision, globally consistent	Requires Redis; adds a tiny network hop (still sub‑millisecond if Redis is close).

For a notification system that needs to be both responsive and fair, the Redis‑backed token bucket was the clear winner.

Wielding the Power (Code & Examples)

The Struggle – Naïve per‑request DB check

# pseudocode – what we had before
def can_send_notification(user_id):
    row = db.fetch_one(
        "SELECT count FROM notification_limits WHERE user_id = %s FOR UPDATE",
        (user_id,)
    )
    if row['count'] >= MAX_PER_SEC:
        return False   # reject
    db.execute(
        "UPDATE notification_limits SET count = count + 1 WHERE user_id = %s",
        (user_id,)
    )
    return True

What went wrong?

• Every request incurred a lock‑heavy FOR UPDATE and a round‑trip to the DB.
• Under load, the DB queue grew, causing latency spikes and occasional deadlocks.
• The counter reset only via a cron job, leading to inaccurate counts if the job lagged.

The Victory – Token bucket with Redis Lua

First, we define the Lua script that does the refill and token grant atomically:

-- ratelimit.lua
local key = KEYS[1]               -- e.g., "notif:ratelimit:user_id"
local tokens = tonumber(ARGV[1])  -- max tokens (burst size)
local refill_rate = tonumber(ARGV[2])  -- tokens per second
local now = tonumber(ARGV[3])   -- current unix time in ms

local data = redis.call('HMGET', key, 'tokens', 'last')
local current_tokens = tonumber(data[1]) or tokens
local last_refill = tonumber(data[2]) or now

-- calculate how many tokens to add since last check
local delta = math.max(0, (now - last_refill) / 1000.0 * refill_rate)
current_tokens = math.min(tokens, current_tokens + delta)

if current_tokens >= 1 then
    current_tokens = current_tokens - 1
    redis.call('HMSET', key, 'tokens', current_tokens, 'last', now)
    redis.call('EXPIRE', key, 3600)  -- keep key alive for an hour
    return {1, current_tokens}       -- grant
else
    redis.call('HMSET', key, 'tokens', current_tokens, 'last', now)
    redis.call('EXPIRE', key, 3600)
    return {0, current_tokens}       -- deny
end

Now the client‑side call (Node.js example) becomes trivial:

const redis = require('ioredis');
const client = new redis({ host: 'redis-cache', port: 6379 });

const MAX_TOKENS = 10;          // allow bursts of 10 notifications
const REFILL_PER_SEC = 5;       // steady‑state rate of 5/sec

async function canNotify(userId) {
  const key = `notif:ratelimit:${userId}`;
  const now = Date.now();

  const result = await client.eval(
    // lua script content as a string
    fs.readFileSync('./ratelimit.lua', 'utf8'),
    1,               // number of KEYS
    key,             // KEYS[1]
    MAX_TOKENS,      // ARGV[1]
    REFILL_PER_SEC,  // ARGV[2]
    now              // ARGV[3]
  );

  const granted = result[0] === 1;
  if (!granted) {
    // optional: inform caller to retry after a short back‑off
    return { allowed: false, retryAfterMs: Math.ceil((1 - result[1]) / REFILL_PER_SEC * 1000) };
  }
  return { allowed: true };
}

Why this works better:

• The Lua script runs inside Redis, guaranteeing that the read‑modify‑write of tokens and timestamp is atomic—no race conditions.
• Only one network round‑trip (request → Redis) is needed, which is typically <1 ms when Redis sits in the same VPC.
• The bucket naturally smooths traffic: a sudden burst consumes saved tokens, then the refill rate brings the level back down, preventing thundering‑herd problems.

Common traps to avoid

1. Using separate GET/SET calls – splits the operation into two round‑trips, reintroducing the race condition we tried to solve.
2. Forgetting to set an expiration – keys would accumulate forever, wasting memory. A simple EXPIRE (or Redis’s built‑in LRU) keeps the dataset bounded.
3. Using wall‑clock time without monotonic fallback – if the system clock jumps backward, the refill math can go negative. Using Date.now() (milliseconds since epoch) is safe because it only moves forward; if you ever need to handle clock adjustments, switch to a monotonic source like process.hrtime.bigint() and convert to ms.

Why This New Power Matters

After we pushed the token‑bucket limiter to production, the pager finally fell silent.

• Latency: The 95th‑percentile notification delivery time dropped from ~350 ms to under 50 ms.
• Throughput: Our service now comfortably handles 150 k requests per second with the same instance count—before we were maxing out at ~40 k.
• Cost: We retired two extra DB read replicas, saving roughly $1,200 a month in managed‑service fees.
• Reliability: No more lost notifications during traffic spikes; users get their alerts exactly when they expect them.

The rate limiter didn’t just fix a symptom; it reshaped how we think about protecting downstream services. By moving the decision‑making close to the edge and keeping the state tiny and fast, we turned a fragile, DB‑centric choke point into a robust, scalable guardrail.

---

Your Turn

Grab a service that’s currently throttling users with a heavyweight check—maybe an API endpoint, a webhook dispatcher, or even a job‑queue worker. Sketch out a token‑bucket limiter for it, throw the Lua script into Redis, and watch the traffic smooth out.

Challenge: Implement the limiter, instrument it with a histogram of granted vs. denied requests, and share the before/after numbers in a comment. I’d love to hear how your quest went! 🚀