Building a Real-Time Notification System at Scale: Like Neo Dodging Bullets

The Quest Begins (The “Why”)

I still remember the night our notification service went down because a single power‑user started spamming alerts. The logs looked like a firehose, the CPU spiked, and our users began complaining that they missed important messages. We were trying to build a real‑time push pipeline—think Slack‑style notifications—but we hadn’t put any brakes on the incoming traffic. The problem wasn’t the delivery mechanism; it was the sheer volume of requests hammering our backend.

We needed a way to say, “Hey, you can only send X notifications per second,” without turning the system into a bottleneck that added latency for everyone else. After a few frantic Slack threads and a lot of coffee, the quest became clear: design a rate limiter that could scale horizontally, stay accurate under bursty traffic, and keep the notification flow smooth.

The Revelation (The Insight)

The breakthrough came when I stopped thinking about limiting requests and started thinking about limiting tokens that represent permission to send a notification. Imagine each user (or API key) owning a bucket that slowly refills with tokens—say, 10 tokens per second. Every time they want to push a notification, they try to consume a token. If the bucket is empty, the request is rejected or queued.

Why does this work better than a simple fixed‑window counter?

• Burst‑friendly: A user can save up tokens during idle moments and spend them in a short burst, which matches real‑world usage patterns.
• Smooth decay: Tokens refill continuously, so the limit isn’t reset at arbitrary clock boundaries that can cause thundering herd effects.
• Distributable: The bucket state can live in a fast, atomic store like Redis, allowing any number of API gateways to consult the same source of truth without locking up a single node.

The critical insight was to implement the token bucket with a Lua script inside Redis. Lua runs atomically, so the check‑and‑consume operation is indivisible—no race conditions even when dozens of instances hit the same key simultaneously.

Here’s a quick ASCII view of the flow:

+----------+      +-------------------+      +----------------------+
|  Client  | ---> | API Gateway (N)   | ---> | Notification Service |
+----------+      +-------------------+      +----------------------+
                                   |
                                   v
                          +-----------------+
                          |  Rate Limiter   |
                          |  (Redis + Lua)  |
                          +-----------------+
                                   |
                                   v
                         +-----------------+
                         |  Token Buckets  |
                         |  (per‑user key) |
                         +-----------------+

Wielding the Power (Code & Examples)

The Struggle: Naïve In‑Memory Counter

Our first attempt was a straightforward per‑user counter stored in a Node Map that reset every second with setInterval. It looked innocent:

// naive-rate-limiter.js (do NOT use in prod!)
const limits = new Map(); // userId => { count, windowStart }

function allowRequest(userId) {
  const now = Date.now();
  const entry = limits.get(userId) || { count: 0, windowStart: now };

  // reset window if a second has passed
  if (now - entry.windowStart >= 1000) {
    entry.count = 0;
    entry.windowStart = now;
  }

  if (entry.count >= 10) return false; // limit: 10 req/sec
  entry.count++;
  limits.set(userId, entry);
  return true;
}

What went wrong?

• The Map lives in a single process. When we scaled to multiple API gateway instances, each had its own view of the count, so the real limit became N × 10 where N is the number of instances.
• The setInterval reset drifted under load, causing windows to drift and either over‑limit or under‑limit users unpredictably.
• No persistence—restarting the service cleared all counters, letting a burst slip through after a deploy.

The Victory: Distributed Token Bucket with Redis

Now let’s see the battle‑tested version. We’ll store a key like ratelimit:{userId} that holds two fields: tokens (float) and updatedAt (timestamp). The Lua script does three things atomically:

1. Refill tokens based on elapsed time.
2. Clamp tokens to the maximum bucket size.
3. If at least one token is available, consume one and return allow = 1; otherwise return allow = 0.

-- ratelimit.lua
local key = KEYS[1]                -- e.g., "ratelimit:user-42"
local now = tonumber(ARGV[1])      -- current unix time in ms
local rate = tonumber(ARGV[2])     -- tokens per second (refill rate)
local capacity = tonumber(ARGV[3]) -- max tokens in bucket

local entry = redis.call('HMGET', key, 'tokens', 'updatedAt')
local tokens = tonumber(entry[1]) or capacity
local updatedAt = tonumber(entry[2]) or now

-- time elapsed in seconds
local delta = (now - updatedAt) / 1000.0
tokens = math.min(capacity, tokens + delta * rate)

local allowed = 0
if tokens >= 1 then
    tokens = tokens - 1
    allowed = 1
end

redis.call('HMSET', key, 'tokens', tokens, 'updatedAt', now)
redis.call('EXPIRE', key, math.ceil(capacity/rate)+2) -- auto‑clean
return allowed

And the thin Node wrapper that calls it:

// rate-limiter.js
const Redis = require('ioredis');
const redis = new Redis({ host: 'redis-service', port: 6379 });
const luaScript = `
  -- (paste the Lua script above)
`;

const REFILL_RATE = 10;   // 10 tokens per second
const BUCKET_SIZE = 10;   // allow bursts up to 10

async function allowRequest(userId) {
  const now = Date.now();
  const result = await redis.eval(luaScript, 1,
    `ratelimit:${userId}`, now, REFILL_RATE, BUCKET_SIZE);
  return result === 1; // true if allowed
}

// Usage in an Express route
app.post('/notify', async (req, res) => {
  const { userId, payload } = req.body;
  if (!(await allowRequest(userId))) {
    return res.status(429).json({ error: 'Rate limit exceeded' });
  }
  // proceed to push notification...
  await notificationService.send(userId, payload);
  res.json({ status: 'queued' });
});

Why This Beats the Alternatives

Approach	Pros	Cons
Per‑instance in‑memory counter	Zero external latency, simple	Incorrect under scale, no persistence, window drift
Fixed‑window counter in Redis	Simple, works across instances	Allows bursts up to 2× limit at window edges, can cause thundering herd
Token bucket (our solution)	Smooth refill, burst‑friendly, accurate, atomic via Lua	Slightly more code, requires Redis (but we already use it for pub/sub)

The Lua script guarantees that the check‑and‑consume step is indivisible, eliminating the race condition that plagued our earlier attempts. Because the bucket lives in Redis, any number of gateway nodes share the same state, so the limit truly is 10 requests per second per user, no matter how many instances we spin up.

Why This New Power Matters

With the token bucket in place, our notification system can now absorb traffic spikes without dropping messages or overwhelming downstream workers. Users get a predictable experience: they can burst when they need to (think a sudden surge of activity in a chat room) but are gently throttled when they abuse the channel.

Operational wins, too:

• Observability – the ratelimit:* keys expose real‑time usage; we can graph token levels to spot hot users.
• Graceful degradation – if Redis hiccups, the EXPIRE ensures stale keys vanish, and we can fall back to a local counter as a last‑ditch safety net.
• Cost efficiency – we no longer need to over‑provision compute just to survive traffic spikes; the limiter does the heavy lifting at the edge.

In short, we turned a chaotic firehose into a controllable stream, letting our service stay responsive and reliable.

Your Turn

Try adding a token‑bucket rate limiter to one of your own projects—maybe a webhook endpoint or an internal API. Start with a modest rate (e.g., 5 req/s) and watch how it smooths out bursts. If you hit a snag, remember: the Lua script is your friend; test it with redis-cli --eval before wiring it into code.

What’s the first rate‑limited feature you’ll build? Drop a comment or tweet your experiments—I’d love to hear how you tame the traffic dragon!

---

Happy coding, and may your limits always be just right.