May the Notifications Be With You: Building a Real-Time Notification System at Scale

The Quest Begins (The "Why")

I still remember the day our notification service started getting slammed by a sudden surge of user activity. Picture this: a popular livestream went viral, and within minutes we were trying to push hundreds of thousands of push notifications to mobile devices. Our API gateway happily accepted the requests, but the downstream push providers (FCM, APNs, you name it) started returning 429 Too Many Requests errors left and right. Users saw delayed or missing alerts, and our monitoring lit up like a Christmas tree.

We had built a “fire‑and‑forget” pipeline: each notification request hit a stateless worker that immediately fired off a push. No buffering, no throttling—just raw firepower. It worked fine under normal load, but as soon as traffic spiked we were basically shouting into a megaphone while the listener had their ears covered.

The problem wasn’t that we lacked workers; it was that we had no global guardrail to smooth out those bursts before they hit the third‑party APIs. We needed something that could sit in front of our workers, understand the shape of incoming traffic, and let through only what the providers could actually handle.

The Revelation (The Insight)

After a few sleepless nights and a lot of coffee, the insight hit me like a power‑up in a classic arcade game: a distributed token bucket rate limiter.

Why a token bucket?

• It allows short bursts (think of a bucket that can hold a few extra tokens) while enforcing a long‑term average rate.
• It’s simple to reason about: each incoming request tries to consume a token; if none are available, we either delay or drop the request.
• When backed by a fast, atomic store like Redis, the limiter works identically across all service instances, eliminating the “each node has its own counter” problem we saw with our first attempt.

The alternative—a fixed‑window counter—would either let a huge burst through at the start of each window (if the window is large) or cause needless throttling at the window edges (if the window is small). The leaky bucket smooths traffic too aggressively, often delaying requests that could have been sent immediately. The token bucket gives us the best of both worlds: burst‑friendly yet globally rate‑limited.

Here’s how the pieces fit together:

+----------------+     +------------------+     +-------------------+
|   API Gateway  | --> | Notification    | --> | Rate Limiter (Redis)|
|   (HTTP/gRPC)  |     | Service (worker) |     |  (Token Bucket)   |
+----------------+     +------------------+     +-------------------+
                                 |
                                 v
                         +-----------------+
                         | Delivery Workers|
                         | (FCM, APNs, …)  |
                         +-----------------+

The gateway hands off the request to our notification service. Before the worker actually sends a push, it asks the Redis‑backed token bucket: “Do I have a token?” If yes, we proceed; if not, we either queue the request for later retry or return a 429 to the caller (so the client can back‑off).

Wielding the Power (Code & Examples)

The Struggle: Naïve In‑Memory Counter

Our first attempt was embarrassingly simple: each Node.js worker kept an in‑memory counter that it incremented on every request and reset every second with setInterval.

// ❌ BEFORE – per‑instance counter (bad!)
let count = 0;
const LIMIT = 1000; // tokens per second

function allowRequest() {
  if (count >= LIMIT) return false;
  count++;
  return true;
}

Why this failed:

• If we ran 10 instances, each thought it could send 1000 req/s → 10 000 req/s total, far above what the push providers allowed.
• The counter reset drift between instances caused windows to misalign, leading to bursts that slipped through.
• No persistence: a restart meant losing the count and potentially spiking traffic right after a deploy.

The Victory: Redis‑Backed Token Bucket

We moved to a Lua script that atomically checks and updates the bucket. The script implements the classic token bucket algorithm:

• capacity – max tokens the bucket can hold (burst size).
• rate – tokens added per second (refill speed).
• On each request, we compute how many tokens should have been added since the last check, add them (capped at capacity), then try to consume one token. If we succeed, we return 1; otherwise 0.

-- redis_token_bucket.lua
-- KEYS[1] = bucket key (e.g., "rate_limit:notifications")
-- ARGV[1] = rate (tokens per second)
-- ARGV[2] = capacity (max tokens)
-- ARGV[3] = now (current unix time in seconds)

local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

local last = redis.call("HGET", KEYS[1], "last") or now
local tokens = tonumber(redis.call("HGET", KEYS[1], "tokens") or capacity)

-- refill
local delta = math.max(0, now - last)
tokens = math.min(capacity, tokens + delta * rate)

local allowed = 0
if tokens >= 1 then
  tokens = tokens - 1
  allowed = 1
end

redis.call("HMSET", KEYS[1], "tokens", tokens, "last", now)
redis.call("EXPIRE", KEYS[1], math.ceil(capacity/rate + 2)) -- auto‑clean

return allowed

And the thin Node.js wrapper that calls it:

// ✅ AFTER – Redis token bucket (works across all instances)
const redis = require('redis');
const client = redis.createClient({ url: process.env.REDIS_URL });
await client.connect();

const BUCKET_KEY = 'rate_limit:notifications';
const RATE = 800;      // tokens per second (≈ provider limit)
const CAPACITY = 2000; // allow bursts up to 2k

async function allowNotification() {
  const now = Math.floor(Date.now() / 1000);
  const allowed = await client.eval(
    // Lua script source (inlined for brevity)
    `
    local rate = tonumber(ARGV[1])
    local capacity = tonumber(ARGV[2])
    local now = tonumber(ARGV[3])
    local last = redis.call('HGET', KEYS[1], 'last') or now
    local tokens = tonumber(redis.call('HGET', KEYS[1], 'tokens') or capacity)
    local delta = math.max(0, now - last)
    tokens = math.min(capacity, tokens + delta * rate)
    local allowed = 0
    if tokens >= 1 then
      tokens = tokens - 1
      allowed = 1
    end
    redis.call('HMSET', KEYS[1], 'tokens', tokens, 'last', now)
    redis.call('EXPIRE', KEYS[1], math.ceil(capacity/rate + 2))
    return allowed
    `,
    { keys: [BUCKET_KEY] },
    [RATE, CAPACITY, now]
  );
  return allowed === 1;
}

// Usage in our worker
async function handleNotification(payload) {
  if (await allowNotification()) {
    await sendPush(payload); // talk to FCM/APNs/etc.
  } else {
    // Optionally push to a retry queue or respond with 429
    throw new Error('Rate limited – try again later');
  }
}

Common traps to avoid:

1. Forgetting the TTL – If you don’t set an expiration on the hash, stale keys can linger forever after a deploy, causing inaccurate limits. The EXPIRE call in the script cleans them up automatically.
2. Using separate GET/SET calls – That introduces a race condition between reading tokens and writing the new value. The Lua script runs atomically, guaranteeing correctness.
3. Picking the wrong key granularity – A bucket per user or per app works fine, but a single global bucket can throttle a low‑traffic user because of a high‑traffic neighbor. Choose the key that matches the policy you want to enforce (we used a global bucket for the whole notification stream because our provider limit is aggregate).

Why This New Power Matters

Switching to the token bucket changed everything.

• Smooth traffic: The push providers now see a steady stream that respects their rate limits, dramatically reducing 429 responses.
• Cost savings: Fewer retries mean less compute and network waste, and we avoid paying for excess push attempts that get dropped.
• Observability: Because the limiter lives in Redis, we can HGETALL the bucket keys to see current token counts and refill rates in real time—great for debugging and capacity planning.
• Resilience: Deploying new workers no longer causes a traffic spike; each instance consults the same source of truth, so the system scales horizontally without re‑tuning limits.

In short, we turned a chaotic firehose into a regulated aqueduct. The system now handles the same viral livestream that once knocked us out, and it does so while keeping latency low and user experience snappy.

Your Turn

Grab a Redis instance (or even a local Docker copy) and try implementing a token bucket for something you already rate‑limit—maybe an API endpoint, a webhook dispatcher, or a background job queue. Play with the rate and capacity values, watch the token count fluctuate with redis-cli, and see how bursty traffic behaves.

If you hit a snag, drop a comment or ping me—I love hearing about the creative ways folks adapt this pattern. Happy rate‑limiting, and may your notifications always be delivered on time! 🚀