"Design a notification system" looks like an integration task — call Twilio, call SendGrid, done. The interview lives in everything around those calls: how do you fan one event out to a million recipients across three channels, make sure a marketing blast never delays someone's login OTP, avoid double-sending when your queue redelivers a message, and survive Twilio having a bad afternoon? It's a queues-and-reliability problem wearing an integration costume.
This is the condensed walkthrough; the full guide (the pipeline, the four hard parts, delivery tracking, and the full production .NET 9 code) is on my site 👇
Full guide: https://prepstack.co.in/blog/design-a-notification-system-system-design
The design at a glance
| Concern | Decision |
|---|---|
| Core shape | Enqueue → per-channel workers → providers (decouple ingestion from delivery) |
| Fan-out | Expand recipients → one queued message per (user, channel) |
| Priority | Separate queues — transactional never waits behind bulk |
| Delivery guarantee | At-least-once + dedup (exactly-once across third parties is a myth) |
| Failures | Retry with backoff + jitter, then dead-letter queue |
| Politeness | User preferences, quiet hours, per-user rate limiting |
Why it's a queue problem, not an integration problem
Estimate it: 100M notifications/day ≈ ~1,160/sec average — trivial. But a campaign blast to 50M users in minutes is tens of thousands/sec. The spike is the design driver, and a queue in the middle absorbs it so you never hand that burst straight to a rate-limited provider.
Event sources (services, campaigns)
│
▼
[ Notification Service ] — validate · apply prefs · dedup · render template
│
▼
[ Message Queue ] (priority: transactional vs bulk)
│ │ │
▼ ▼ ▼
[Push worker] [SMS worker] [Email worker] — scale independently
│ │ │
▼ ▼ ▼
APNs / FCM Twilio SES — third-party providers
│ │ │
└──── delivery receipts (webhooks) ────▶ update status · DLQ on repeated failure
The four hard parts
1. Fan-out. One event can target millions. Expand the recipient list and enqueue one message per (user, channel) — in batches for huge audiences, so one request doesn't block producing millions of messages. Fan-out is where a tiny API call becomes a tidal wave; the queue is the seawall.
2. Priority — the OTP must not wait. A marketing blast enqueues tens of millions of messages. If a login OTP lands behind them in the same FIFO queue, it's useless by the time it arrives. Separate queues by priority — a high-priority transactional queue with its own workers, and a bulk queue for campaigns. Never let them share a lane.
3. Deduplication. Queues are at-least-once: a worker crash after sending but before acking causes redelivery. Guard every send with an idempotency key (eventId + userId + channel) checked against Redis before dispatch. If it's already marked sent, skip.
4. Retries + dead-letter queue. Providers fail (timeouts, 500s, throttling). Retry with exponential backoff + jitter up to a max, then send to a dead-letter queue for inspection rather than losing it or retrying forever. Wrap each provider in a circuit breaker, and ideally fail over to a backup provider.
"Sent" ≠ "delivered" ≠ "read"
Handing a message to Twilio isn't the phone buzzing. Consume delivery-receipt webhooks to move status from sent → delivered (or bounced/failed), and suppress future sends to a hard-bounced address. Apply user preferences and quiet hours before enqueueing, and cap frequency per user (batch low-priority notifications into a digest). This is both compliance (TCPA/GDPR/CAN-SPAM) and anti-fatigue.
I shipped this in production
We fan out ~15M events/day (budget.threshold.crossed, campaign.completed, conversion.tracked) to customer webhooks, email, and push. V1 dispatched inline in the request that produced the event — a deploy that recycled the app mid-flight dropped every in-memory delivery, and one slow customer endpoint dragged API latency up with it. Moving to outbox → Azure Service Bus → worker pool:
| Metric | Before | After |
|---|---|---|
| Events dropped per deploy | Thousands | 0 (outbox + queue) |
| First-attempt delivery success | n/a | ~96% |
| Eventual delivery success | Lost on first failure | ~99.98% (backoff + retry) |
| API write-path p95 | Coupled to slow endpoints | 120ms (decoupled) |
| Failing-endpoint handling | Retried inline, blocked workers | Auto-disabled after 20 consecutive failures |
The producer never blocks on a customer's endpoint — it only writes to the queue — and the worker owns the retry clock via scheduled messages, so a slow or dead endpoint can never back-pressure the API. A Redis SET-based dedup key makes at-least-once safe to retry; the breaker sheds load from any endpoint that has failed 20 times in a row. (Full .NET 9 producer + worker + circuit breaker is in the post.)
The model to carry forward
A notification system is a queue that buffers your events from unreliable third-party channels. Everything hard about it — fan-out, priority, dedup, retries — is solved by not calling providers inline. Three habits it teaches: put a queue in the middle (it answers bursts, outages, and retries at once); separate urgent from bulk (the OTP-behind-the-newsletter failure is the one interviewers probe for); assume redelivery, so dedup.
The full guide has the full pipeline, all four hard parts in depth, delivery tracking + preferences, the design checklist, the complete production .NET 9 outbox→Service Bus worker (dedup, backoff+jitter, circuit breaker, DLQ), and the "when it's overkill" honest section:
https://prepstack.co.in/blog/design-a-notification-system-system-design
Originally published on PrepStack.
Top comments (0)