Design a notification system: the system design interview walkthrough
The request to design a notification system sounds like one line — send a user a notification — and that one line hides the whole interview. Firing one message to one device is trivial; doing it across push, SMS and email for hundreds of millions of users, reliably, through third-party providers that fail and throttle, without spamming or double-sending anyone, is a distributed-systems problem with a dozen sharp edges. This is one of the most common system design interview prompts because it forces every reliability muscle at once: decoupling, fanout, retries, and deduplication.
The round turns on one framing you have to reach early: a notification system design is not a data-modelling problem, it is a reliable-fanout-through-unreliable-dependencies problem. The moment you accept that a push notification system must reach iOS and Android, SMS, and email through providers you do not control — Apple's APNs, Google's FCM, an SMS gateway, an email service — the design becomes queues, workers, and idempotency, not tables. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.
How the notification system round is actually run
A notification system round runs about 60 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite because the one-line problem opens onto deep water fast: the moment you draw "service sends a notification," the question becomes what happens when the provider is down, and every answer after that (retries, duplicates, bursts) compounds. Get the reliability spine clean and you have shown the whole loop in one problem.
The interviewer rarely lets "the service sends it" stand. "The APNs gateway starts timing out — what happens to those notifications?" That question — delivery when a third party you do not control fails — is the one a copied answer can never rehearse, and the one this round is really testing.
The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.
The five-step framework for any HLD round
Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:
- Clarify requirements — functional and non-functional, and what you are explicitly not building.
- Estimate — events per day, fanout per event, and per-provider throughput, to justify every later choice.
- High-level design — the boxes and arrows: producers, notification service, queue, channel workers, store.
- Deep dive — the two or three decisions that actually decide this system.
- Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.
What the debrief measures
The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you reach a queue-based design that absorbs bursts and survives provider failure), communication (did you say why a queue over synchronous sending in a sentence), technical execution (idempotency keys, retry with backoff, a dead-letter queue, per-channel rate limiting), and time management (did you spend your 60 minutes on reliability and fanout, not on the preferences schema). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.
Step 1 — Requirements: what a notification system must do
Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.
| Type | Requirement |
|---|---|
| Functional | Accept a notification request from many internal services; deliver across multiple channels (iOS and Android push, SMS, email); respect user preferences and opt-out; support reusable templates. |
| Functional (optional) | Scheduling (send later), analytics and delivery tracking, priority tiers (urgent / normal / low) — call these out, then scope most of them for later. |
| Non-functional | Reliable at-least-once delivery; no duplicates reaching the user; high throughput; the pipeline is decoupled from senders (a producer never blocks on a provider); gracefully handle third-party failure and rate limits. |
The sentence that earns points: "Delivery runs through third-party providers we do not control, so the core of the design is a queue that decouples senders from providers plus the idempotency and retry logic that makes at-least-once safe — the data model is almost the easy part." Say that and you have framed the entire design before drawing a box.
Step 2 — Capacity estimation (back of the envelope)
Estimation here is about proving one thing: the hard axis is reliable fanout through unreliable third parties, not storage.
| Quantity | Assumption | Result |
|---|---|---|
| Producing services | dozens of internal services | many event sources, one entry point |
| Notification events | tens of millions to billions/day | ~10K–100K+ events/sec at peak |
| Channels per event | up to 3 (push + SMS + email) | fanout multiplies the send volume |
| Third-party providers | APNs, FCM, SMS gateway, email | each has its own rate limit and fails intermittently |
Two conclusions fall out, and you should say both out loud. First, the providers are the bottleneck — you cannot send faster than APNs or an SMS gateway will accept, and any one of them can start failing, so the design is dominated by a buffer (a queue) and the retry logic around it, not by the message store. Second, each event fans out to several channels, so you want per-channel worker pools that scale and rate-limit independently. You have now pointed the interview at the real deep dive — decoupling with a queue — before anyone asked.
Step 3 — API and data model
The API is small: internal services call one endpoint; the heavy lifting is asynchronous behind it.
| Channel | Operations |
|---|---|
| Internal HTTP (request/response) |
POST /notifications with {user_id, type, template_id, payload, channels, idempotency_key} → returns 202 Accepted + notification_id, status queued. |
| Admin HTTP | Set user preferences / opt-out, create and edit templates, query delivery status by notification_id. |
The data model is thin. Preferences decide whether and where to send; the notification log records every status transition so a retry or a duplicate check has a source of truth:
| Entity | Notes |
|---|---|
notification |
notification_id, user_id, type, template_id, idempotency_key, status (queued / sent / delivered / failed), created_at. |
preference |
user_id → per-channel opt-in/opt-out, quiet hours, preferred channel. |
template |
template_id, channel, body with placeholders filled from the request payload. |
Step 4 — High-level design: the queue-based architecture
Now the boxes. Internal services publish to one notification service, which validates, checks preferences, renders the template, and enqueues the work — then per-channel workers deliver it. Around that sit the pieces that make reliability work:
- Producer services: any internal service (payments, shipping, security) that needs to notify a user, calling one API.
- Notification service: the single entry point — validates the request, checks the preference service, renders the template, and publishes one message per target channel to the queue. It never talks to a provider directly.
- Message queue (Kafka / SQS): one topic per channel; it decouples senders from delivery and absorbs bursts so a producer never blocks on a slow provider.
- Channel workers + adapters: per-channel worker pools (push, SMS, email) that consume their topic and call a provider-specific adapter (APNs, FCM, SMS gateway, SES / SendGrid).
- Preference + template service and a notification store / log: decide whether to send and record every status; a dead-letter queue holds messages that exhaust their retries.
Say explicitly that the notification service never calls a provider synchronously — it enqueues, and workers own the provider call. That single observation is what sets up the decoupling deep dive, and naming it unprompted is a signal the interviewer is listening for.
Deep dive: decoupling with a queue and workers
This is the first decision that actually decides the round. How does a producer send a notification without being coupled to a slow or failing provider?
| Approach | How it works | The catch |
|---|---|---|
| Synchronous send | The producer calls the notification service, which calls APNs / the SMS gateway and waits for the result inline. | Couples every producer to the slowest provider; one provider timing out stalls callers, and a burst of events has nowhere to wait — requests pile up and fail. |
| Queue + workers | The notification service validates and enqueues; per-channel worker pools consume and call providers asynchronously. | The right fit — producers return in milliseconds, the queue absorbs bursts, and each channel scales and retries on its own; cost is eventual (not instant) delivery and queue infrastructure. |
Pick queue + workers and say why in one line: the providers are slow and unreliable and traffic is bursty, so you put a durable buffer between the producer and the provider instead of making every caller wait on a dependency it cannot control.
Then name what the queue buys you, concretely:
- A producer's
POST /notificationsreturns202 Acceptedthe instant the message is enqueued — it never waits on APNs. - One topic per channel means the email backlog cannot stall push; each worker pool scales to its provider's throughput independently.
- A burst from one service lands in the queue and drains at the workers' pace, so a spike does not knock over the rest of the system.
"One service suddenly emits ten million events — does everything else stall?" No: the queue absorbs the burst, each producer still returns in milliseconds, and the channel workers drain at their own capacity while per-channel rate limiting keeps the providers inside their quotas. The queue is the shock absorber.
System Design
HLD · Messaging & realtime · 60 min
Design a notification system out loud, with follow-ups
Deep dive: multi-channel delivery via provider adapters
This is the decision most candidates under-think. "Send a notification" is really "send the right notification on the right channel through the right third party" — and each channel is a different integration.
One adapter per provider. Each channel worker pool wraps a provider behind a common adapter interface, so the worker logic (consume, render, send, record status, retry) stays identical while the provider specifics live in the adapter:
| Channel | Provider(s) | What the adapter handles |
|---|---|---|
| iOS push | APNs | device-token auth, payload size limits, token-invalid responses. |
| Android push | FCM | project credentials, topic vs device sends, throttling responses. |
| SMS | SMS gateway (e.g. Twilio) | character limits, country formatting, invalid-number errors. |
| SES / SendGrid | HTML vs plain text, bounces, suppression lists. |
The preference and template service decides channel — and whether to send at all. Before anything is enqueued, the notification service asks: has this user opted out of this channel? Is it within their quiet hours? Which channel do they prefer for this notification type? A security OTP may force SMS; a marketing nudge may be push-only and skipped entirely if the user opted out. Deciding not to send is as important as sending — spamming users is a failure mode the debrief notices.
Per-provider rate limiting and failover. Each provider has a quota, so each worker pool rate-limits itself to stay under it; if a provider is degraded, the worker can fail over to a secondary (a second SMS gateway, SES → SendGrid) or shed low-priority traffic first.
"A user has opted out of marketing email but this is a password reset — how does the system know to still send it?" Preferences are keyed by notification type and channel, not a global on/off; transactional types bypass marketing opt-out, and the preference service makes that call before the message is ever enqueued.
System Design
HLD · Messaging & realtime · 60 min
Defend multi-channel delivery out loud
Deep dive: reliability — idempotency, retries, and the dead-letter queue
Groups and channels are the easy half; this is where the round is won. At-least-once delivery means messages get retried — and a retry is exactly what can double-send a notification or lose one forever if you are careless.
Idempotency so a retry does not double-send. Every request carries an idempotency_key (or the system derives one from user_id + type + a dedup window). Before a worker sends, it checks a fast dedup store (Redis) for that key; if it is already marked sent, the worker drops the duplicate. This is what lets you retry freely — the same event replayed five times still produces one push, because the fifth attempt sees the key and stops.
Retry with exponential backoff on transient failure. When a provider returns a timeout or a 5xx, the worker does not give up and does not hammer the provider — it re-enqueues the message with an increasing delay (1s, 2s, 4s, …) up to a cap, with jitter so retries do not synchronise into a thundering herd. Permanent errors (invalid token, bad number) are not retried — they fail fast and are logged.
Dead-letter queue for poison messages. A message that exhausts its retries is moved to a dead-letter queue rather than dropped or retried forever. The DLQ is monitored and replayable: an on-call engineer can inspect why those messages failed (a provider outage, a malformed payload) and replay them once the cause is fixed. Nothing is silently lost.
The honest tradeoff to name: true exactly-once delivery across a third party you do not control is effectively impossible, so you choose at-least-once + idempotent dedup, which is cheaper and gives the user the same experience — one notification — without the coordination cost exactly-once would demand.
"A bug retries the same event five times — does the user get five pushes?" No: each send checks the idempotency key in the dedup store first, so the first attempt sends and the next four are dropped. At-least-once delivery plus idempotent dedup gives the user exactly one notification without paying for exactly-once.
The follow-ups that decide the notification system round
Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.
- "Why a queue instead of sending synchronously?" → Providers are slow and unreliable and traffic is bursty; a durable buffer stops producers blocking on a dependency they do not control and absorbs spikes. (communication)
- "The APNs gateway starts timing out — what happens?" → Transient failures retry with exponential backoff; messages that exhaust retries go to the dead-letter queue, monitored and replayable — never lost. (problem solving)
- "A bug retries the same event five times — five pushes?" → No: the idempotency key is checked in the dedup store before each send, so the user gets exactly one. (technical execution)
- "One service emits ten million events — does everything stall?" → The queue absorbs the burst; per-channel workers drain at capacity; per-provider rate limiting keeps each provider inside its quota. (problem solving)
- "A user opted out of marketing but this is a password reset — send it?" → Preferences are keyed by type and channel; transactional types bypass marketing opt-out, decided before enqueue. (technical execution)
Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.
How to prepare: a one-week plan
You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.
- Days 1–2: Learn the five-step framework until it is reflex. Do the notification system end to end on paper — requirements, estimate, API, data model, the queue-based boxes.
- Days 3–4: Drill the decoupling deep dive: synchronous send vs queue + workers, one topic per channel, how a burst drains. Say each tradeoff in one sentence.
- Days 5–6: Own multi-channel and reliability — provider adapters, preferences and opt-out, idempotency keys, exponential backoff, the dead-letter queue, at-least-once vs exactly-once. Answer each of the five follow-ups out loud.
- Day 7: A full timed 60-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.
This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the notification system is automatic, the same spine carries the rest of the list.
Frequently asked questions
How do I design a notification system step by step?
Follow the five-step HLD framework. Clarify requirements (accept requests from many services, deliver across push, SMS and email, respect preferences, support templates; reliable at-least-once delivery, no duplicates, high throughput); estimate to show the hard axis is reliable fanout through unreliable third parties, not storage; define one POST /notifications endpoint plus preference and template admin APIs, with a thin notification, preference and template model; draw the queue-based design (producers → notification service → message queue → per-channel workers with provider adapters → store, plus a dead-letter queue); then deep-dive the decisions that matter — decoupling with a queue, multi-channel delivery via adapters, and reliability through idempotency, retries and the DLQ.
Why use a message queue in a notification system instead of sending directly?
Because delivery runs through third-party providers that are slow, rate-limited, and fail intermittently, and traffic is bursty. If the notification service called providers synchronously, every producer would block on the slowest provider and a spike would have nowhere to wait. A message queue is a durable buffer: producers enqueue and return in milliseconds, the queue absorbs bursts, and per-channel worker pools drain at each provider's capacity. One topic per channel means an email backlog cannot stall push.
How does a notification system avoid sending duplicate notifications?
With idempotency. Every request carries an idempotency key (or one derived from user, type and a dedup window), and before a worker sends it checks a fast dedup store such as Redis for that key. If the key is already marked sent, the duplicate is dropped. This is what makes at-least-once delivery safe: the same event replayed five times still produces exactly one notification, because every attempt after the first sees the key and stops.
How do you handle a third-party provider failing in a notification system?
You distinguish transient from permanent failures. A timeout or 5xx is transient: the worker re-enqueues the message with exponential backoff and jitter, up to a retry cap, so it does not hammer a degraded provider. A permanent error like an invalid device token or bad phone number fails fast and is logged, not retried. Messages that exhaust their retries move to a dead-letter queue that is monitored and replayable, so an outage never silently drops notifications.
What is the difference between at-least-once and exactly-once notification delivery?
At-least-once means a message may be delivered more than once, which is why retries are safe but duplicates are possible. Exactly-once means each message is delivered precisely one time, which is effectively impossible across a third party you do not control without heavy coordination. In practice you choose at-least-once plus idempotent deduplication: retries guarantee the notification is not lost, and the idempotency key guarantees the user still sees it only once — the same user experience as exactly-once at a fraction of the cost.
How do I practise a system design interview out loud?
Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.
Practise it on Zynter
You can read every hop. The round is won on the retries and the no-duplicate case.
Pick the Notification Service round, draw the queue-based design on the canvas, and defend it out loud to a voice interviewer that keeps asking "the provider is timing out — now what?" and "a bug retried it five times — five pushes?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).
Practise the Notification Service round →
Browse all interview tracks
Top comments (1)
The tricky retry is the one where the provider accepts the send, then the worker crashes before marking the key sent in Redis. The next worker still sees an unsent key. An internal dedup check alone doesn't close that gap; I'd ask about provider-side idempotency and the policy when the outcome is unknown.
I recently saw the notification follow-up here in Amazon SWE guide walks through that exact failure trace as an original practice exercise.