Imagine you're in a system design interview, and a senior engineer asks you to design a notification system that can deliver millions of notifications.
You open Excalidraw and start drawing. An API server, Kafka, Redis, a bunch of workers, and a database. You've seen similar architectures before, so you know roughly where everything goes.
Then the senior engineer asks:
"How many of these notifications need to be delivered immediately, and how many can wait?"
You pause.
Because the answer could change how you design the entire system.
And that's just one of the questions you haven't asked yet.
The hardest part of system design isn't drawing the architecture. It's figuring out what architecture the problem actually needs.
So, before we draw another box, let's start with the questions that should come first.
One notification system, several different problems
Let's start with something familiar.
Imagine you're building a social application. Someone likes a post, another user sends a message, and someone else logs in from a new device.
The application wants to notify the relevant users about these events.
At first, this sounds like a straightforward feature: receive an event, find the recipient, and send a notification.
But consider what happens if we treat all three events the same way.
A message notification might need to arrive within seconds. A like notification can probably tolerate a short delay. A security alert about an unfamiliar login might need to be delivered as quickly as possible.
And a weekly email digest? That can be processed in batches whenever the system has capacity.
We have one product feature, but several different expectations.
This is where the design starts.
Not with a database choice. Not with Kafka. Not with a diagram copied from another system.
With the behavior the system is expected to provide.
Let's say we ask the product team about these use cases and establish the following initial requirements:
These are illustrative assumptions, not universal requirements. A real system might have different latency targets, channels, or priorities.
But writing them down gives us something useful: a description of the problem that we can challenge and refine.
For example, what does "near real-time" actually mean? Two seconds? Five seconds? And does the target refer to accepting a notification, beginning delivery, or the recipient actually receiving it?
Those distinctions will matter later.
For now, we know that our system must support different notification types, multiple channels, and different latency expectations.
That's already more useful than simply saying "millions of notifications."
What does "millions" actually tell us?
Let's say the requirement is ten million notifications per day.
Is that enough to decide how many servers we need?
Not yet. But we can start with a rough estimate.
Our average arrival rate is approximately 116 notifications per second.
That calculation is useful, but it doesn't tell us how the traffic is distributed.
Suppose most of those notifications arrive during a two-hour window. Or a scheduled campaign suddenly creates a large burst of work. Or an external provider goes down, and the system needs to process a backlog after it recovers.
In each case, the system might need to handle much more work at particular moments than the daily average suggests.
Let's record what we know and what we don't:
Notice that we haven't invented a peak number just to make the design look complete.
We have an estimate, an assumption about delivery behavior, and an unanswered question about peak traffic.
We can ask for more information if it's available. Otherwise, we can state an assumption, proceed with the design, and revisit it if the assumption turns out to matter.
This is an important habit: distinguish facts from assumptions instead of silently treating both as requirements.
Now let's see what happens when we try to build the simplest possible system.
Version 1: Just send the notification
Our first design could be very simple.
The application calls our Notification API. The API validates the request and calls the appropriate external provider.
An external provider is simply another service that performs the actual delivery. For email, that might be an email delivery service; for push notifications, it might be a platform's push notification service.
Our application doesn't need to implement the entire delivery infrastructure itself.
For a small application with modest traffic, this design might be perfectly reasonable.
But let's introduce a constraint.
Suppose the email provider takes several seconds to respond. Our Notification API is waiting for that response, which means the application calling our API may also have to wait.
Now imagine the provider becomes temporarily unavailable.
Requests can pile up, connections remain occupied, and a problem with an external dependency can affect the application itself.
The issue isn't that direct delivery is always wrong. It's that direct delivery couples the application's request to the provider's response time and availability.
Can we separate those two things?
Version 2: Accept the work now, deliver it later
Imagine going to a restaurant during a busy evening.
The staff takes your order, records it, and gives it to the kitchen. You don't need to stand in the kitchen and wait for the food to be prepared before the restaurant acknowledges your order.
The order has been accepted. Preparation happens separately.
A notification system can use a similar idea.
Instead of sending the notification directly from the API, we can place the work in a queue.
A queue is a structure that holds work until a consumer is ready to process it. A worker can take a queued notification and attempt delivery independently of the original API request.
A durable queue goes a step further: it retains accepted messages in persistent storage so that a process restart or machine failure doesn't simply erase pending work. The exact guarantees depend on the queue technology and its configuration.
Our architecture now looks like this:
The Notification API validates the request and places a message in the queue. A delivery worker reads the message and contacts the provider.
The API no longer needs to wait for the provider to finish sending the notification.
We've separated two operations:
Accepting responsibility for processing the notification.
Attempting to deliver it to the recipient.
This is asynchronous processing. The system accepts work now and completes it separately, rather than making the caller wait for the entire operation.
But this introduces an important question: what exactly does the API promise when it returns success?
If the API returns success after the notification has been durably accepted, it can tell the caller that the system has accepted the work.
It cannot honestly claim that the recipient has received the notification.
Those are different guarantees.
Even this simplified flow forces us to define what each state means.
For example, a provider accepting an email doesn't necessarily mean the email reached the recipient's inbox. If the product needs that distinction, our state model must represent it.
And notice how we arrived at the queue. We didn't add it because a large system is supposed to have one. We added it because we wanted to isolate API request handling from slow or unavailable delivery providers.
That's a design decision we can explain.
Version 3: A queue doesn't magically solve every problem
Our queue protects the API from waiting for the provider. It can also hold pending work when delivery slows down.
But what happens when the system receives notifications faster than its workers can process them?
The queue grows.
A queue is useful for absorbing bursts, but it doesn't create unlimited processing capacity. If incoming work continuously exceeds processing capacity, the backlog will keep increasing.
We need to understand both sides of the system.
If the arrival rate temporarily exceeds the processing rate, the queue can absorb the difference.
If the arrival rate stays higher for long enough, the system eventually needs more capacity, less incoming work, or a different processing strategy.
We should monitor the queue's size and age, not just whether the API is returning successful responses.
Why the age of queued messages?
Because a queue containing a million messages isn't necessarily unhealthy if they are allowed to wait for hours. A much smaller queue might be a serious problem if security alerts are sitting in it for thirty seconds.
The important measurement depends on the requirement.
For our system, useful measurements might include:
How many notifications are waiting.
How long the oldest notification has been waiting.
How long delivery attempts take.
How many attempts fail.
Whether high-priority notifications meet their latency targets.
This is where requirements become operational measurements.
"Security alerts should arrive quickly" isn't enough on its own. We need a measurable target to determine whether the system is meeting that expectation.
Version 4: What if an important notification is stuck behind millions of emails?
Let's return to the notification types we identified earlier.
Imagine a marketing campaign schedules ten million emails. At the same time, a user triggers a security alert.
If both kinds of work share the same queue and processing capacity, the security alert could wait behind bulk traffic.
The queue is working exactly as designed. The problem is that the design doesn't necessarily protect our high-priority workload.
We now have a few options.
We could let the workers prioritize messages within a shared queue. We could create separate queues for different classes of work. Or we could reserve some worker capacity for high-priority notifications.
Each option has trade-offs.
A shared queue can be simpler to operate, but priority scheduling depends on the capabilities of the queue and consumer design.
Separate queues offer more isolation, but require additional configuration and operational attention. They also don't automatically guarantee fast delivery: if the high-priority workers are overloaded or the provider is slow, latency can still increase.
Suppose we decide that the isolation is worth the added complexity.
Our design could evolve into this:
This is one possible design, not a rule that every notification system needs two queues.
The decision depends on the latency requirements, workload, queue technology, and cost of operating separate processing paths.
More importantly, we've now seen the design change for a reason: our original requirements told us that not all notifications could wait equally long.
That is how architecture should evolve.
Now let's talk about failure
Suppose a worker sends a notification to an external provider.
The provider receives the request, but its response never reaches the worker. The worker times out.
Did the provider send the notification?
We don't know.
The worker has an ambiguous outcome: the request may have failed before delivery, or the provider may have accepted it even though the response was lost.
If the worker retries, the recipient might receive the same notification twice. If the worker doesn't retry, a notification that was never accepted might never be sent.
This is why "just retry on failure" isn't a complete reliability strategy.
At a high level, we need to think about:
Which failures are temporary and worth retrying.
How many attempts are allowed.
How long to wait between attempts.
What happens when all attempts fail.
How we track delivery attempts and outcomes.
Whether the provider supports idempotency mechanisms that can reduce duplicate effects.
A retry policy defines when and how failed work is attempted again. Some systems also move repeatedly failing messages into a dead-letter queue, a separate place for messages that need further investigation or special handling.
We don't need to implement all of that in this first design exercise. The point is to recognize that a new requirement, recovering from temporary failures introduces new responsibilities.
And even a durable queue doesn't solve all of them. It can preserve pending work, but it cannot guarantee that an external provider delivers a notification exactly once.
We'll explore retries, idempotency, and reliable state changes in more depth later in the series.
For now, let's capture the design decisions we've made and the reasons behind them.
This is much more useful than a list of technologies.
It tells us what we chose, why we chose it, and what information could still change the design.
So, do we need Kafka and Redis?
We haven't actually established that yet.
We've established that we need a way to accept notification requests, process them asynchronously, and preserve pending work according to our reliability requirements.
A durable queue is one possible component that provides those capabilities.
Kafka is a distributed event-streaming platform that can retain records and allow consumers to process them independently. It may be a good fit when we need high throughput, replay, or multiple independent consumers, but it also brings operational and design considerations.
A managed queue service might satisfy our requirements with less operational effort.
Redis can be useful for caching, rate limiting, and certain queue-like workloads, depending on the data structures and persistence guarantees involved. But we shouldn't automatically assume that a Redis-backed queue provides the same durability and delivery guarantees as every dedicated queue or event-streaming system.
We would compare the actual guarantees, workload, operational complexity, and cost before choosing.
The same applies to our database. We might need to persist notification state, user preferences, and delivery attempts. But the appropriate data model depends on which queries we need to support, how much data we retain, and what consistency guarantees the product requires.
The technology should follow from those decisions.
If we choose Kafka simply because the prompt says "millions," we've skipped the most important part of the exercise.
The architecture is still a hypothesis
Let's step back and look at what we have:
This isn't a complete production architecture. It doesn't yet specify the schema, exact API contract, worker concurrency, retention policy, provider failover strategy, or every failure-handling detail.
And that's okay.
We now have a baseline design that follows from the requirements we've discussed. We can identify the remaining questions and investigate the ones that could materially change our choices.
For example, if users must query the status of every notification, we'll need to define how delivery state is stored and retrieved.
If the system must meet strict latency targets, we'll need to calculate peak capacity and examine how queueing affects end-to-end latency.
If notifications must respect changing user preferences, we'll need to decide when those preferences are checked and what happens to work already waiting in a queue.
Each new constraint can lead to another revision.
The goal isn't to draw the most complicated architecture possible. It's to reach a design that satisfies the known requirements, then understand where it might fail as those requirements change.
The habit worth building
When you face an unfamiliar system, don't start by asking which database, queue, or cache you should use.
Start by understanding what the system must do.
Find out who uses it, what operations it supports, what the important latency and reliability expectations are, and how much work it needs to handle.
Write down the requirements and assumptions so you can distinguish what is known from what is merely plausible.
Then build the simplest design that satisfies them.
When a constraint exposes a limitation, evolve the design. Explain why the change is necessary, what it improves, and what new complexity it introduces.
You don't need to answer every possible question before drawing anything. You do need to know which unanswered questions could invalidate your current design.
And don't treat a technology name as an explanation. Saying "we'll use Kafka" doesn't explain how the system handles a provider outage, how long work is retained, or what the API promises when it returns success.
Those are the decisions that give an architecture its meaning.
A good system design is not a collection of familiar boxes. It's a set of decisions that you can trace back to requirements, constraints, and explicit trade-offs.
Once you learn to think that way, the diagram becomes a way to communicate your reasoning not a substitute for it.
In the next article, we'll explore what happens after a system accepts a request: how to design APIs, model data, and represent state transitions so that the system's behavior remains predictable as it grows.
Top comments (1)
Bro I like that u published here as well, besides X. and really looking forward to next article as well.