This is Part 10 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
In Part 9, we solved the problem of data that had grown too large for a single database. We split it across multiple shards, each holding its piece of the whole, so that no single machine ever had to carry everything.
At that point, the architecture could scale in almost every direction we'd tried to push it. Traffic was distributed across application servers. Repeated database work was absorbed by the cache. Read traffic was spread across replicas. Data itself was partitioned across shards.
And yet.
We ended Part 9 by noticing something that none of those solutions addressed. Some user requests trigger a lot of downstream work. Saving an order is one thing. But saving the order, sending a confirmation email, generating an invoice, updating inventory, firing off a notification, recording an analytics event, triggering the recommendation engine: that's an entirely different conversation.
Right now, all of that happens before the user gets a response.
The question we left with was this: what if they didn't have to wait for all of it?
--
Section 1: The User Doesn't Need Everything Right Now
Before we look at any solution, it's worth asking a simpler question.
When a user places an order, what do they actually need to know before they can move on?
They need to know the order was received. They need confirmation that the important thing happened: their money was accepted, their items are reserved, the transaction is real. That's it. That's what they're waiting for.
They do not need to wait for the confirmation email to land in their inbox. They do not need to wait for the invoice to be generated and stored somewhere. They do not need to wait for the analytics system to record that this purchase happened. They certainly don't need to wait for the recommendation engine to update its model based on what they just bought.
All of that will happen. It should happen. But none of it needs to happen before the user gets their confirmation screen.
This seems obvious when you say it directly. Of course the user doesn't need to wait for the analytics event. Of course they don't need to hold their breath while the recommendation system recalculates.
But the way most applications are initially built, that's exactly what happens.
--
Section 2: Doing Everything Synchronously
Here's what a typical order flow looks like when everything is wired together the simple, obvious way.
User clicks "Place Order"
|
v
Application receives request
|
v
Save order to database (50ms)
|
v
Send confirmation email (200ms)
|
v
Generate invoice (120ms)
|
v
Update inventory (80ms)
|
v
Record analytics event (90ms)
|
v
Trigger recommendations (150ms)
|
v
Return response to user
Total wait: ~690ms
The user clicked a button and waited almost 700 milliseconds for a confirmation screen. More than half a second, for a response to something they did in an instant.
And that's assuming nothing goes wrong. What if the email service is having a slow moment and takes two seconds instead of 200 milliseconds? The user waits two seconds. What if the analytics system is down entirely? The request fails, and the user gets an error, even though the order itself was saved perfectly.
The application has tied its response time and its reliability to every piece of downstream work it performs. Every slow step makes the user wait longer. Every failing step makes the whole request fail.
That's not a hardware problem. It's not a database problem. It's an architectural problem. The application is doing work in a sequence when most of that work doesn't actually depend on the steps before it.
The confirmation email doesn't need the invoice to be generated before it can be sent. The analytics event doesn't need the email to succeed before it can be recorded. These tasks are all independent of each other. They're only sequential because that's how they got wired together.
So the question becomes: what would it look like to stop treating them as sequential?
--
Section 3: Put the Work in a Queue
The insight is simple once you see it.
Not all work needs to happen now. Some work needs to happen eventually, but the user doesn't need to wait for it.
If that's true, then instead of doing all that work before responding, the application can do this:
- Save the order. That's the important part, and it must happen now.
- Make a note that a confirmation email needs to be sent.
- Make a note that an invoice needs to be generated.
- Make a note that an analytics event needs to be recorded.
- Return a response to the user. Then, separately, something else comes along, reads all those notes, and does the work.
User clicks "Place Order"
|
v
Application receives request
|
v
Save order to database (50ms)
|
v
Leave notes for background work
|
v
Return response to user
Total wait: ~60ms
The user gets their confirmation in 60 milliseconds instead of 690. Then, in the background, the email goes out, the invoice is generated, the analytics are recorded. The user never waits for any of it.
This is the idea behind a message queue. Instead of doing background work inline, the application puts a message into a queue. Each message is a piece of work that needs to happen: "send this email," "generate this invoice," "record this event." The queue holds onto those messages. Something else picks them up and does the actual work.
The application that creates messages is called the producer. It produces work to be done.
The queue is the place that holds work waiting to be processed. It doesn't do the work itself. It holds the work.
The thing that picks up messages and processes them is called a consumer or worker. It consumes messages from the queue and does the actual task.
BEFORE:
User --> Application --> [save] --> [email] --> [invoice] --> [analytics] --> Response
AFTER:
User --> Application --> [save] --> Queue --> Response
Queue
|
.------------+-------------.
| | |
[Worker] [Worker] [Worker]
email invoice analytics
The user's request touches the queue for a moment, hands off the work, and returns a response. The workers operate completely independently. They process messages at their own pace, without the user having to wait.
--
Section 4: Let Workers Do the Work Later
This separation between "the work that happens now" and "the work that happens later" is called asynchronous processing. The user's request doesn't wait for every piece of work to complete before finishing. It hands off what it can, responds quickly, and trusts that the rest will happen.
It's worth being clear about what changes for the user.
From their perspective, they clicked a button and got a confirmation almost instantly. The confirmation email arrives a second or two later, while they're already looking at the confirmation screen. The invoice appears in their account shortly after. They experience everything they expected to experience. They just didn't have to wait for it all at once.
And from the system's perspective, the work is now separated. The application server that handled the user's request is free to handle the next request. It's not blocked, waiting for an email to send. The workers that process the queue can be scaled independently, separate from the application servers. If there's a backlog of emails to send, you add more email workers. The application servers that handle user requests don't need to change at all.
It also makes the system more resilient. Before, if the email service went down, user requests would fail entirely, even though the order itself was being saved correctly. Now, if the email service goes down, the messages just accumulate in the queue. When the email service recovers, the workers process the backlog. Orders are never affected. Users get their emails a little late, but the system doesn't break.
--
Section 5: The Queue as a Buffer
There's a second, equally important reason queues exist that has nothing to do with user experience.
Imagine a flash sale. A limited-edition product goes on sale at noon, and 50,000 users try to place an order at exactly the same moment.
Without a queue, 50,000 requests arrive simultaneously. Every single one of them tries to do all the work at once. The email service receives 50,000 requests in the same second. The invoice system receives 50,000 requests in the same second. The analytics system receives 50,000 requests in the same second. Systems that were designed for a normal pace of traffic suddenly face a spike that's fifty times larger than usual.
Without a queue:
50,000 simultaneous orders
|
v
Email Service <-- overwhelmed
Invoice System <-- overwhelmed
Analytics <-- overwhelmed
Some of those systems buckle. Response times explode. Errors start appearing. The spike that should have been a celebration turns into an incident.
Now add a queue.
With a queue:
50,000 simultaneous orders
|
v
Queue <-- absorbs the spike instantly
Queue
|
v
Workers process at steady pace
Email: one at a time, as fast as they can
Invoice: one at a time, as fast as they can
Analytics: one at a time, as fast as they can
The 50,000 orders arrive in a second. They all get saved. They all drop their background work into the queue. Users all get their confirmation. Then the queue drains over the next few minutes as workers process messages at a sustainable rate.
The queue acts as a buffer between the rate at which work arrives and the rate at which it can be processed. Producers and consumers don't have to run at the same speed. The queue absorbs the difference.
This property is what makes message queues so valuable during traffic spikes. The burst of user activity doesn't translate directly into a burst of load on every downstream system. It translates into a larger queue, which then drains steadily. The downstream systems see a smooth, consistent workload regardless of how spiky the incoming traffic was.
--
Section 6: The Trade-off: Work Can Fail
It would be easy at this point to conclude that queues solve everything. They reduce user latency. They isolate failures. They smooth out traffic spikes. What could go wrong?
Quite a bit, actually. Moving work to the background introduces a new category of problems that synchronous systems don't have to worry about.
Workers can fail. A worker picks up a message and crashes halfway through processing it. The email was never sent. The invoice was never generated. If nothing else intervenes, the user never gets their email, and nobody knows.
Most queue systems handle this by keeping a message in the queue until a worker explicitly confirms it's done. If the worker crashes without confirming, the message goes back into the queue and another worker picks it up. That's a good default, but it creates the next problem.
Work can happen twice. If a worker processes a message, sends the email, and then crashes before confirming completion, the queue puts the message back. Another worker picks it up and sends the email again. The user receives the same confirmation email twice.
This is called duplicate processing, and handling it correctly requires extra care. Either the worker has to be designed so that doing the same work twice causes no harm (the technical term for this is idempotency), or the system has to track which messages have already been completed.
Some messages keep failing. Imagine an email address is invalid. Every time a worker tries to send to it, it fails. The message goes back in the queue. Another worker picks it up, tries again, fails again. This can loop indefinitely, taking up queue space and worker time while never making progress.
Most queue systems have a mechanism for this: after a message fails a certain number of times, it gets moved to a separate place called a dead-letter queue. The dead-letter queue holds messages that couldn't be processed, so that engineers can inspect them, understand why they failed, and decide what to do.
None of these are unsolvable problems. They're the trade-offs that come with asynchronous processing. You gain speed, resilience, and the ability to absorb bursts. You take on the responsibility of handling failures gracefully, ensuring work isn't lost, and being careful when the same work might happen more than once.
That's the contract. Synchronous work is simple: it either works or it doesn't, and the user knows immediately. Asynchronous work is more powerful, but it requires the system to think carefully about what happens when individual pieces go wrong in the background where nobody is watching.
--
Conclusion
Let's look at what we've now built, from the very beginning of this series.
We started with a single server. We scaled traffic across many application servers behind a load balancer. We reduced unnecessary database work with caching, and kept that cache accurate with invalidation strategies. We distributed database reads across replicas. We split the data itself across shards when a single database became too large.
And now we've changed when work happens, not just how it's distributed.
The complete picture so far:
Users
|
v
[ Load Balancer ]
|
v
[ Application Servers ]
| |
v v
[ Cache ] [ Message Queue ]
|
v
[ Workers ]
|
v
[ Read Replicas ] [ Database Shards ]
At each step in this series, scaling meant something slightly different.
Load balancers let us scale traffic. Caching let us scale by reducing work. Read replicas let us scale reads. Sharding let us scale storage and writes. Message queues let us scale by changing when work happens.
That last idea is the most unusual one. Scaling doesn't always mean making something faster. Sometimes it means moving work out of the user's way entirely, letting them continue while the system handles the rest at its own pace.
But we've been making one quiet assumption this entire time.
We've assumed that users are somewhere near our servers. That when a user in Bengaluru or Berlin or São Paulo sends a request, the server receives it quickly and the response arrives quickly.
That assumption starts to break down at global scale.
A server sitting in a data center in Virginia is fast for users in Virginia. For a user in Singapore, that same request has to travel thousands of kilometers across the internet and back. No amount of caching, sharding, or queueing changes the speed of light.
What happens when the physical distance between your users and your servers becomes the bottleneck?
That's the problem Part 11 is about.
Top comments (0)