Week 6: Async & Messaging: Designing Systems That Don't Have to Do Everything Now
Overview
In Week 6, I focused on a part of system design that changes how I think about communication between components:
- Synchronous vs asynchronous processing
- Queues and message brokers
- Pub/Sub and event-driven architecture
- Message delivery guarantees
- Retries and Dead Letter Queues
- Idempotency and duplicate messages
- Ordering and failure handling
The main idea I took away this week:
Not every piece of work needs to happen while the user is waiting for the response.
Moving work to asynchronous processing can improve responsiveness, absorb traffic spikes, and reduce coupling between services, but it also introduces a completely new set of failure scenarios that we need to design for.
Synchronous vs Asynchronous Processing
Let's start with something simple.
Imagine an e-commerce system where a user creates an order.
A synchronous approach could look like this:
User
↓
Order Service
↓
Payment
↓
Inventory
↓
Email
↓
Response
Everything happens before the user receives a response.
This is straightforward, but it creates a problem:
If one of those operations is slow or unavailable, the entire request can become slow or fail.
With asynchronous processing, we can separate the work:
User
↓
Order Service
↓
Queue
↓
Response
Then:
Queue
↓
Order Worker
↓
Payment
↓
Inventory
↓
Email
The user doesn't need to wait for every operation to finish.
The system can acknowledge the request and process the remaining work in the background.
Why Use Asynchronous Processing?
There are several reasons.
1. Reduce user-facing latency
Suppose sending an email takes a couple of seconds.
There is usually little value in forcing the user to wait for the email service before receiving an order confirmation.
Instead:
Create order
↓
Queue email task
↓
Respond to user
The email can be processed separately.
2. Handle traffic spikes
Imagine a system normally receives:
100 orders/minute
but suddenly receives:
10,000 orders/minute
A queue can act as a buffer:
10,000 requests
↓
Queue
↓
Workers process at manageable speed
Instead of forcing every downstream service to immediately process the entire spike.
3. Decouple services
Without messaging:
Order Service
↓
Payment Service
↓
Inventory Service
↓
Email Service
The Order Service becomes tightly connected to everything it needs to call.
With messaging:
┌→ Payment
│
Order → Queue├→ Inventory
│
└→ Email
Different consumers can process the work independently.
This creates a more loosely coupled architecture.
Producer and Consumer
Two terms appear constantly when working with messaging systems:
Producer
The component that creates or sends a message.
Order Service
↓
Producer
↓
Queue
Consumer
The component that reads and processes the message.
Queue
↓
Consumer
↓
Payment Service
The basic pattern is:
Producer → Queue → Consumer
The message might represent work that needs to happen:
{
"type": "PaymentRequested",
"orderId": "12345"
}
Or an event that something already happened:
{
"type": "OrderCreated",
"orderId": "12345"
}
Queues vs Pub/Sub
One distinction I wanted to understand clearly this week was the difference between a queue and Pub/Sub.
Queue
A queue is generally about work that needs to be processed.
For example:
┌→ Worker 1
Queue ───────┼→ Worker 2
└→ Worker 3
Multiple workers can share the workload.
The queue allows the system to buffer work and process it as capacity becomes available.
A simple way to think about it:
"Someone needs to do this work."
Pub/Sub
Pub/Sub is more about broadcasting an event to multiple interested consumers.
For example:
┌→ Email
│
OrderCreated ────┼→ Analytics
│
└→ Inventory
The Order Service doesn't necessarily need to directly call each service.
It publishes:
OrderCreated
and different consumers can react to that event independently.
The mental model I use is:
Queue = process this work.
Pub/Sub = something happened; whoever cares can react.
Event-Driven Architecture
This leads to a bigger concept: event-driven architecture.
Instead of building a chain of direct service calls:
Order Service
↓
Inventory Service
↓
Email Service
↓
Analytics Service
we can have:
Order Service
↓
OrderCreated
↓
Message Broker
↓
┌───┼────┬──────┐
↓ ↓ ↓ ↓
Email Inventory Analytics Fraud
The Order Service is essentially saying:
"An order was created."
It doesn't necessarily need to know exactly which services are listening.
This can make systems more loosely coupled and easier to extend.
For example, adding a new analytics consumer doesn't necessarily require changing the Order Service itself.
The Problem With Asynchronous Systems
This is where the topic becomes much more interesting.
Once we introduce queues, we also introduce new failure modes.
Imagine:
Queue
↓
Order Worker
↓
Charge Customer
↓
Update Database
What happens if the worker crashes after charging the customer but before acknowledging the message?
Queue
↓
Order Worker
↓
Charge Customer
↓
💥 CRASH
The system may not know whether the message was successfully processed.
Should it try again?
This leads to message delivery guarantees.
At-Most-Once Delivery
At-most-once means:
A message is processed zero or one time.
The message might be lost if processing fails.
Message
↓
Consumer
↓
Failure
No retry means no duplicate processing, but the message may never be processed.
This can be acceptable for some non-critical use cases, such as certain analytics events.
At-Least-Once Delivery
At-least-once means:
The system tries to ensure the message is processed, but it may be processed more than once.
For example:
Message #123
↓
Consumer
↓
Process message
↓
💥 Crash before acknowledgement
↓
Message delivered again
Now:
Message #123
↓
Processed
↓
Processed AGAIN
This is one of the most important concepts from this week.
At-least-once delivery means consumers need to be prepared for duplicates.
Exactly-Once Delivery
Exactly-once sounds like the ideal solution:
Process every message exactly once.
In distributed systems, however, guaranteeing true exactly-once processing across multiple components can be difficult.
For example:
Consumer
↓
Payment Provider
↓
Database
The payment might succeed, but the consumer could crash before recording that success.
When the message is retried, the consumer may not know whether the original payment happened.
Because of this, a common design approach is:
At-least-once delivery + idempotent consumers.
Rather than assuming duplicates will never happen, we design the system so duplicates are safe.
Idempotency
This was one of the concepts from Week 4 that became much more concrete this week.
An operation is idempotent when performing it multiple times produces the same final result as performing it once.
For example:
SET status = "ACTIVE"
Doing this once or ten times still results in:
ACTIVE
But:
balance = balance + €100
is not naturally idempotent.
Doing it twice gives:
+€200
instead of:
+€100
This becomes particularly important for payments and other operations where duplicate processing has real consequences.
Handling Duplicate Messages
One common approach is to give every event a unique ID:
{
"eventId": "abc-123",
"type": "PaymentRequested",
"orderId": "order-456"
}
The consumer can keep track of processed event IDs.
If it receives:
abc-123
again, it can recognize that the event was already processed and avoid performing the operation twice.
The important principle is:
If your messaging system can deliver a message more than once, your consumer needs to be designed accordingly.
Retries
Not every failure is permanent.
A downstream service might simply be temporarily unavailable.
For example:
Attempt 1 → failure
↓
wait
↓
Attempt 2 → failure
↓
wait
↓
Attempt 3 → success
Instead of retrying immediately and indefinitely, systems commonly use:
- Limited retry attempts
- Backoff
- Jitter
For example:
1 sec
↓
2 sec
↓
4 sec
↓
8 sec
This reduces the risk of making an already struggling service even more overloaded.
Dead Letter Queues
But what happens when a message keeps failing?
We don't want:
Message
↓
Retry
↓
Retry
↓
Retry
↓
Retry
↓
Forever...
Instead, after a certain number of failures, we can move it to a Dead Letter Queue (DLQ):
Message
↓
Attempt 1 ❌
↓
Attempt 2 ❌
↓
Attempt 3 ❌
↓
DLQ
The DLQ gives engineers a place to inspect problematic messages without allowing them to continuously interfere with normal processing.
For example, a message might end up there because of:
- Invalid data
- A persistent downstream failure
- A software bug
- An unexpected edge case
Message Ordering
Another question I considered was:
Does the order of messages matter?
Imagine these events:
1. OrderCreated
2. OrderCancelled
If they are processed in the opposite order:
OrderCancelled
↓
OrderCreated
the system could end up in an incorrect state.
But not every system needs strict ordering.
For example:
ProductViewed
ProductViewed
ProductViewed
may not require a specific order.
This means ordering should be treated as a business requirement, not something we automatically enable everywhere.
There is also a trade-off: stronger ordering requirements can restrict how much work can be processed in parallel.
Designing an Order Processing System
For this week's design exercise, I used an order processing system to bring these concepts together.
A simplified architecture looks like:
User
↓
POST /orders
↓
Order Service
↓
Order DB
↓
OrderCreated
↓
Queue
↓
Order Worker
/ | \
↓ ↓ ↓
Payment Inventory Email
The synchronous boundary could be:
User
↓
Create Order
↓
Save Order
↓
Publish message
↓
Return response
Then the remaining processing happens asynchronously.
The exact boundary depends on the requirements.
For example, if the user needs an immediate payment result, payment might need to remain synchronous. If the result can be processed later, it may be a good candidate for asynchronous processing.
The important design question is:
Which operations actually need to happen before the user receives a response?
Backend Lens
This week's design exercise made me look at asynchronous systems differently.
Instead of only asking:
"How do I process this message?"
I started asking:
What if the message is processed twice?
Can the consumer safely handle it?
What if processing fails halfway?
Could the operation leave the system in an inconsistent state?
What if the downstream service is temporarily unavailable?
Should we retry?
How many times?
What if the message never succeeds?
Should it go to a DLQ?
Does ordering matter?
If so, what level of ordering is actually required?
These questions are where a simple queue turns into a real system-design problem.
Key Takeaways
The biggest lessons I took from Week 6 were:
1. Async processing isn't just about performance
It can also help with:
- Decoupling
- Traffic spikes
- Resilience
- Independent scaling
2. Async systems introduce new failure modes
Once work becomes asynchronous, we have to think about:
- Duplicates
- Lost messages
- Retries
- Partial failures
- Ordering
3. At-least-once delivery changes how consumers are designed
If duplicates are possible, consumers need to be able to handle them safely.
4. Idempotency is extremely important
Especially for operations such as payments, where processing the same event twice can have real consequences.
5. Don't make everything asynchronous
The important question isn't:
"Should this be async?"
It's:
"What actually needs to happen before I can respond to the user?"
Reflection
Week 6 helped me connect several concepts from the previous weeks.
In Week 4, I learned about retries, timeouts, circuit breakers, and idempotency in the context of distributed calls.
This week, I saw the same ideas from a different perspective.
When communication becomes asynchronous, failure is no longer just a request that returns an error.
A message can be delayed, duplicated, processed partially, retried, or moved to a dead letter queue.
That means reliable asynchronous systems aren't simply about adding a queue.
They're about designing what happens when things don't go as planned.
And that is probably the biggest lesson I took from this week:
Asynchronous architecture gives you more flexibility, but it also makes failure handling part of the design itself.
What's Next?
Next week, I'll dive deeper into distributed systems and consistency, including replication, consistency models, and the trade-offs involved when data exists across multiple machines.
Top comments (0)