This is where distributed systems get interesting.
If you've worked with backend systems long enough, you've probably written code that looks roughly like this:
await using var transaction = await db.Database.BeginTransactionAsync();
order.Status = OrderStatus.Paid;
db.Orders.Update(order);
await db.SaveChangesAsync();
await messageBus.PublishAsync(
new PaymentCompleted(order.Id)
);
await transaction.CommitAsync();
At first glance, it looks reasonable.
The order is updated.
The event is published.
The transaction commits.
Done.
Except there is a problem hiding in the middle.
What happens if PublishAsync() fails?
What happens if RabbitMQ is temporarily unavailable?
What happens if the network connection drops after the broker accepted the message but before your application receives the response?
What happens if the application crashes at exactly the wrong millisecond?
And most importantly:
What does your system believe happened?
This is one of those backend problems that doesn't show up in a happy-path demo.
It shows up at 3:17 AM in production.
The problem with "just use a transaction"
Let's simplify the system.
Imagine an Order Service.
It has:
┌──────────────┐
│ Order API │
└──────┬───────┘
│
▼
┌──────────────┐
│ Order Service│
└──────┬───────┘
│
┌───────┴────────┐
▼ ▼
┌────────────┐ ┌──────────────┐
│ PostgreSQL │ │ RabbitMQ │
└────────────┘ └──────────────┘
A request comes in:
POST /orders/123/pay
The service needs to do two things:
- Update the database.
- Tell other services that the payment succeeded.
For example:
Order.Status = Paid
+
PaymentCompleted event
And this is where the trouble starts.
Your database and your message broker are two different systems.
A database transaction can guarantee atomicity inside the database.
RabbitMQ doesn't magically become part of that transaction.
So this is not really:
BEGIN
UPDATE DATABASE
PUBLISH MESSAGE
COMMIT
It is more like:
DATABASE MESSAGE BROKER
│ │
│ UPDATE │
│─────────────────────────────>│
│ │
│ COMMIT │
│ │
│ │
│ ??? PUBLISH ??? │
│ │
There is a gap.
And that gap is where distributed systems become difficult.
Failure scenario #1: Database first
Let's say we do the sensible-looking thing:
await db.SaveChangesAsync();
await messageBus.PublishAsync(message);
Suppose the database succeeds.
Then:
Database
──────────────
Order 123
Status = Paid
Everything looks good.
But immediately afterward:
RabbitMQ
──────────────
PaymentCompleted
❌
Maybe RabbitMQ is temporarily unavailable.
Maybe DNS failed.
Maybe the pod restarted.
Maybe the process crashed.
Maybe there was a network timeout.
Now your database says:
"Payment completed."
But the rest of your system never heard about it.
The Inventory Service doesn't know.
The Notification Service doesn't know.
The Analytics Service doesn't know.
Whatever depends on PaymentCompleted doesn't know.
And if you simply retry the entire HTTP request, you could create another problem.
Failure scenario #2: Message first
So perhaps we reverse the order:
await messageBus.PublishAsync(message);
await db.SaveChangesAsync();
Now imagine the message is successfully published.
Then the database transaction fails.
Maybe there is a deadlock.
Maybe a constraint violation occurs.
Maybe the database connection disappears.
Now we have the opposite situation:
RabbitMQ
──────────────
PaymentCompleted
✅
Database
──────────────
Order.Status = Pending
❌
The Inventory Service receives:
PaymentCompleted
But the Order Service says:
Actually... no.
Now we're inconsistent in the other direction.
"Can we just use distributed transactions?"
This is where someone usually says:
"Why not use a distributed transaction?"
In theory, we could try to coordinate the database and broker through a distributed transaction protocol.
In practice, this introduces another set of problems.
Distributed transactions can be complex, expensive, operationally awkward, and tightly couple infrastructure components.
The classic transactional outbox pattern exists largely because we want the database update and the intent to publish an event to become atomic without requiring a two-phase commit across the database and broker.
And this is where I think a very simple idea becomes extremely powerful.
The Outbox Pattern
Instead of immediately publishing the message, we save the message inside the same database transaction as the business change.
Something like:
┌─────────────────────────┐
│ Transaction │
│ │
│ UPDATE Orders │
│ + │
│ INSERT OutboxMessage │
│ │
└────────────┬────────────┘
│
COMMIT
│
┌────────────▼────────────┐
│ Database │
│ │
│ Orders │
│ OutboxMessages │
└────────────┬─────────────┘
│
│
Background Worker
│
▼
┌──────────────┐
│ RabbitMQ │
└──────────────┘
Now the important part:
The application isn't trying to atomically update two systems anymore.
It only has to atomically update one: the database.
A simple Outbox table
For example:
CREATE TABLE OutboxMessages
(
Id UUID PRIMARY KEY,
Type VARCHAR(200) NOT NULL,
Payload JSONB NOT NULL,
OccurredAt TIMESTAMP NOT NULL,
ProcessedAt TIMESTAMP NULL,
RetryCount INT NOT NULL DEFAULT 0
);
Now our application transaction becomes:
await using var transaction =
await db.Database.BeginTransactionAsync();
order.Status = OrderStatus.Paid;
db.Orders.Update(order);
var message = new OutboxMessage
{
Id = Guid.NewGuid(),
Type = nameof(PaymentCompleted),
Payload = JsonSerializer.Serialize(
new PaymentCompleted(order.Id)
),
OccurredAt = DateTime.UtcNow
};
db.OutboxMessages.Add(message);
await db.SaveChangesAsync();
await transaction.CommitAsync();
Notice something important.
We're not talking to RabbitMQ inside the transaction anymore.
We're just changing the database.
Either both changes succeed:
Orders
+
OutboxMessages
Or neither does.
That's the part we can make truly atomic.
The transactional outbox pattern specifically works by storing the outgoing message in the same database transaction and having a separate relay publish it to the broker.
But now we have another problem
The outbox worker has to read those messages and publish them.
Something like:
while (!stoppingToken.IsCancellationRequested)
{
var messages = await db.OutboxMessages
.Where(x => x.ProcessedAt == null)
.OrderBy(x => x.OccurredAt)
.Take(100)
.ToListAsync();
foreach (var message in messages)
{
await messageBus.PublishAsync(
message.Type,
message.Payload
);
message.ProcessedAt = DateTime.UtcNow;
}
await db.SaveChangesAsync();
}
Looks good.
Until we consider this:
1. Worker reads message
2. Worker publishes message
3. RabbitMQ accepts message
4. Worker crashes
5. Worker never marks message as processed
6. Worker restarts
7. Worker publishes message AGAIN
Congratulations.
We've solved one consistency problem and created another.
Duplicate messages.
And this is one of the most important lessons in distributed systems:
"Exactly once" is usually much harder than it sounds.
The outbox relay itself can publish a message more than once if it crashes after publishing but before recording that it was published. The standard pattern therefore expects consumers to be able to process duplicate messages safely.
Which brings us to my favorite word in distributed systems:
Idempotency
An operation is idempotent when performing it multiple times has the same effect as performing it once.
For example:
Set status = Paid
is naturally easier to make idempotent than:
balance += 100
because:
Set Paid
Set Paid
Set Paid
still results in:
Paid
But:
+100
+100
+100
doesn't.
Make the consumer idempotent
Suppose our PaymentCompleted event reaches the Notification Service.
We could create an inbox/processed-message table:
CREATE TABLE ProcessedMessages
(
MessageId UUID PRIMARY KEY,
ProcessedAt TIMESTAMP NOT NULL
);
Then:
await using var transaction =
await db.Database.BeginTransactionAsync();
var alreadyProcessed =
await db.ProcessedMessages
.AnyAsync(x => x.MessageId == message.Id);
if (alreadyProcessed)
{
return;
}
await notificationService.SendPaymentConfirmationAsync(
message.OrderId
);
db.ProcessedMessages.Add(
new ProcessedMessage
{
MessageId = message.Id,
ProcessedAt = DateTime.UtcNow
});
await db.SaveChangesAsync();
await transaction.CommitAsync();
The MessageId should also be protected by a database-level unique constraint.
Why?
Because this is not enough:
if (!exists)
{
insert();
}
Two instances can execute the check concurrently:
Instance A Instance B
│ │
│── Does it exist? ───────>│
│ │
│<────── No ───────────────│
│ │
│ │
│── Does it exist? ───────>│
│ │
│<────── No ───────────────│
│ │
▼ ▼
INSERT INSERT
Now both think they're the first.
That's why the database should enforce the invariant.
For example:
ALTER TABLE ProcessedMessages
ADD CONSTRAINT PK_ProcessedMessages
PRIMARY KEY (MessageId);
PostgreSQL's unique constraints are enforced through unique indexes, making the database itself responsible for preventing duplicate keys.
This is a pattern I really like:
Application logic decides what should happen.
The database enforces what must never happen.
But wait... there's another race condition
Let's make the system more realistic.
We have multiple instances:
┌───────────────┐
│ Load Balancer │
└───────┬───────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
API Pod 1 API Pod 2 API Pod 3
│ │ │
└──────────┼──────────┘
▼
Database
Now imagine all three workers poll the outbox at the same time.
They might all see:
Message #123
ProcessedAt = NULL
What stops them from all publishing it?
This is where things get interesting.
One approach is to atomically claim rows.
For example, conceptually:
SELECT *
FROM OutboxMessages
WHERE ProcessedAt IS NULL
ORDER BY OccurredAt
FOR UPDATE SKIP LOCKED
LIMIT 100;
The exact implementation depends on your database, workload and broker semantics, but the underlying idea is important:
Reading a message and claiming responsibility for it are not necessarily the same operation.
And once you start thinking about multiple instances, retries and crashes, the design becomes much more than:
"Let's add RabbitMQ."
What about retries?
The worker will fail.
That's normal.
A production system should expect failure.
So instead of:
try
{
await Publish(message);
}
catch
{
// ¯\_(ツ)_/¯
}
we need an actual retry strategy.
For example:
Attempt 1
↓
failure
↓
wait 1 second
↓
Attempt 2
↓
failure
↓
wait 5 seconds
↓
Attempt 3
↓
failure
↓
wait 30 seconds
↓
Attempt 4
↓
failure
↓
Dead Letter / Failed State
I generally prefer exponential backoff rather than hammering a dependency that is already unhealthy.
Something like:
var delay = TimeSpan.FromSeconds(
Math.Pow(2, retryCount)
);
But retrying forever is not a strategy.
Eventually you need to know:
"This message is failing too many times. What do we do with it?"
That's where a dead-letter mechanism, failed state, alerting and operational tooling become important.
A message stuck for 30 seconds might be normal.
A message stuck for 30 minutes might be a problem.
A message stuck for 3 days is probably an incident.
Observability is part of the architecture
This is another thing I think backend developers sometimes underestimate.
Having an outbox is not enough.
You need to be able to answer:
How many messages are waiting?
How old is the oldest message?
How many have failed?
How many retries happened?
Which message is failing?
Which service consumed it?
How long did it take?
Are duplicates increasing?
Is the broker unavailable?
For example, one metric I'd absolutely want is:
outbox_oldest_message_age_seconds
Because:
Queue size = 10
doesn't tell me much.
Those 10 messages might have been created 100 milliseconds ago.
But:
Queue size = 10
Oldest message = 47 minutes old
is a very different story.
There is no "perfect" outbox implementation
And this is probably the most important point.
The outbox pattern isn't magic.
It introduces its own costs.
You now have:
- Another table
- Another background process
- Retry logic
- Duplicate handling
- Cleanup/retention
- Monitoring
- Potential ordering problems
- Additional database load
- Operational complexity
The polling-publisher approach, for example, is straightforward and works with SQL databases, but ordering and efficient publishing become concerns. Transaction-log-based approaches can reduce some polling concerns but introduce database-specific infrastructure and their own duplicate-publishing considerations.
So I don't think the answer should be:
"Always use the Outbox Pattern."
The better question is:
"What consistency guarantee does this particular business operation actually require?"
That's the architectural question.
Payment systems make this especially interesting
Imagine this:
Customer
│
▼
Payment API
│
├──────────────► Payment Provider
│
▼
Database
│
▼
Outbox
│
▼
Message Broker
│
├────────► Order Service
│
├────────► Notification Service
│
└────────► Analytics
Now think about retries.
The customer clicks:
Pay
The request times out.
They click again.
The first payment might have succeeded even though the client never received the response.
Now you potentially have:
HTTP retry
+
Payment retry
+
Message retry
+
Consumer retry
Every layer can independently retry.
And suddenly:
Idempotency isn't an optimization.
It's a business requirement.
You might need an idempotency key such as:
Idempotency-Key: 4f1a7c...
and persist that key with the operation.
Now your system can distinguish:
New request
from:
Same request being retried
This is one reason I find payment and identity systems particularly interesting: the cost of "doing the same thing twice" can be much higher than simply returning a duplicate record.
The architecture I would start with
For a typical .NET microservice, I'd be comfortable starting with something like:
┌───────────────┐
│ Client │
└───────┬───────┘
│
▼
┌───────────────┐
│ ASP.NET Core │
│ API │
└───────┬───────┘
│
┌─────────▼─────────┐
│ DB Transaction │
│ │
│ Business Data │
│ + │
│ Outbox Message │
└─────────┬─────────┘
│
COMMIT
│
▼
┌───────────────┐
│ Outbox Worker │
└───────┬───────┘
│
retry/backoff
│
▼
┌───────────────┐
│ RabbitMQ │
└───────┬───────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Order Service Notification Analytics
Service
│ │
└───────┬────────┘
▼
Idempotent Consumer
And I'd explicitly design for these failure modes:
| Failure | Expected behavior |
|---|---|
| Database unavailable | Request fails; nothing is committed |
| Business transaction fails | No event exists |
| Broker unavailable | Event remains in outbox |
| Worker crashes before publish | Event is retried |
| Worker crashes after publish | Duplicate is possible |
| Consumer receives duplicate | Duplicate is safely ignored |
| Consumer fails | Message is retried |
| Message permanently fails | Dead-letter/failed state + alert |
| Multiple workers process same event | Database constraint protects invariant |
That's a much more realistic definition of "reliable" than:
"It works when everything works."
The part I like most about backend engineering
This is why I still find backend development so interesting.
The difficult part usually isn't writing:
await db.SaveChangesAsync();
The difficult part is asking:
What happens if the next line never executes?
And then:
What if it executes but the response is lost?
And:
What if the application crashes immediately afterward?
And:
What if two instances do it simultaneously?
And:
What if the message arrives twice?
And:
What if the database succeeds but the broker doesn't?
And eventually:
Can the system recover without someone manually fixing the data at 3 AM?
That's where architecture stops being a collection of boxes and arrows.
It becomes a set of guarantees.
My current rule of thumb
When designing distributed backend systems, I try to think in terms of failure boundaries, not just components.
Instead of asking:
"Should we use RabbitMQ?"
I'd ask:
"What happens if RabbitMQ disappears for 10 minutes?"
Instead of:
"Should this be a microservice?"
I'd ask:
"What happens when this service is unavailable?"
Instead of:
"Should we add retries?"
I'd ask:
"What happens when the operation succeeds but the response is lost and we retry it?"
Instead of:
"Can we process this asynchronously?"
I'd ask:
"What consistency guarantee does the business actually need?"
Those questions usually lead to much better architecture decisions.
One final thought
I think one of the biggest misconceptions about backend engineering is that reliability comes from adding more infrastructure.
More services.
More queues.
More replicas.
More caching.
More Kubernetes.
More distributed components.
Sometimes it does.
But sometimes the best reliability improvement is much simpler:
Make the important invariant explicit.
Then decide where that invariant should be enforced.
Sometimes that's the application.
Sometimes that's the database.
Sometimes it's both.
And sometimes the correct answer is to avoid distributing the operation in the first place.
The more distributed our systems become, the more valuable these fundamentals become.
Because eventually every distributed system has the same question waiting for us:
"What happens when things fail between step A and step B?"
That's the question I'm increasingly interested in.
And I suspect it's one of the questions that separates code that works from software that can actually survive production.
What would you choose?
Suppose you have:
Database update
+
Event publication
and you cannot use a distributed transaction.
Would you choose:
A) Transactional Outbox
B) Direct publish + retry
C) Event sourcing
D) Something else
And more importantly:
What failure scenario would drive your decision?
I'd genuinely like to hear how other backend engineers approach this.
I'm planning to write more about the practical side of backend engineering—especially .NET, distributed systems, microservices, databases, identity/security, and the kinds of problems that only become obvious when software meets production.
#BackendDevelopment #DotNet #CSharp #Microservices #DistributedSystems #SoftwareArchitecture #RabbitMQ #PostgreSQL #Database #SystemDesign
Top comments (0)