Introduction
In a microservice architecture, it is often recommended that each service has its own separate
data storage. This is because the nature of microservices — and why they are created —
mandates such design.
Implementing such architecture would allow for many things, such as: separate schema updates,
the use of proper storage technologies for the task the microservice is trying to achieve, ...
But how can we handle data operations (retrieval/update/...) that span multiple services? For example:
- A transaction requires multiple updates: update balance and place order
- A transaction requires joins across different services: what is a customer's most commonly ordered items?
These data operations can fall under one of two categories: either we are trying to retrieve data
from multiple services and aggregate them, or we are trying to make modifications/deletions/insertions
that span multiple services.
In this post, we will focus on the latter — or to be more specific:
how do we handle cross-service transactions?
Transactions
Before we look into how to solve cross-service transactions, we first need to understand:
what are transactions?
Suppose you want to update a single row in a database: a customer's address. You would need to
execute a single instruction/operation:
UPDATE customers SET address='***' WHERE id='***';
Since this is a single operation, it would either succeed or fail. This is what we call an
atomic operation since it satisfies the all-or-nothing behaviour, meaning it either
succeeds in its entirety or fails in its entirety.
Now suppose it is our product's anniversary and we are feeling generous. We want to give our loyal
customers (those that stayed for at least 10 years) a staggering 1 dollar!!! as in-app credits.
The SQL for that is:
UPDATE customers SET balance = balance + 1 WHERE created_at < ***;
Suppose now that we have this amounts to 1000 updates, and the database server fails mid-operation.
This would leave the system in an inconsistent state: some of the customers wouldn't receive our
generous offer. This is an example of a non-atomic operation. In such situation, it is more
desirable to at least go back to the previous state and retry later.
Transactions aim to address the problem of atomicity. They allow you to execute multiple
instructions that either succeed all together, or rollback — allowing the preservation of the
consistency of the database.
Transactions in distributed systems
Now that we know what transactions are, how do they look when the operations need to happen across
different nodes connected through a network? Or more formally: how do they work in distributed systems?
But wait... what do distributed systems have to do with microservices? In some sense,
microservices are a kind of distributed system where nodes are services connected through a network.
Let us consider a scenario as an example to understand why distributed transactions differ from
traditional ones: a customer makes a purchase, and as such we need to insert a new row and update his
balance in an atomic manner. Why atomic, you might say? Suppose that we are able to insert a
purchase row, but the update to the customer balance fails; then it is as if the customer has
made a free purchase.
For a single database, the atomicity of the operation can be insured using the traditional transaction we talked
about earlier, that either succeeds or rolls back. But what if the customers and purchases
table live on different nodes? How can we coordinate the two nodes to ensure the atomicity of the operation?
This problem is not new, and databases have come up with multiple solutions for what is called
distributed transactions. A common example is Two-Phase Commit (or **2PC* for short*),
but it is out of the scope of this blog post.
"Ok then, problem solved!!!" you might say... but there are a few things that prevent us from
using this solution directly when it comes to microservice:
- Even though some databases already have support for distributed transactions, in a microservice architecture, different services might use different storage systems (for example, MongoDB and Postgres), which prevents implementing them at the database layer. Using a single storage backend across all services is also not an option, since the whole point of the architecture is to allow agility and the use of the proper tool for the job.
- Storage systems sit behind the services themselves, which prevents direct access and requires that we handle such transactions at the level of the services rather than leave it out for storage systems.
- Implementing a fully blown 2PC distributed transaction is complex even for a single service, let alone across dozens or hundreds.
So to solve for this, we need a different method that allows for the all-or-nothing capability of
transactions, but that is simpler and implementable at the service level.
This is where the Saga pattern comes in...
The Saga Pattern
Let us take a simple example of a banking application:
We have two services:
- A transactions service to handle all transactions done by customers.
- A customers service which stores information about customers and their banking information — most importantly, their account balance.
Now suppose in our example that the customer makes a purchase to buy an AI subscription
(because of all of what he has being hearing about its productivity benefits).
The bank's backend receives the request for the purchase. In a normal monolithic application,
the process is simple: you would insert a new row in the transactions table referencing the customer,
update the customer's balance, wrapping everything in a transaction to make sure this is applied atomically.
But in a microservice architecture, this isn't so straightforward as discussed earlier.
Let us consider the happy path. The transactions service inserts a new row with a PENDING status.
It then synchronizes with the customers service, which handles the update to the customer row.
Upon confirmation of the update, the transactions row would update its own newly inserted row to COMPLETE,
for example.
Let us suppose that the transactions service fails to synchronize the update with the customers service,
or the customers service fails to inform the transactions service about its successful update.
This is where the rollback part of the saga comes in, where each saga step has a
(compensating transaction) which works by rolling back all previous steps if a single one fails.
This is in part similar to how a database does transactions, but only ensures eventual consistency
rather than the atomicity insured by the database's ACID transactions.
There are two ways of implementing Sagas and coordinating services:
- Choreography: a service processes its part of the saga and publishes events that trigger updates to other services.
- Orchestration: a central saga orchestrator manages the full saga, saving its state and rolling back appropriately on failure.
One important caveat about Sagas is that the local write and the network call need to be atomic
— often resolved using the **Transactional Outbox Pattern** — or else the system
will become stuck in an inconsistent state.
We also ignored one part, which is the client waiting for his transaction to end (to start his Agentic life).
The most plausible way of solving this — other than keeping him waiting — would be to provide a
transactionId to the customer and a special endpoint for him to poll to see if his purchase succeeded or not.
Do you know of any other patterns?
Top comments (0)