The transition from a monolith to a microservice architecture brings flexibility and scalability, but also creates new challenges. One of the key issues is data consistency and transactions. In a monolith, you can typically wrap multiple operations in a single ACID transaction: either all operations succeed, or an error triggers a full rollback. In the world of microservices, this straightforward approach doesn't work. Each service is autonomous, each has its own database, and they communicate over a network. As a result, guaranteeing the atomicity and integrity of processes spanning multiple services is difficult. This creates the risk of partial updates: one part of the system changes while another doesn't, causing data to drift apart.
To address this, several patterns and protocols have been developed. Below are two poles — saga and two-phase commit — and what lies between them: TCC and Outbox. Separately, there's isolation: the single letter of ACID that a saga completely loses. This is usually ignored in articles about saga, and the cost is paid in production.
Limitations of ACID transactions in microservice architecture
ACID is the classic set of properties of a local transaction: Atomicity, Consistency, Isolation, and Durability. In the context of a single database, ACID guarantees that a transaction is indivisible — either executed entirely or rolled back without a trace, transitioning the database from one consistent state to another. However, in a microservice architecture, where data is distributed across multiple services and databases, ensuring an ACID model "system-wide" is extremely difficult.
The data is physically separated: each service has its own database, and a transaction that affects two of them cannot be local. Built-in DBMS mechanisms do not work across network boundaries. The interaction itself occurs over the network — REST, gRPC, messages — with all the expected delays and interruptions. So, attempting to chain operations together as a single transaction runs into the problem that some operations are committed and others are not. There's also no isolation between services: without a shared transaction manager, one service can read another's intermediate state.
This is all well-known. But the following is usually formulated incorrectly.
The cost of coordination, not just the CAP theorem. CAP describes a narrow scenario — what to do during a network partition: maintain consistency and deny service, or respond, risking data divergence. A global ACID transaction chooses consistency here. But it incurs the main cost in normal operation, and CAP is silent about this. PACELC states it more completely: if Partition then A or C, else L or C — during a partition, we choose between availability and consistency, and the rest of the time, between latency and consistency. For 2PC, the latter half is more important than the former. Two network round trips per transaction are latency that every operation pays, even when the network is perfect. Plus, availability arithmetic: if each of the five participants is available 99% of the time, a transaction requiring agreement from all five is available 0.99⁵ ≈ 95% of the time — degradation without a single connection interruption. A bottleneck in a high-load system occurs not during a crash, but constantly.
Classic ACID transactions "across service boundaries" are either impossible without special protocols or lead to serious scalability and fault tolerance issues. A distributed system needs different approaches to consistency. Next, we'll consider two main solutions: the Saga pattern, based on transaction splitting and compensation, and the 2PC protocol, which coordinates atomic confirmation.
Saga Pattern: Distributed Transactions via Compensating Actions
A saga breaks a large business transaction into a sequence of local ones: each step is committed to its own service and immediately becomes visible. There is no global transaction at all, so there is no global rollback. If all steps are successful, the whole saga is considered successful. If an error occurs at some stage, the Saga pattern runs compensating transactions to undo actions already completed and return the system to a consistent state.
Motivation and how Saga works
A saga solves the problem of a business operation requiring data changes in multiple services. Example: placing an order in an online store — you need to create an order in the Order service, charge the payment in the Payment service, and reserve the product in the Inventory warehouse. In a monolith, we would do this in a single transaction. In microservices, a saga allows you to achieve a similar effect by sequentially executing local transactions.
The steps occur in a predetermined order, and each is a regular local transaction in its own service. Order creates an order and publishes OrderCreated, Payment debits the funds on that event, and Inventory reserves the product. While everything is running, the saga simply moves forward.
It breaks on the first failure. If Inventory responds that the product is out of stock, there's nothing to roll back: both previous transactions have long been committed. Instead of a rollback, compensation is triggered: Payment returns the money, and Order marks the order as "canceled." Compensation is the same transaction as everything else; it just does the opposite.
This result is often called "saga atomicity." More precisely, a saga provides ACD without I: atomicity is simulated by compensation, consistency and durability are provided by local transactions, and there is no isolation at all. There is no "without a trace" rollback — compensation is a new transaction that performs the reverse action, and the intermediate state becomes visible to others in the meantime. A canceled order remains in history with the "canceled" status, rather than disappearing.
Three types of steps: compensatable, pivot, retriable
Not all saga steps are the same, and before writing compensations, they should be sorted into three categories.
Compensatable — a step that can be undone by a reverse action. Order creation is canceled by changing the status, a reserve in the warehouse is removed, and a card hold can be released as long as the funds haven't been captured yet: the money hasn't been sent anywhere; the bank simply releases the frozen amount. Compensating transactions are written only for steps in this category.
Pivot — the point of no return. This is either the last compensatable step or the first non-compensatable step. Once it's committed, the saga cannot be rolled back: the only option is to move forward.
Retriable — everything that comes after the pivot. Such steps must eventually succeed, and they are never undone, only repeated. Examples include sending an email, publishing an event, or accruing bonuses.
Hence the rule by which any saga is designed: the order of steps must be compensatable → pivot → retriable, and nothing else. If a non-compensatable step is in the middle, the saga is broken at the design level; if the next step fails, there will be nothing to roll it back with.
How to find a pivot in your saga: go through the steps and at each one, ask — if this step has been completed, are we ready to cancel it automatically, without human intervention? The first "no" is the point of no return. The answer is provided by the business, not the code: if the payment provider has idempotent reversals and the business allows them without confirmation, the charge remains compensatable; if the reversal is subject to manual review, the charge becomes a pivot, and everything after it must be retriable.
A practical trick for when a pivot occurs too early: split the step into two phases. This is precisely why payments are almost always divided into authorization and capture — authorization is compensated by lifting the hold, while capture can be postponed until the very end. The later the point of no return, the more room the saga has for rollback.
And the opposite requirement applies to retriable steps: they cannot refuse for business reasons. If a step after a pivot can respond "no," it is not retriable, and the saga is designed incorrectly. Network errors and service unavailability can be retried, but domain-rule rejections cannot.
The most common mistake here seems harmless: a notification is sent as a regular saga step, and its failure triggers a rollback. Formally, everything is logical: the step failed, so we roll it back. In essence, we refund the client and cancel the order because the email was not sent.
Orchestration vs. choreography
The logic of a sequence must live somewhere, and there are exactly two options.
With orchestration, it is handled by a separate component — the saga orchestrator. It knows the entire scenario, calls Service A, then Service B, and triggers compensation itself if an error occurs. The entire sequence is read in a single file, but another service is added to the system that knows about all the others.
With choreography, the scenario doesn't exist anywhere; there are only reactions. The Order Service publishes OrderCreated, the Payment Service picks up and responds with PaymentApproved or PaymentFailed, the Order Service listens and decides whether to confirm or cancel the order. There's no central component, but to understand what actually happens when an order is placed, you'd have to open four repositories.
Choreography is often sufficient in simple systems, but orchestration can be more convenient for complex processes with multiple conditions.
What it costs
A saga has one advantage that an ACID transaction fundamentally can't have: compensation doesn't have to be an exact reversal. If a payment can't be automatically reversed, the compensation is marking the order "requires manual review." ACID rollback can't do this; it either cancels everything or nothing. The other advantages have already been mentioned above: there are no global locks, no coordinator, and services don't wait for each other.
Now the bill, and it's quite long.
Compensation must be written for each compensatable step, and this isn't a mechanical task: you have to decide what the system does if step X succeeds but step Y fails. "Cancel a payment" is a business decision, not a line of code, especially once the funds have already been debited.
Everything, including compensation, must be idempotent. Retries and duplicate messages are inevitable in a distributed environment: the service will receive the same event twice, and compensation will be retried after a failure. The "already done?" check against internal status is not enough — two retries arrive simultaneously, and both see "not done." Only an external operation identifier in a unique index works.
Debugging is more difficult than in a monolith: the sequence is not visible anywhere, especially with choreography. A correlation identifier in all messages and distributed tracing are essential from day one — without them, the saga simply cannot be reconstructed after an incident.
The three remaining cost items are discussed separately below: intermediate inconsistency in the next section, progress storage in the code example, and unsuitability for operations requiring instant atomicity in the section on 2PC.
No isolation: what this means in practice
A saga loses exactly one of its four ACID letters, and that's no small thing. The lack of isolation means that the saga's intermediate results are visible to everyone else: another saga, a background job, or a regular query reads the data when half the steps are completed and half are not. This is impossible in a local transaction; the DBMS is responsible for that. In a saga, responsibility shifts to you, and if you don't explicitly take it, the system suffers from three classic anomalies.
Lost updates. A saga overwrites a change someone made while it was running. A user placed an order, the saga reached the payment step, at which point the user clicked "cancel," the order was marked as canceled, and the next saga step overwrote it with "confirmed." The cancellation disappeared, and no error occurred: every local transaction executed correctly.
Dirty read. Someone makes a decision based on data that the saga will later reverse. An order is created, funds are debited, the loyalty service sees the debit and credits cashback. Two seconds later, the saga crashes when reserving the item, the funds are returned, and the cashback remains. The compensation canceled the debit, but didn't cancel someone else's decision based on it.
Unrepeatable read. Two steps in the same saga read the same thing and see different results. The first step checked that there are sufficient funds in the account, the third debits it, and in between, another transaction occurred, and the funds are gone. The check performed at the beginning of the saga is out of date by the time the action is taken.
All three have one thing in common: none of them show up in happy path tests, and none of them produce errors in the logs.
Countermeasures
The set of techniques here has long been established, and you have to choose deliberately; there is no universal solution.
Semantic locking. The record is marked with an "in progress" flag — PENDING, PROCESSING, RESERVED status. Anyone reading it must respect this flag: wait, refuse the user, or show the data with a caveat. This is the most common countermeasure, but it has a price that is often overlooked: you need to decide what the reader does when they see the flag, and a timeout is essential, otherwise a failed saga will leave the record locked forever.
Commutative updates. Operations for which order is irrelevant: balance = balance - 100 instead of balance = 900. Then a lost update is impossible in principle, and compensation becomes trivial — add back. This is the cheapest technique of all: it requires nothing more than writing deltas instead of absolute values.
Pessimistic step ordering. Sometimes an anomaly is easier to fix by reordering the steps than by protecting against it. If cashback is awarded after the entire saga, rather than after the debit, the dirty read in the example above disappears on its own. Reordering doesn't provide formal guarantees, but it significantly reduces business risk and doesn't require a single line of infrastructure code.
Re-read before write. Before writing, the step rereads the record and checks that it hasn't changed since it read it. If it has, the saga is aborted and restarted. This is essentially standard optimistic version locking, and it works against lost updates.
Version log. Operations are written to a log and applied in order, even if they arrive out of order. If an order cancellation arrives before its creation, both operations will be logged and applied correctly. This technique converts non-commutative operations into commutative ones, at the cost of additional storage.
Choose the mechanism by what's at stake. A system-level, not a step-level strategy: the mechanism is chosen based on the cost of error. A thousand-ruble transfer is processed by a saga, while a million-ruble transfer goes through a strict transaction or manual confirmation. This is an admission that a single consistency model for all operations is a compromise, and somewhere it will be the wrong one.
Where to start
First, it's worth checking whether operations can be made commutative; it costs nothing and eliminates a whole class of problems. Place semantic locks on the records the saga holds between steps and immediately set a timeout for them. Next, reread the value where a concurrent write is likely. And check the order of the steps separately: some anomalies can be resolved with a simple rearrangement.
The main thing is not to treat this as an additional reliability feature that can be added later. The lack of isolation doesn't manifest itself under low load and doesn't break tests. It manifests itself in production as discrepancies that are impossible to reproduce.
Example: a saga for placing an order
Let's take the same scenario: create an order, take payment, send confirmation. A full-fledged saga engine won't fit in this article, but it doesn't need to be shown in its entirety — two things are enough: how the saga is described and what the engine does with this description.
A saga is defined by a list of steps. Each step has an action, a compensation, and a type — the same compensatable / pivot / retriable discussed above. The type determines what happens in the event of a failure.
// Step = action + compensation + type.
// The type decides what the engine does on failure.
List<SagaStep> ORDER_SAGA = List.of(
step("create-order", COMPENSATABLE,
ctx -> ctx.put("orderId", orders.create(ctx.sagaId(), ctx.payload())),
ctx -> orders.cancel(ctx.sagaId(), ctx.getLong("orderId"))),
step("authorize-payment", COMPENSATABLE,
ctx -> ctx.put("authId", payments.authorize(ctx.sagaId(), ctx.amount())),
ctx -> payments.releaseHold(ctx.sagaId(), ctx.get("authId"))),
// the money is gone - no rollback from here on
step("capture-payment", PIVOT,
ctx -> payments.capture(ctx.sagaId(), ctx.get("authId")),
ctx -> { throw new UnsupportedOperationException("a pivot cannot be compensated"); }),
// the email failed - not a reason to refund
step("send-confirmation", RETRIABLE,
ctx -> notifications.confirm(ctx.sagaId(), ctx.getLong("orderId")),
ctx -> { throw new UnsupportedOperationException("retriable steps are only retried"); })
);
Pay attention to the payment. The authorization is compensatable — the bank simply removes the hold; the money never went anywhere. The capture can't be compensated for, so it's declared the point of no return, and everything after that must be retriable. Because of this, a failure to send the email will result in a new attempt to send it, rather than a cancellation of the paid order.
Now the engine. The saga progress is stored in a standard table: saga ID, current step number, accumulated context, next attempt time. A background worker selects the sagas that are due and advances each one exactly one step:
// One tick: take a saga, run the next step, persist the progress
void advance(SagaRecord saga) {
SagaStep step = steps(saga).get(saga.step());
try {
// network call, Idempotency-Key = sagaId
step.execute(ctx);
// local commit: step number + context
repo.save(saga.advanced(ctx));
// If the process dies between these two lines, the step runs again.
// That is not a bug, it is the contract: execute must be idempotent.
} catch (TransientFailure e) {
// timeout, 5xx, dropped connection - retry later
repo.save(saga.retryLater(backoff(saga.attempts()), e));
} catch (BusinessRejection e) {
// a business "no" - compensate, but no further back than the pivot
repo.save(saga.startCompensation(e));
}
}
Ten lines, and they contain almost everything that distinguishes a working saga from a textbook one. The state survives process restarts — a crashed worker will recover and continue from the same step. Failures are divided into two kinds, and confusing them is expensive: a timeout doesn't mean the operation failed. A rollback on timeout compensates a payment that actually went through, and the money is returned to the client twice. And an idempotency key is required not for aesthetics, but because a retry is guaranteed to happen.
A lot remains behind the scenes: the reverse pass over compensations, escalation to manual review when compensation fails after N attempts, deduplication on the receiving end, and a deadline for the saga as a whole. But it is this scaffolding that makes up the bulk of the code, not the business logic. Ready-made solutions handle exactly this: Camunda, Temporal and Cadence workflow engines, long-running actions (LRA in MicroProfile), Seata in SAGA mode. Inside they are all the same: a sequence of steps, compensations, and durably saved progress. The only question is whether you write it yourself or get it ready-made.
Two-Phase Commit (2PC)
Saga tolerates the intermediate state being visible. 2PC doesn't: it coordinates multiple nodes through a central coordinator so that only the final result is visible — either everyone commits or everyone rolls back.
How 2PC works
There can be two or more participants — services, databases, anything that can prepare and commit. A coordinator is placed over them, and then the process proceeds in two rounds.
Prepare phase (voting phase). The coordinator sends a request to all participants to prepare for commit. Each participant locally performs their part of the transaction (for example, makes the necessary changes to their database) but doesn't commit them, marking them as "ready to commit" (in databases, this usually means writing the changes to the log and locking resources). After this, the participant responds to the coordinator with either "ready" (Yes) if their stage has been successful and they are ready to commit, or "cannot" (No) if an error has occurred and they are unable to commit. All participants essentially vote "yes" or "no" on a shared commit.
Commit phase. The coordinator collects responses. If all participants respond "yes," the coordinator sends a Commit command to all participants. Each participant receives this command and commits their changes (permanently applies them to their system). If at least one participant rejects the request ("No") or fails to respond due to a failure, the coordinator sends a Rollback command to all participants who were ready. Either all participants commit or all participants roll back — the protocol allows no third outcome.
After completing the second phase, the coordinator can notify the initiator (e.g., the application that started the transaction) that the transaction was successfully completed or rolled back.
A full implementation of the coordinator would take several hundred lines, but the entire protocol rests on three points. We'll show them here.
The first thing the coordinator does after collecting votes:
// Phase 1: voting
boolean allReady = participants.stream().allMatch(p -> p.prepare(txId));
Decision decision = allReady ? COMMIT : ABORT;
// The key line of the whole protocol: the decision hits the log BEFORE it is broadcast.
// If the coordinator dies right after it, on startup it will read the log
// and drive the transaction to completion. Without fsync this is pointless.
log.writeDecision(txId, decision);
// Phase 2: the participants have already voted "yes" and must comply.
// An error at this stage is no reason to roll back - it is a reason to retry.
for (Participant p : participants) {
retryUntilSuccess(() -> p.apply(txId, decision));
}
A naive implementation broadcasts the decision immediately and gives up at the first error. Here, the decision is recorded in the log before anyone else knows about it: this record is what makes the protocol recoverable. Broadcasting is retried until it succeeds — after the voting phase, a participant's failure no longer changes anything; the transaction must complete as the coordinator decided.
What happens on the participant's side:
boolean prepare(TxId id) {
// the only moment when "no" is still an option
if (!canCommit()) return false;
// changes are on disk, but visible to no one
writeUndoRedoLog(id);
// these are the locks that will hang
lockRows(id);
// the state survives a restart of the participant itself
markInDoubt(id);
// after this, backing out is no longer possible
return true;
}
The markInDoubt line is the very same suspended transaction discussed above. The participant has written the changes, holds the locks, and is physically unable to make a decision on their own: they don't know how the vote went for everyone else. Any XA-compliant DBMS displays such transactions in a system view, and they must be resolved either by the returning coordinator or by an administrator by hand. Manual resolution is called a heuristic decision and is dangerous precisely because the administrator might choose something different from the coordinator: then one part of the system will commit while another rolls back — the very violation of atomicity that the protocol was designed to prevent.
Recovery: three lines that explain why the log was needed.
// On coordinator startup
for (Tx tx : log.unfinished()) {
// no decision in the log - abort
Decision d = tx.decision().orElse(ABORT);
for (Participant p : tx.participants()) retryUntilSuccess(() -> p.apply(tx.id, d));
}
The logic is simple: if the decision made it into the log, we play it out; if not, the coordinator died before the point of no return, and it's safe to abort. Inside retryUntilSuccess live timeouts, exponential backoff, and the attempt limit after which a human is paged. I've left those out of the listing, but that's where the protocol's real price hides.
This also reveals 2PC's real weakness. It's not the two phases or the locks, but the fact that the decision log exists in a single copy. While the coordinator is down, the participants wait and can't do anything. Distributed DBMSs solve exactly this problem by replicating the coordinator's log via Raft or Paxos: there are now multiple copies of the decision, the loss of a node is no longer fatal, and the protocol turns from unreliable to workable.
In real systems, the role of participants is performed by resource managers: each database can write changes to its log and confirm its readiness upon a prepare request, and complete the transaction upon a commit request. The coordinator is a transaction manager embedded in the application or the runtime environment.
2PC applications and tool support
The 2PC protocol was widely used in traditional enterprise applications, especially before the microservices era. Typical places where 2PC is encountered:
Distributed relational databases — for example, a transaction affects two different databases (two DBMSs or two connections). Java provides JTA (Java Transaction API) and XA drivers for this purpose; the coordinator (Narayana, Atomikos, etc.) ensures a two-phase commit between the databases.
A database plus a message queue — the classic use case of writing data to the database and sending a message without ending up with "written to the database but the message was lost," or the reverse. If the broker can be an XA resource, it can be included in a single global transaction with the database. Not everyone can do this: XA is supported by ActiveMQ (Classic and Artemis), IBM MQ, Oracle AQ, and other JMS providers implementing XAConnectionFactory. However, the two most popular brokers today, Kafka and RabbitMQ, are not XA resources. Kafka has its own producer transactions, but they are atomic only within Kafka and have no knowledge of your database. RabbitMQ has AMQP channel transactions, which are also not XA. For a typical modern stack, this path is simply a dead end, and this is where the Outbox pattern, which we'll return to below, comes from.
Classic monolithic systems with multiple resources — a transaction updating several subsystems (for example, two databases, or a database and a file system) could also be coordinated via 2PC.
Implementing 2PC requires support from all participants. Each resource must have a prepare/commit interface. In the world of relational databases, this is the XA standard; NoSQL stores or custom services often lack such support. Therefore, in pure microservices, where services are heterogeneous, implementing 2PC yourself is difficult — you either need to write an adapter layer for each one (so that the services can accept prepare/commit commands) or limit yourself to resources that already support XA. There are ready-made transaction managers (coordinators) — the aforementioned Atomikos, Narayana, Bitronix, and others — that can be embedded into an application and configured with resources to participate in a global transaction. However, this entails significant limitations, which are discussed below.
Why it's almost never used between services
Locks are held for the entire time between the phases. A participant who responded "ready" holds the rows locked until a command arrives and cannot release them on its own. If there are five participants, the slowest one sets the pace: while one is thinking, everyone is holding resources. The scaling ceiling here is set by the number of participants and the spread of their response times.
Availability degrades multiplicatively — we've already done that arithmetic above. The practical consequence: if one participant is unavailable, the whole operation fails. In this situation, a saga can wait until the service returns, or go into compensation. 2PC has no such choice: it either collects all the votes or cancels the transaction.
And coupling. All participants are required to obey a common coordinator and a common protocol — exactly what you were escaping when you split the monolith into services. The result is a distributed monolith at the transaction level: the services are separate, but they can only commit together.
2PC remains useful where the participants are few and homogeneous. A typical example is transferring money between accounts stored in the same sharded database but on different shards. A saga is bad here not because of code complexity but because of semantics: there's a window between the debit and the credit when no funds exist in either account, and a report taken at that moment will show an incorrect system-wide total. And the conditions that ruin 2PC between microservices aren't met here: there are exactly two participants, they're homogeneous, they're in the same cluster, and the lock is held for microseconds. That's why sharding layers like Citus perform cross-shard writes using a two-phase commit via the standard PREPARE TRANSACTION and COMMIT PREPARED statements in PostgreSQL.
And that's just the application layer. Within the storage layer, 2PC is alive and well: Google Spanner, CockroachDB, TiDB, and YugabyteDB perform distributed transactions using a two-phase commit, simply on top of consensus. This fixes the protocol's core problem: classic 2PC blocks when the coordinator holding the single copy of the decision goes down. If the coordinator's log is replicated via Raft or Paxos, the decision survives the loss of a node, and participants can always learn the outcome. Add homogeneous participants, a single cluster, sub-millisecond RTT, and short locks — and a protocol that was unbearable between microservices turns out to be perfectly workable.
Hence the practical conclusion: don't write 2PC yourself — get it with the database. If the invariant truly requires strict atomicity, it's cheaper to put related data in a single distributed DBMS that handles 2PC for you than to build an XA coordinator on top of your own services.
Comparing Saga and 2PC
They share the same goal, but differ in four areas.
| Saga | 2PC | |
|---|---|---|
| What's visible from the outside while the operation runs | The intermediate state is visible to everyone: the order is created, the money isn't charged yet. This has to be described in the service contract. | Nothing. Only the final result crosses the boundary. |
| What's required of the participants | Nothing beyond local transactions. Any service, any database. | XA support. Kafka, RabbitMQ, and most REST services cannot be participants. |
| What happens on failure | Compensations, which you write yourself. There is no "without a trace" rollback; intermediate effects stay in the history. | Centralized rollback. If the failure happens after the prepare phase, the transaction hangs until the coordinator or an administrator comes back. |
| Where it fits | Long business processes: orders, billing, bookings. Anywhere steps are compensatable and a second of latency hurts no one. | Participants are homogeneous, few, and in the same cluster: cross-shard writes, distributed DBMSs, several databases under one XA coordinator. |
Alternative approaches and patterns
Saga and 2PC are poles apart, but there are intermediate options between them, and these are the most common in practice.
TCC (Try-Confirm-Cancel)
A customer books a flight and a hotel together. The ticket is issued, but the rooms are sold out — and now they have to return the already-sold ticket. TCC (Try-Confirm-Cancel) addresses these situations by not selling anything until the last minute: first, everything is reserved (Try), and only when all parties have reservations does a confirmation (Confirm) or cancellation (Cancel) follow.
How TCC works. Let's imagine booking an air ticket and a hotel together. In the TCC model, the steps are as follows:
Try. The ticket service receives a booking request and makes a preliminary reservation (without finalizing the sale). At the same time, the hotel service reserves the room. These "Try" steps are performed for all participating services — they allocate the necessary resources, marking them as busy, but the transaction is not yet considered complete.
Confirm. If all services have successfully completed the Try step and are ready to complete the transaction, a Confirm command is sent to each service — tickets are issued permanently, and hotel reservations are confirmed. Each service makes the final change.
Cancel. If any of the Try steps fails (or one of the services responds that it cannot complete the transaction), a Cancel command is sent to all services that have already completed the Try step — they cancel any previously made reservations (free up seats, do not charge funds, and so on).
Externally, this appears atomic: either all Confirms are successful, or everything reserved is canceled with Cancel. This is similar to 2PC (Try is analogous to prepare, Confirm/Cancel to commit/rollback), but the difference isn't in the presence of a coordinator: in real implementations, one exists — in Seata, the same Transaction Coordinator is responsible for TCC mode as for the other modes. The difference is who implements the phases and what is held between them.
Who implements the phases. In XA/2PC, prepare and commit are implemented by the resource manager — the DBMS itself; the application is unaware of them. In TCC, the developer writes three methods: Try, Confirm, and Cancel — regular service methods called over regular HTTP or gRPC. This is the main advantage: any service that can serve three HTTP methods can become a participant.
What locks are held. In 2PC, a physical row lock is held between prepare and commit, held by a transaction opened over the network. In TCC, the Try step is committed locally and immediately — the "lock" becomes semantic: a reservation row with a status and a lifetime. There are no physical locks between services, and a stuck reservation is released by timeout rather than waiting for an administrator.
Compared to Saga, TCC is a saga whose intermediate effects are not visible from the outside: before Confirm, the order isn't placed and the money isn't debited, only a reservation exists.
TCC is a compromise: the strictness of 2PC, achieved by the application and without XA. Here's what you pay for it.
The service contract triples in size. Instead of one method there are three — Try, Confirm, Cancel — and the two-phase nature has to be dragged into the local logic. The hotel booking service now has to hold a room reserved, mark it as occupied and paid, and cancel a reservation: three states where there used to be two.
Reservations have to be released by someone. While the client is deciding whether to confirm, the room is unavailable to others. If a Confirm doesn't arrive within the agreed-upon time, Cancel must execute on its own, otherwise phantom reservations accumulate until someone notices. A Cancel that never arrives is a phantom too, so a background process is needed to find expired reservations and release them itself.
Idempotency — the same as in a saga, only now for three methods instead of two. A repeated Confirm shouldn't break an already confirmed reservation, and a repeated Cancel shouldn't break an already canceled one.
On the upside, failure is visible early: if a resource is unavailable during Try, no actual change occurs at all, and there's nothing to compensate for.
The main limitation is that not everything can be reserved. Bookings, holding funds on a card, capacity allocation — yes. Sending an email or writing to a log can't be reserved; those have to be handled by a saga with compensation, or ignored on cancellation.
In practice, TCC is implemented either by hand or using ready-made solutions: Seata has a TCC mode (alongside AT, XA, and SAGA), as does ByteTCC, while Atomikos supports it in the commercial ExtremeTransactions, under the same Try-Confirm/Cancel name. There is no established synonym for this pattern; in the literature, it is simply called TCC.
Outbox Pattern (Transactional Outbox)
The dual-write problem is one of the most common in microservices: how can we guarantee that an action in a local database and the sending of an event or message to another service occur atomically? For example, an Order Service has saved an order in its database and needs to send an OrderCreated event to Kafka for other services. If the write to the database succeeds but sending the message fails, the data has already changed, and other services won't know about it. Conversely, if the message is sent but the database isn't saved, other services will learn about an order that doesn't exist. Classic 2PC between the database and the broker is often unavailable. The Outbox pattern solves this problem by ensuring that the database and the message stay in sync.
The essence of the Outbox pattern: instead of sending a message directly, the service first stores the message information in a special Outbox table in its local database within the same transaction as the main data. Then, a separate process or thread reads this table and actually sends the messages to external recipients. The algorithm:
Local transaction. A request (for example, to create an order) arrives at the service. The service opens a transaction to its database. Within it, it performs the usual changes (creates an order record) and simultaneously inserts a record into the Outbox table — for example, a JSON payload describing the OrderCreated event that needs to be sent, plus a "new" status. The transaction is then committed. If for some reason the database write didn't go through, neither the order nor the event ends up in the Outbox. If the commit succeeds, both the order and the message record are durably in the database.
Sending from the Outbox. A separate component — let's call it the Outbox Processor — periodically reads new records from the Outbox table. For each record, it performs the actual send to the broker or calls an external service. After a successful send, it marks the Outbox record as sent. This operation is also transactional, locally within the database.
An order and an event are linked by a single commit: either both exist, or neither does. Even if sending to the broker is temporarily impossible, the record sits durably in the database and will wait for the next attempt. The database and the event stream can no longer diverge.
The technique comes down to a single trade: you get a delivery guarantee and pay for it in duplicates. The event will definitely be sent — it's in the database and will wait until the broker recovers. But it will be sent at least once, not exactly once: if the processor crashes after sending and before writing the "sent" mark, it will send again on restart. This means deduplication by message ID is mandatory on the receiving end — either an Inbox table or a unique index. Strict exactly-once isn't achievable here, and it's not worth relying on.
The rest is operational overhead. The table grows, it needs to be cleaned up, and its indexes watched: you've created your own little queue inside the service, with all the responsibilities of a queue. Sending happens with a delay of milliseconds to seconds, depending on the polling frequency or CDC settings.
You don't have to write your own processor: Debezium reads the transaction log and publishes events to Kafka. The price is a separate piece of infrastructure: Kafka Connect with a connector, access to the WAL or binlog with replication rights, and for the outbox table, usually an SMT router that fans records out into topics. "No code" here doesn't mean "no operations."
Despite these drawbacks, the Outbox pattern has become the de facto standard for building reliable asynchronous integrations between microservices. It complements Saga particularly well: for example, one service uses Outbox to publish an event that triggers the next Saga step in another service, ensuring that no step is lost.
Other techniques: eventual consistency, transactional messaging
A few more things that come up in any discussion of distributed transactions.
BASE model and eventual consistency. The counterpart to ACID. Basically Available, Soft state, Eventually consistent — a principle widely used in distributed systems: the system is always available for operation, but allows data to temporarily diverge, with consistency achieved "eventually." A saga is a special case of eventual consistency. Other examples: Event Sourcing (where state is computed from a stream of events), CQRS (separate command and query models, synchronized through events). In microservices, immediate consistency isn't always necessary; it's often enough that all services converge on the same data within a second or two.
Transactional messaging. A term for different ways to ensure atomicity between sending a message and changing state. Outbox is one of them. Another approach is to use the broker itself as a state store: send a command to a topic and consider the operation complete only after the other side confirms processing. Brokers also have their own transactions, but, as mentioned, they are atomic within the broker and don't cover your database, so they don't solve the dual-write problem on their own. There's a pattern called Transactional Inbox/Outbox, where incoming messages on the receiving end are also written to a local table and processed atomically with the service's local transaction — essentially an outbox on both ends.
Commercial distributed transactions. Historically, there have been products that enable two-phase commits between disparate systems. For example, Atomikos is a popular transaction manager for Java that allows multiple resources (databases, queues) to be included in a single JTA transaction. IBM MQ and IBM TX Series are examples of industrial-strength distributed transaction solutions. These solutions work, but, as noted, are rarely used in microservices due to their complexity and the requirements they place on participants.
Three-Phase Commit (3PC). Adds a third phase between voting and committing to allow participants to complete the transaction without a coordinator. It's non-blocking only in a model with fail-stop nodes and without network partitions. Under a partition, 3PC breaks not availability but correctness — two parts of the cluster can make opposing decisions. That's why it didn't catch on: the coordinator was made fault-tolerant by replicating the decision instead (Paxos Commit, Gray and Lamport, 2006).
An infrastructure-level solution. Related data is stored in a single distributed DBMS, which performs the distributed commit itself (see above on 2PC over consensus). Consistency is provided by the database, and this is perhaps the cheapest way to achieve strict atomicity where it's truly needed.
Practical recommendations: how to choose an approach
Here's what I check, in order, when the first operation spanning two services appears in a project.
Do there really need to be two services? The test isn't "is it difficult to implement" but "does one of these services ever change without the other?" If it never does, the boundary is drawn wrong, and a distributed transaction here only treats the symptom. Merging the services or duplicating the data is cheaper than building a protocol on top of a decomposition error.
What does the client get back? This question decides more than all the others. If the API can respond "accepted, ask for the status later," then a saga fits, and from there it's all about compensations. If the client needs a final answer within the same request, the saga is out: you'd have to expose an intermediate state, and there's nothing to expose. That leaves TCC with reservations, or moving the related data into a single database.
Who are the participants? If Kafka, RabbitMQ, or someone else's HTTP service is among them, the question of 2PC is off the table — XA isn't there and won't be: that leaves Outbox for delivery and a saga for the process. If the participants are your own, homogeneous, and in the same cluster, it's worth checking whether the problem can be solved by moving to a distributed DBMS that will do the two-phase commit for you.
Where is the point of no return? Go through the steps and find the first one that can't be undone automatically. If it turns out to be the second of five, the saga is nearly useless — there'll be nothing to roll back. Then either the step is split into two phases, like authorization and capture, or the order of steps changes, or the operation isn't suited to a saga at all.
How much does an error cost? This question comes last, because the answer to it can override the previous four. The mechanism is chosen by the cost of the operation, not one for the entire product — that was covered above in the countermeasures. And if the operation is expensive and irreversible, it's cheaper to take it out of automation entirely and require human confirmation.
What's required regardless
Idempotency and a correlation identifier are mandatory in any case, and both were covered above. There's a third thing that wasn't: every distributed operation needs a deadline and an escalation target. A saga that can neither complete nor roll back must not hang forever, and must not disappear quietly — after N attempts it has to land in a human's review queue. This is the most boring piece of the scaffolding and the first one people forget to write, because it never fires before production.
A distributed transaction is visible from the outside. The client receives "payment processing" instead of "paid" and should be able to ask for the status later. This decision is made when designing the API, not when choosing a library. 2PC hides the complexity in the infrastructure and works as long as the participants are homogeneous and sit close together. Saga moves it into the code and the business logic: you have to write more, but you can see what happens on failure and who is responsible for it. The worst option is the third one — to assume there's no complexity, and find it in production when the money has been debited and the order hasn't been created.
Top comments (0)