DEV Community

NARESH
NARESH

Posted on

When One Order Becomes Multiple Transactions: Inside a Saga-Based System.

Banner

A customer places an order. The Inventory Service successfully reserves the product, but the Payment Service fails to process the payment.

The expected outcome seems straightforward. Cancel the order and release the reserved inventory.

But what happens if the Inventory Service crashes while processing that release request? What if the same command is delivered twice? Or the payment succeeds, but the response never reaches the service coordinating the order?

Each service might have completed its own database transaction correctly, yet the overall checkout can still end up in an inconsistent state.

This was one of the problems I wanted to understand when I started building Orchestrix.

My interest in system reliability actually began with a different project.

In 2025, I built PromptNova, an AI-powered prompt engineering application hosted on Vercel and Render. During a prompt engineering session with around 60 participants, I shared the application for everyone to try.

It had worked during my own testing. But when others started using it, some encountered errors.

I never established the exact cause. What bothered me was that I couldn't confidently explain what had gone wrong or how the application would behave under those conditions.

That experience changed how I approached building software. I became more interested in what happens when requests fail, services become unavailable, or operations complete only halfway.

Eventually, I built Orchestrix as a way to explore those questions through a distributed e-commerce system.

And the first problem I wanted to solve was this: How do you keep one business transaction consistent when several independent services are responsible for completing it?


Why a normal database transaction is no longer enough

Consider a simple e-commerce application where orders, inventory, and wallet balances are managed within a single PostgreSQL database.

When a customer places an order, the application can begin a database transaction, create the order, reserve inventory, update the wallet balance, and commit everything together.

If an operation fails before the commit, the database can roll back the transaction. Its ACID guarantees allow these changes to succeed or fail as a single unit.

For many applications, this is a perfectly reasonable design. A modular monolith backed by a relational database can handle substantial workloads without introducing the coordination complexity of microservices.

But I wanted to understand what changes when these responsibilities are separated.

In Orchestrix, the Order, Inventory, and Payment services each own their data in independent PostgreSQL databases.

Suppose the Inventory Service successfully reserves a product and commits the transaction. The Payment Service then attempts to debit the customer's wallet, but the payment fails.

The inventory reservation has already been committed. The Payment Service cannot roll back that change because it belongs to another database.

Both services may have behaved correctly within their own transaction boundaries, yet the overall checkout remains incomplete.

A successful local transaction does not necessarily mean the business transaction has succeeded.

This creates a coordination problem. We need a way to ensure that independently executed operations produce an acceptable overall business outcome, including when some operations succeed and others fail.

There are different approaches to solving this. We could coordinate the database commits through a distributed transaction protocol, or allow services to commit independently and explicitly manage the consequences of partial success.

I chose to explore the second approach through the Saga pattern.

But that decision came with trade-offs, and understanding those trade-offs was just as important as implementing the workflow.


Why I explored Saga orchestration instead of distributed 2PC

Once I separated the Order, Inventory, and Payment services into independently owned databases, I had to decide how their transactions should be coordinated.

One option was Two-Phase Commit (2PC), which allows participating databases to reach a coordinated commit decision.

The protocol works in two phases.

Phase 1: Prepare. A transaction coordinator asks each participant to prepare its changes. Each database records the information needed for recovery and confirms whether it can commit.

Phase 2: Commit or abort. If every participant prepares successfully, the coordinator instructs them to commit. Otherwise, it instructs them to abort.

The appeal is clear. For a checkout operation, we would prefer inventory reservation and payment to succeed together rather than leave the system partially updated.

But 2PC introduces constraints.

Participating databases may need to retain locks and transaction resources while waiting for the final decision. If the coordinator fails after participants have prepared, those transactions can remain blocked until recovery resolves the outcome.

Every participant must also support the distributed transaction protocol. That becomes particularly restrictive when a workflow involves external systems, such as payment gateways, that cannot participate in the same database transaction.

In Orchestrix, both inventory and wallet operations were backed by PostgreSQL, so using 2PC was not impossible in principle. But it would have required distributed transaction coordination across the services and introduced dependencies I wasn't trying to build around.

My goal was to explore asynchronous communication, independently committed operations, and recovery from partial failures.

That made Saga a better fit for this experiment.

Why Saga made more sense for Orchestrix

With a Saga, services commit their own local transactions, and the workflow coordinates what happens afterward.

Inventory can commit a reservation without holding its database transaction open until payment finishes. If payment is explicitly declined, a new operation releases the reservation.

This avoids holding a distributed transaction across the entire checkout process, but it changes the consistency model.

A Saga does not provide an atomic commit across all participating databases. Other operations may observe intermediate states, and compensation can fail independently.

There is also no guarantee that every completed business action can be perfectly reversed. Compensation must be designed around the actual business operation.

I accepted those trade-offs deliberately. They were precisely the failure scenarios I wanted to investigate through Orchestrix.

Why orchestration instead of choreography?

Choosing Saga still left another decision: how should the workflow be coordinated?

With choreography, services react to events and publish new events without a dedicated coordinator. For a straightforward workflow, this can be an effective design with fewer central coordination responsibilities.

The difficulty appears as the workflow grows.

When payment fails, inventory must be released. When a response is delayed, the system needs to determine whether it should retry, wait, or recover. In a choreographed Saga, that coordination logic is distributed across event handlers and services.

For Orchestrix, I wanted the overall transaction state to be easier to inspect.

I introduced a dedicated Saga Orchestrator, backed by PostgreSQL, to persist workflow progress, issue commands, and decide which transition should happen next.

The Inventory and Payment services remain responsible for their own business operations. They don't need to understand the entire checkout sequence.

This gives the orchestrator a clear responsibility: tracking the business transaction across independently operating services.

The trade-off is additional centralization. The orchestrator becomes an important availability dependency, and its own persistence, event publication, recovery, and scaling need careful design. I also had to implement coordination behavior that a mature workflow engine could otherwise provide.

For a smaller checkout application, I would still consider a modular monolith before introducing this complexity. Orchestrix was an engineering experiment, not an argument that every checkout system needs microservices or Saga orchestration.

For the problems I wanted to explore, though, a persistent orchestrator provided a useful place to make workflow decisions explicit.

The next challenge was implementing those decisions without losing track of the transaction when messages were delayed, duplicated, or never delivered.


Inside the Orchestrix checkout Saga

After choosing Saga orchestration, I needed to decide how an order would move through the system without keeping the customer's HTTP request open until every operation completed.

In Orchestrix, the Order Service validates the checkout request and creates an order with a PENDING status. Within the same PostgreSQL transaction, it also records an ORDER_CREATED event in an outbox table.

Once the transaction commits, the customer can receive an acknowledgment. The order has been accepted, but inventory reservation and payment are still pending.

A background relay publishes the recorded event to Redpanda, allowing the checkout workflow to continue asynchronously.

I chose Redpanda as a Kafka-compatible event broker primarily because of my local development constraints. Its smaller Docker image, compared with the Kafka setup I had considered, was a practical advantage while running several microservices and databases on my personal computer. This wasn't a production performance comparison.

How the orchestrator coordinates the transaction

![orchestrator(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/k3nskxl9qse7y4w6yt7d.png)

Conceptual architecture of the checkout workflow. Commands and outcomes travel asynchronously through Redpanda, while each service maintains its own database.When the Saga Orchestrator receives the ORDER_CREATED event, it creates a Saga instance in its own PostgreSQL database.

This record tracks the progress of the checkout independently of the order record maintained by the Order Service.

The orchestrator then coordinates three main steps.

1. Reserving inventory

The Saga transitions to PENDING_INVENTORY and issues an inventory reservation command through Redpanda.

The Inventory Service checks the available stock and attempts to reserve the requested quantity.

Inventory updates use optimistic locking to detect concurrent modifications. If two transactions attempt to modify the same stock record, a version conflict prevents a stale update from being committed. The conflicting operation can then be retried according to the application's retry policy.

Once the reservation succeeds, the Inventory Service publishes the outcome, allowing the orchestrator to proceed.

2. Processing payment

After receiving confirmation that inventory has been reserved, the orchestrator moves to PENDING_PAYMENT and issues a payment command.

For Orchestrix, I deliberately chose an internal wallet-based payment system rather than integrating an external payment gateway.

My goal was to explore distributed transaction coordination, and an internal wallet allowed me to focus on database concurrency, transaction boundaries, and failure recovery without introducing the additional dependencies of a payment provider.

It also gave me direct control over the payment records needed to investigate transaction outcomes.

The Payment Service uses pessimistic database locking when updating wallet balances. By serializing competing operations on the same wallet, it can prevent concurrent debits from independently spending the same funds, provided the balance validation and update happen within the protected transaction.

If sufficient funds are available, the service commits the wallet debit and corresponding payment record locally, then communicates the outcome asynchronously.

This design was appropriate for my experiment, but it does not address every challenge of a real payment integration.

I later explored how the service would need to evolve to handle external providers, ambiguous payment outcomes, webhooks, and reconciliation. I documented that investigation in I Thought Idempotency Was Enough: Redesigning a Payment Service for Real-World Failures.

3. Completing the order

When the orchestrator receives a successful payment result, it transitions the Saga to COMPLETED and publishes the final outcome.

The Order Service consumes that result and updates the order to CONFIRMED.

Throughout this process, each service remains responsible for its own business data. The Order Service does not modify inventory or wallet records, and neither Inventory nor Payment needs to manage the entire checkout workflow.

That coordination belongs to the Saga Orchestrator.

Why persisting the Saga state matters

Consider a failure after inventory has been reserved but before the payment result reaches the orchestrator.

If the workflow existed only in memory, an orchestrator restart could lose the information needed to determine which operations had already been requested or confirmed.

By persisting the Saga state in PostgreSQL, Orchestrix retains the last committed workflow transition across application restarts.

The explicit state machine also helps the orchestrator validate incoming events against the current workflow state, rather than treating every received message as a new instruction.

However, persistence alone does not guarantee safe recovery.

A command might have been processed even though its response was lost. The orchestrator may know which state it last committed without knowing whether another service completed an operation.

Recovering from these situations requires additional mechanisms, including idempotency, timeout handling, and reconciliation of uncertain outcomes.

The successful checkout flow is relatively straightforward. The more difficult case begins when payment fails after inventory has already committed its reservation.

At that point, the orchestrator must coordinate another business operation to reverse the effect of the first.

That is where compensating transactions become important.


Compensation is more complicated than rollback

When payment fails after inventory has already been reserved, the expected recovery action is to release the inventory.

But compensation works differently from a database rollback.

In a database transaction, changes can be rolled back before they are committed. With a Saga, the Inventory Service has already committed its reservation in its own database. The orchestrator cannot reverse that committed transaction directly.

Instead, it must request another business operation to reverse its effect.

In Orchestrix, when the Payment Service explicitly declines a payment, such as when the customer's wallet has insufficient funds, the Saga Orchestrator transitions to COMPENSATING_INVENTORY.

It publishes a RELEASE_INVENTORY command through Redpanda. The Inventory Service processes the command, restores the reserved quantity, and publishes an INVENTORY_RELEASED event.

Once the orchestrator receives confirmation, it can mark the Saga as COMPENSATED and notify the Order Service that the checkout should be cancelled.

That is the expected recovery path. The difficult cases begin when the compensation command is duplicated, fails, or produces an uncertain outcome.

What if the compensation command is processed twice?

Consider a customer ordering the last available unit of a product.

Inventory starts with one available unit. The reservation succeeds, reducing the available quantity to zero. Payment fails, so the orchestrator requests that the reservation be released.

The Inventory Service restores the quantity to one and commits the change. However, its consumer crashes before acknowledging the command.

When the consumer restarts, Redpanda may deliver the same command again.

If the handler blindly increments the available stock each time it receives a release command, the inventory could increase to two, even though only one unit originally existed.

The recovery operation would have created a new inconsistency.

This is why compensation needs idempotency. Repeating a release request for the same reservation must not repeatedly increase the available stock.

Orchestrix uses database-backed command deduplication to recognize previously processed operations. However, a processed-command record alone is not enough to establish correctness.

The important requirement is that the inventory change and the record marking its command as processed must commit atomically.

If those operations use separate transactions, a crash between them could still allow a duplicate inventory release or cause a necessary release to be skipped.

This is a correctness requirement I would verify at the transaction boundary rather than assume is guaranteed simply because a deduplication mechanism exists.

What happens when compensation itself fails?

Duplicate delivery is only one possible problem.

Suppose payment has failed, but the Inventory Service is unavailable when the orchestrator publishes the release command.

The original reservation remains committed, and the Saga cannot safely declare compensation complete.

Retries can help with temporary failures, provided the repeated command retains a stable identity and the handler processes it idempotently.

But retries do not guarantee eventual success. A persistent database error, invalid reservation state, or recurring application failure may require investigation.

For a reliable recovery design, the unresolved compensation must remain visible until the system can establish its outcome. Depending on the failure, that may involve further retries, reconciliation, or operational intervention.

The important distinction is that publishing a compensation command is not the same as successfully compensating the transaction.

A timeout does not always mean failure

There is another failure scenario that exposes a more serious limitation.

Suppose the Payment Service successfully debits the customer's wallet, but the success event is lost or delayed before reaching the orchestrator.

The payment has succeeded locally. However, the Saga remains in PENDING_PAYMENT because the orchestrator has not received confirmation.

Orchestrix includes a timeout sweeper that examines Sagas remaining in pending states beyond a configured duration and initiates recovery actions.

This helps identify workflows that are no longer progressing, but the timeout cannot establish what happened inside the Payment Service.

If the sweeper triggers inventory release and order cancellation while the wallet debit has actually succeeded, the system can end up with a cancelled order and a charged customer.

A timeout sweeper provides a mechanism for recovering stuck workflows. It does not, by itself, guarantee the correct business outcome.

This is an important limitation of the recovery design.

A safer approach would distinguish a confirmed payment failure from an unknown payment outcome. When the outcome is uncertain, the system should reconcile the payment operation against the wallet's committed records before deciding whether to complete or compensate the Saga.

For an external payment gateway, that reconciliation may involve querying the provider or processing a delayed confirmation.

I would treat this as a necessary improvement before considering the checkout workflow production-ready.

Working through these scenarios changed how I thought about compensation. The challenge was no longer just identifying which operation could reverse a previous one. I also needed to consider whether that reversal had happened, whether it was safe to repeat, and whether the system had enough information to make the decision.

Even with those safeguards, another failure window remains. The orchestrator can successfully update its Saga state in PostgreSQL and crash before publishing the command that moves the workflow forward.

That is the problem I explored next with the transactional outbox.


The failure window between PostgreSQL and Kafka

The Saga Orchestrator now knows which operation should happen next. But updating the workflow state and delivering the next command are two separate operations.

Consider what happens after the Inventory Service successfully reserves the products.

The orchestrator updates the Saga state to PENDING_PAYMENT in PostgreSQL. It then needs to publish a payment command to Redpanda, the Kafka-compatible message broker used by Orchestrix.

A straightforward implementation might commit the database transaction first and publish the command afterward.

But suppose PostgreSQL commits successfully and the application crashes before publishing the message.

The Saga now indicates that payment is pending, but the Payment Service never receives the command.

Reversing the order doesn't solve the problem. If the message reaches the broker but the subsequent database transaction fails, the Payment Service could process a command that has no corresponding committed Saga state.

This is the dual-write problem. We need to update PostgreSQL and publish a message to another system, but those two operations do not share an atomic transaction.

How I approached this with the transactional outbox

In Orchestrix, I used the Transactional Outbox Pattern to make event publication recoverable.

Instead of immediately publishing a message after changing business data, the service writes an event record to an outbox_events table as part of the same PostgreSQL transaction.

For example, when the Order Service creates an order, it commits both the PENDING order record and the corresponding ORDER_CREATED outbox event together.

The same principle is important when the orchestrator records a Saga transition and prepares the command for the next operation.

If the database transaction fails, neither the business change nor its outbox record is committed. If it succeeds, the intention to publish remains stored in PostgreSQL, even when the message broker is temporarily unavailable.

A separate worker handles the actual publication.

Orchestrix uses an application-level OutboxRelayWorker that periodically queries PostgreSQL for unpublished events. The relay uses FOR UPDATE SKIP LOCKED to help concurrent workers claim different records without waiting on rows another worker has locked.

It publishes the selected events to Redpanda and, after receiving acknowledgment from the broker, records them as published.

I chose scheduled polling rather than introducing Debezium-based Change Data Capture (CDC).

My development environment was already running multiple services, databases, and supporting infrastructure. A polling relay gave me a relatively straightforward way to implement and inspect the publication process without operating another connector.

The trade-off is additional database queries and publication latency between polling cycles.

CDC offers another approach. A connector such as Debezium can capture committed outbox changes from PostgreSQL's write-ahead log and publish them without periodically querying the outbox table.

That can reduce polling overhead, but it introduces its own operational responsibilities, including connector management, replication slots, and monitoring.

For Orchestrix, polling was a reasonable choice given the project's scope and local development constraints.

Reliable publication still allows duplicates

The transactional outbox addresses the gap between committing a database change and recording the message that needs to be published.

It doesn't eliminate every failure scenario.

Suppose the relay publishes a payment command and Redpanda acknowledges it. Before the relay can mark the outbox record as published, the worker crashes.

After restarting, it finds the same unpublished record and sends the command again.

The Payment Service may now receive the same logical command twice.

This is an important consequence of at-least-once delivery: we can retry publication without intentionally discarding an undelivered event, but duplicate delivery remains possible.

The receiving service must therefore recognize repeated commands and avoid applying the same business operation more than once. For Orchestrix, this is where database-backed command deduplication becomes necessary.

Idempotency also matters at the entry point. A client-generated checkout idempotency key helps the Order Service recognize repeated requests, while command deduplication protects downstream operations. They address different retry boundaries and both require appropriate database-level safeguards to remain correct under concurrency.

The outbox also creates an operational responsibility. If the relay repeatedly fails to publish events, those records can accumulate. Monitoring publication failures and the age of pending records is important because a committed outbox event does not guarantee prompt delivery.

What I found useful was recognizing the boundaries of each mechanism.

The transactional outbox protects the relationship between a committed database change and the intention to publish a message. Idempotency protects business operations from being applied repeatedly. Neither mechanism eliminates the need for recovery and reconciliation.

Together, these decisions gave Orchestrix a more deliberate approach to handling failures between services, while making the remaining reliability challenges easier to identify.


What building Orchestrix changed about how I think about transactions

When I started building Orchestrix, I thought the main challenge would be coordinating operations across independent services.

But as I worked through the failure scenarios, I realized that deciding which operation should happen next was only part of the problem. The harder question was what the system should do when it couldn't confidently determine what had already happened.

A payment could succeed without its confirmation reaching the orchestrator. A compensation command could be processed twice. Even a successfully committed database transaction could leave the overall checkout in an unresolved state.

Working through these cases made me more conscious of the guarantees I was relying on and the assumptions I still needed to verify.

I chose Saga orchestration because it gave me a way to make workflow coordination explicit. But that flexibility came with additional responsibilities, particularly around recovery, intermediate states, and uncertain outcomes.

I wouldn't introduce this complexity into every application. For many systems, simpler transactional boundaries would be easier to maintain and reason about.

Orchestrix helped me understand why these architectural trade-offs matter. It also made me recognize an important distinction: designing a system to handle failures is different from demonstrating that it actually handles them correctly.

That became the focus of my next exploration: examining how Orchestrix behaved under concurrent requests, failures, and recovery conditions, and separating what I expected the architecture to guarantee from what I could actually verify.


๐Ÿ“– Blog by Naresh B. A.

๐Ÿ‘จโ€๐Ÿ’ป Backend & AI Systems Engineer | Distributed Systems ยท Production ML

๐ŸŒ Portfolio: [Naresh B A]

๐Ÿ“ซ Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree.โค๏ธ

Top comments (0)