Handling Partial Failures in Distributed Systems

Quick Answer: To handle partial failures in distributed systems, avoid complex distributed transactions. Instead, design for the "unhappy path" using an orchestration pattern (like the Saga pattern). This ensures that if a downstream service fails after a payment is processed, compensating actions automatically roll back the transaction and refund the user.

Let's be honest: a lot of the code running in production right now is built on a prayer. I've watched too many developers spend weeks writing elegant features, only to completely ignore what happens when the network inevitably hiccups. They design exclusively for the "happy path" and treat system failures as a minor afterthought.

But here is the reality of distributed systems: the network is unreliable, APIs crash, and validation fails. If you do not actively write code to handle these failures, you will eventually leave your data in a corrupted state, which usually means taking a customer's money and giving them nothing in return.

Why does the "happy path" fallacy break distributed systems?

The happy path fallacy is the assumption that every API call, database write, and network request in a sequence will succeed. When developers write linear code without handling intermediate failures, it leads to inconsistent system states—like charging a customer without delivering their purchase.

I cannot tell you the number of times I've seen systems written in such a way where the code takes the money, sits in a middle server, and then calls a downstream server to deliver the product. If that downstream call fails, you are stuck in the absolute worst-case scenario: you have the customer's money, but they have no product.

Some developers might joke that keeping the money without delivering the product is a win, but I guarantee you it is the fastest way to get your engineering team into deep trouble.

// The dangerous "Happy Path" approach I see all the time
async function checkout(order) {
  // 1. Take the money
  await paymentService.charge(order.userId, order.amount);

  // 2. Network drop? Validation failure?
  // If this fails, the user is charged but gets no inventory!
  await inventoryService.reserve(order.items);
}

Why should I avoid distributed transactions (2PC)?

Distributed transactions, such as Two-Phase Commit (2PC), force services to lock databases until all nodes agree on a change. This tight coupling destroys system throughput, increases latency, and introduces a single point of failure that defeats the purpose of microservices.

When developers realize their linear code is fragile, their first instinct is often to reach for distributed transactions. I strongly advise against this. Distributed transactions require database locks across service boundaries. If one of your services experiences a network lag or goes down mid-transaction, your entire system grinds to a halt while holding those locks open. It simply does not scale.

How do I design a resilient rollback workflow?

Rather than relying on database locks, use an orchestration-based Saga pattern to manage distributed state. An orchestrator tracks each step of a multi-service workflow and automatically triggers compensating actions (like refunds) if a downstream step fails.

Instead of trying to prevent failures with database locks, I build systems that accept failure as an inevitability. I recommend using an orchestration system. An orchestrator is a resilient state machine that coordinates the workflow steps. If a downstream step fails, the orchestrator detects the failure and executes compensating actions—like issuing a refund—to roll back the entire workflow to a clean state.

Strategy	Latency	Coupling	Failure Handling	Best For
Two-Phase Commit (2PC)	High	Tight (database level)	Automated rollback, but blocks resources	Monoliths, single DBs
Choreographed Saga	Low	Loose (event-driven)	Complex to trace, relies on event chains	Simple, decentralized workflows
Orchestrated Saga	Medium	Loose (central controller)	Explicit compensating steps (rollbacks)	Complex business transactions

FAQ

How do I test unhappy paths in my local development environment?

I suggest using chaos engineering tools or configuring mock downstream APIs to return random timeouts, 500 Internal Server Error responses, or validation failures. If your orchestrator does not automatically trigger a compensating rollback during these simulated network drops, your recovery logic is broken.

What is a compensating transaction?

A compensating transaction is an explicit action designed to undo the effects of a previous, successful step in a workflow. For example, if step one charges a card and step two fails to allocate warehouse stock, the compensating transaction is an API call that issues a refund to the customer's card.

What tools should I use for workflow orchestration?

I highly recommend looking at dedicated workflow engines like Temporal, Camunda, or AWS Step Functions. These platforms allow you to define resilient state machines that handle retries, timeouts, and compensating rollbacks out of the box, saving you from writing custom, fragile error-handling boilerplate.