Saga Rollback Mechanics: Compensating Transaction Ordering, Failure Atomicity, and the Partial Execution Trap

The Problem Sagas Were Supposed to Solve

Distributed transactions via 2PC are operationally expensive: coordinator becomes a single point of failure, participants hold locks across network round-trips, and any participant going down blocks the whole cohort. Sagas replace atomicity with a sequence of local transactions, each paired with a compensating transaction that semantically undoes its effect. The promise is looser coupling and no cross-service lock contention. The trap is that "semantically undo" is not the same as "atomically undo," and most production failures happen in that gap.

What Compensation Actually Means

A compensating transaction is not a rollback in the database sense. It is a new forward-moving operation that brings the system to a state that is equivalent to the pre-transaction state from a business perspective. This distinction matters for three reasons:

1. Side effects are already in the world. If step T3 sent an email, the compensation C3 cannot unsend it. You can send a follow-up, but the system is now in a different observable state.
2. Compensation can fail. C3 runs over a network against a service that may be unavailable. You now have a failed compensation, which is a strictly harder problem than the original failure.
3. Order of compensation is not the reverse of execution by default. It must be explicitly designed to be, and the ordering has semantic consequences.

Ordering Guarantees in the Compensation Sequence

Consider a five-step saga:

T1 → T2 → T3 → T4 → T5

If T4 fails, you must execute C3, C2, C1 in that order. Reversing out of order—say, running C1 before C3—can produce invariant violations. In an order-fulfillment context: if T2 reserved inventory and T3 charged payment, running C1 (cancel order record) before C3 (refund payment) and C2 (release inventory) leaves payment captured against a cancelled order until compensation catches up. That window is your partial execution trap.

The compensation sequence must be strictly LIFO with respect to successfully committed steps. Any coordinator that tracks step completion must persist that state before declaring a step committed. If the coordinator crashes between T3 succeeding and recording T3's success, on recovery it cannot safely determine whether to run C3.

State Machine Design for the Coordinator

The coordinator must be a durable state machine. Each saga instance has a sequence of steps, each step has a status enum, and transitions are persisted transactionally before being acted upon. A minimal Go representation:

type StepStatus int

const (
    StepPending StepStatus = iota
    StepExecuting
    StepCommitted
    StepCompensating
    StepCompensated
    StepFailed // terminal: compensation itself failed
)

type SagaStep struct {
    ID          string
    Name        string
    Status      StepStatus
    ExecutedAt  *time.Time
    CompensatedAt *time.Time
    Attempts    int
}

type SagaInstance struct {
    ID      string
    Steps   []SagaStep
    Version int // optimistic concurrency on coordinator state
}

The coordinator persists SagaInstance before invoking each step. When a step returns success, it atomically transitions the step to StepCommitted and persists before moving to the next step. On failure, it sets the failed step to its terminal state and begins walking backward through StepCommitted steps, setting each to StepCompensating, invoking the compensation, then transitioning to StepCompensated.

The Version field enforces optimistic concurrency if multiple coordinator instances could recover the same saga (e.g., after a pod restart with competing workers). A MongoDB update with a filter on both ID and Version prevents split-brain compensation runs:

filter := bson.M{"_id": saga.ID, "version": saga.Version}
update := bson.M{
    "$set":  bson.M{"steps": saga.Steps},
    "$inc":  bson.M{"version": 1},
}
result, err := col.UpdateOne(ctx, filter, update)
if result.MatchedCount == 0 {
    return ErrConcurrentModification
}

If MatchedCount is zero, another coordinator instance has advanced the saga. The current instance must re-fetch and re-evaluate rather than continue blindly.

Idempotency at Every Step

Because the coordinator retries on transient failures—including failures that happen after a step completes but before the coordinator records that completion—each step and each compensation must be idempotent. The standard mechanism is a client-supplied idempotency key derived from the saga ID and step index:

func idempotencyKey(sagaID string, stepIndex int, phase string) string {
    return fmt.Sprintf("%s:step%d:%s", sagaID, stepIndex, phase)
}

The downstream service stores this key with its result. On re-delivery, it returns the cached result without re-executing side effects. Without this, retrying a payment step after a network timeout may double-charge. Without this on compensations, retrying a refund may double-refund.

This means every participant service in a saga must implement idempotent endpoints—not as a nice-to-have but as a hard interface contract. Services that cannot provide this contract cannot safely participate in a saga.

The Failed Compensation: Your Actual Worst Case

If C3 fails persistently, you have a saga stuck in a partially compensated state. No automated path resolves this without human intervention or a separate remediation saga. Production systems need:

1. An alerting threshold on StepFailed transitions. Any saga reaching this state should page the on-call engineer.
2. A manual intervention API. The coordinator exposes an endpoint to force-advance a compensation step (mark it compensated without invoking the downstream service) or to force-abort the entire saga with an audit log entry. Access to this endpoint must be gated behind elevated authorization—it is a dangerous escape hatch.
3. Audit trails for every state transition. Compensation decisions made months after a failure need a full reconstruction of what happened and when.

In AWS-deployed Go services, persisting the saga state to DynamoDB (for single-digit millisecond reads) or MongoDB Atlas (for richer querying against saga history) and emitting state transition events to SQS gives you both the durability and the observability surface. The SQS consumer can drive alerting logic without coupling it to the coordinator hot path.

Pivot: Choreography vs. Orchestration and Where Rollback Gets Harder

Choreography-based sagas (services react to domain events, no central coordinator) make compensation harder to reason about. There is no single authority tracking which steps completed. Each service must publish a compensating event when it receives a rollback signal, and it must determine from its own state whether it was previously committed.

This sounds appealing from a coupling perspective, but the operational consequence is that diagnosing a stuck partial rollback requires correlating event streams across every participant—potentially across different teams, different Kafka topics, different retention windows. Orchestration places that complexity in one place (the coordinator), where it can be instrumented, queried, and operated against as a single artifact. For systems where rollback correctness is a hard business requirement, orchestration wins on operability.

Observability Dimensions

Minimum instrumentation for a production saga coordinator:

• Histogram of saga duration by terminal state. Sagas that succeed in 200ms vs. sagas that take 45 seconds reveal where retries are accumulating.
• Counter of compensation invocations per step, segmented by outcome. A spike in C3 failures before they resolve is a leading indicator of a downstream service issue.
• Gauge of sagas in non-terminal states older than threshold. Sagas stuck for more than 5 minutes in StepCompensating are candidates for alerting.

Label discipline matters here. Tag by saga type and step name, not by saga instance ID—that would explode cardinality.

Decision Framework

Use an orchestrated saga when: you need auditable rollback history, have more than three participants, or operate across team boundaries where choreography ownership becomes ambiguous.

Make every participant endpoint idempotent before wiring the saga. This is a precondition, not a follow-up task.

Persist coordinator state before acting, not after. The ordering is non-negotiable. Acting before persisting makes at-least-once delivery into at-most-once recovery.

Design compensation as a first-class operation with its own retry budget and failure terminal state. Compensations that silently time out are invisible partial failures.

Instrument the stuck-saga gauge and alert on it. A saga coordinator without this alert is a silent failure accumulator.

Prefer LIFO compensation order as an explicit invariant in code, not a convention. Derive the compensation sequence by reversing the list of StepCommitted steps at rollback time—do not maintain a separate compensation list that can drift from the execution list.

Sagas trade atomicity for availability. That trade is worth making in the right contexts. The cost is compensating transaction correctness as ongoing operational work, not a one-time design decision.