Test a Saga When Compensation Times Out and the Message Is Delivered Twice

A saga test that stops after “payment succeeded, inventory failed, refund called” assumes compensation is reliable. It is another distributed operation, so test it under the same faults.

Use this sequence:

1. payment capture succeeds
2. inventory reservation times out
3. refund succeeds at provider
4. refund response is lost
5. compensation message is delivered again
6. late inventory-failed event arrives again

The invariant is not “refund endpoint called once.” It is:

captured amount - confirmed refunded amount = final charged amount
and final charged amount is never negative

Persist an inbox record for consumed message IDs and an outbox record for each intended side effect. Give the provider request a stable idempotency key derived from saga and compensation step:

{
  "sagaId": "order-42",
  "step": "refund-payment-v1",
  "idempotencyKey": "order-42:refund-payment:v1",
  "amount": 4900
}

A deterministic simulator should permute duplicate delivery, delayed acknowledgement, worker crash, and out-of-order events. After every run, assert one terminal order state, at most one economic refund, and a complete evidence trail. “Already refunded” must reconcile to success only after amount and payment identity match.

AWS describes coordination choices and rollback behavior in its Saga pattern guidance. The implementation detail that deserves its own test is durable compensation progress: a process restart cannot erase whether the external side effect occurred.

Alert on sagas stuck in compensating, but do not let an operator click “retry” with a new key. The runbook should first query provider state, compare amount and currency, and resume the same operation identity.

Exactly-once delivery is not required to preserve money. Stable operation identity plus reconciliation is.