A workflow can look perfectly transactional in an architecture diagram: validate the request, call an external system, save the result, and return success.
Reality is less tidy.
The process can stop after the external call but before the local save. A callback can arrive twice. A mutable rule can change between validation and completion. A timeout can leave the caller uncertain even though the external system accepted the operation.
A more useful design question is not merely, "What is the happy path?"
It is: Where does this workflow cross a boundary that a local rollback cannot undo?
Once that point is explicit, the surrounding design becomes much easier to reason about.
What "irreversible" really means
Irreversible does not mean that an action can never be reversed. A charge may be refunded. A message may be followed by a correction. A provisioned resource may be removed.
It means the local database transaction cannot erase what happened elsewhere.
A refund, cancellation, or compensating message is a new operation. It has its own permissions, timing, audit trail, and failure modes.
The same pattern appears in many systems:
- Charging through an external provider
- Sending a message to an external recipient
- Issuing a credential
- Dispatching physical work
- Publishing an event another system consumes
- Starting a long-running third-party process
After any of these actions, "roll back" is no longer a complete recovery strategy.
Validate while rejection is still cheap
Normal business rejection should happen before the irreversible call.
In a recent committed .NET change, an existing checkout gained a new non-inventory line type. That line had no stock, but it did have mutable eligibility rules and required downstream records after payment. The important design choice was the order of operations.
Before a wallet capture or external redirect, the server revalidated the current rules. A failed validation produced no order, no payment intent, no stock movement, and no charge.
For a general workflow, pre-flight checks may include:
- Confirming the caller may perform the operation
- Rechecking availability or eligibility
- Resolving the intended destination
- Recalculating totals and currency
- Verifying referenced records still exist
- Confirming the request matches the current business context
If these checks happen after the external action, a clean rejection becomes compensation or repair work.
This does not make validation infallible. Mutable facts can change immediately after they are checked. Where contention matters, use a reservation, version check, conditional write, or provider-side guarantee. The goal is to remove avoidable failure windows, not pretend that races have disappeared.
Persist enough context to replay
Before control leaves the system, persist a durable description of the intended operation.
That record might contain:
- A stable operation identifier
- Normalised amount and currency
- The intended destination
- A version or fingerprint of important inputs
- The current workflow state
- The key used to make the external action idempotent
The exact fields depend on the domain. The principle is that a later process should not need the original browser session, HTTP request, or in-memory object graph to understand what was intended.
A useful sequence is:
- Validate mutable rules.
- Persist a prepared operation with a stable key.
- Cross the irreversible boundary once.
- Receive or poll for authoritative confirmation.
- Complete the local transition through replay-safe logic.
If the process restarts after step three, confirmation still has durable context. If the caller retries after a timeout, the same business-operation key can be reused rather than creating another external action.
Treat duplicate callbacks as normal input
Callbacks travel over networks, so exactly-once delivery should not be assumed.
The sender may retry because a response was lost. Two application instances may receive overlapping deliveries. A delayed callback may arrive after a repair process has already completed the work.
A robust handler converges rather than repeats.
The stable key should identify one intended business operation, not one transport attempt. The handler should compare the confirmation with the stored expectation, perform one valid state transition, and make deeper side effects idempotent too.
The first valid callback completes the operation. A duplicate observes the same operation and produces the same effective outcome without allocating, notifying, or dispatching twice.
Idempotency only at the HTTP endpoint is not enough if deeper side effects can still duplicate.
Repair is part of the design
Careful sequencing still leaves crash windows.
The external action may succeed while confirmation is delayed. A downstream projection may fail after the primary outcome has committed. Validation may become stale between the check and the action.
That is why idempotency needs observability and a repair path.
A bounded repair process can find operations that remain uncertain, obtain authoritative status, and invoke the same idempotent completion logic used by the callback. It should be limited, safe to repeat, isolated per record, and conservative when evidence is incomplete.
In the reviewed change, duplicate delivery could safely re-drive the downstream fan-out, but a dedicated sweep remained follow-up work. That is an important distinction: replay safety is evidence of a good boundary, not proof that every failure will automatically heal.
The engineering trade-off
This design introduces more machinery: extra reads, durable intermediate state, idempotency keys, explicit transitions, callback tests, and reconciliation.
It may also expose uncomfortable states such as "prepared", "externally accepted", and "completion uncertain". Those states already existed. The design simply makes them visible.
The return is not perfect consistency. It is controlled uncertainty. Duplicate delivery becomes harmless, restarts become recoverable, and support teams can explain what happened without guessing from partial logs.
For an important workflow, ask:
- What is the first action our local transaction cannot undo?
- Which mutable rules can be checked before it?
- What context must survive a restart?
- Which stable key represents one business operation?
- Can every callback and downstream effect replay safely?
- How will uncertain operations be found and repaired?
Finding the point of no return is often the fastest way to expose hidden assumptions in an otherwise clean happy path.
Where is that boundary in the workflow you are reviewing now?
Top comments (0)