The Dual-Write Problem: Keeping Track of Unfinished Work

One business operation may need two writes. If they commit separately, the first can finish while the second is still pending. What should the system do with that unfinished state? This question has been a subject of research since the 1970s. It still appears whenever we update a database and send a message, move money between banks, or book a journey through several airlines.

Where exactly is the problem?

Suppose we transfer 10 units from account A to account B. Both accounts start with 100. Ignore fees and other transfers. Our program performs two writes: subtract 10 from A, then add 10 to B.

If the first write is rejected and makes no change, we have no half-completed transfer. That failure needs handling, but let us leave it aside. We are interested in what happens after the first write succeeds.

Point in the operation	A	B	What we have
Before the writes	100	100	No transfer yet
After the first write	90	100	An unfinished transfer
After the second write succeeds	90	110	A completed transfer

The second write may succeed or fail. In both cases, the operation passes through the middle row. If the second write succeeds, the unfinished state lasts for a while. If it fails, that state remains until we do something about it. A crash between the writes has the same practical consequence: somebody must finish or reverse the work.

What is wrong with the middle row? If our rule says that the two balances must always total 200, it violates that rule. Such a rule is called an invariant. It tells us what must remain true as the system works. A correct final row does not make an earlier violation disappear.

We therefore have a choice. We can prevent other operations from seeing the unfinished result, or we can explicitly allow transfers in progress and give that state a clear meaning. In the second design, the balances alone do not tell the whole story. The system must also account for the pending transfer. It must never present that transfer as completed.

Changing the order does not solve this. Crediting B first gives us 210 in the middle. Running the writes together still allows one to finish before the other. Making them faster reduces the delay, but leaves a place where the process can stop. This is the dual-write problem: one business operation has two outcomes that can become permanent separately.

Doesn't a database already handle this?

It does, within the guarantees it offers. If both account changes belong to one database transaction, the database can commit them together or roll them back together. That is atomicity. With suitable isolation, other transactions are also protected from treating our unfinished work as a completed result.

The same underlying difficulty exists inside the database. It writes log records, updates data pages, and may send changes to replicas. These steps do not happen at the same instant. A machine can stop between them. Write-ahead logging, or WAL, gives the database a durable log from which it can recover changes. Replication uses its own rules to decide which copies must acknowledge a change.

The database handles this internal work behind its API. Its transaction and consistency guarantees form the boundary we rely on. We do not have to recover each physical write ourselves. But we must know what that boundary promises: for example, a replica may still return older data if the database permits that.

Now add a call to another bank. Our database cannot roll back that bank's work just because our local transaction failed. We have crossed the boundary of the guarantee.

What solutions do we already have?

When a single transaction includes changes in several databases or other cooperating systems, it is a distributed transaction. The participants must follow one commit or abort decision. We can also choose to let steps commit separately and manage the unfinished work.

Two-phase commit, or 2PC, asks every participant to prepare first. Each promises that it can finish its part and wait for the decision. Once all agree, a coordinator records commit and tells them to finish. Otherwise, it aborts. The difficulty is waiting: if the coordinator disappears at the wrong moment, a prepared participant may not know which decision was made. It cannot safely guess.

Three-phase commit, or 3PC, adds a message between voting and committing. The coordinator tells participants that everyone voted yes. They record that fact and acknowledge it before the final commit decision. Survivors now have more information for recovery. This can let them continue after the coordinator fails, but only with suitable assumptions about delays and failures. It does not solve arbitrary network partitions, where working machines cannot reach each other.

Paxos lets several machines agree on a decision even if some fail. Progress needs a communicating majority. It can help preserve decisions without depending on one coordinator. It does not, by itself, turn two external API calls into one transaction. The systems performing the work still have to cooperate.

A saga accepts separate commits and defines what to do if the larger operation cannot finish. If the first flight is booked and the second is unavailable, cancel the first reservation. This is compensation. It is another operation, so it can also fail or cost money. A cancellation or refund does not make the original action disappear.

A transactional outbox handles the database-and-message case. Save the business change and the message to be sent in one local transaction. A separate worker publishes saved messages. A crash can delay publication, but the pending message remains available for recovery. The worker may send it more than once if it crashes after sending and before recording success.

These ideas can work together. An outbox can deliver work for a saga. A durable workflow can track both the original steps and their compensation. To understand that arrangement, we need to look at what the system remembers.

Why do third-party APIs make this harder?

An international transfer may involve banks that we do not control. A journey may require reservations from several airlines. These systems have their own records, delays, and failure handling. Their APIs may let us submit a request and ask for its status, but offer no way to join our transaction.

There is also a difference between failure and silence. If another bank rejects a transfer, we have a result. If our request times out, we may not. The bank might have completed it while its reply was lost. We must keep that uncertainty in our own record. Otherwise, a replacement worker may repeat a successful operation or abandon one that never happened.

So the practical question changes. We cannot make every external step happen together. Can we remember enough to recover the work safely?

What is a durable, checkpointed workflow?

A workflow describes the steps of an operation and the rules for moving between them. It becomes durable when the information needed to continue is saved in storage that survives the worker's restart. A checkpoint is saved progress. Together, they let another worker continue an operation after the original worker stops.

For our transfer, the record needs an ID, the amount, the two accounts, confirmed results, and the work still unresolved. We create it before starting external work. When a local account change and its progress record belong to the same database, we save them in one transaction. Otherwise, we could debit the account and forget that we owe the next step.

What should that record say? Consider a few possible states:

Saved state	What it means	What can happen next
CREATED	The request is saved; no debit is confirmed	Perform or establish the result of the debit
DEBIT_CONFIRMED	A was debited; B's credit is not confirmed	Establish whether credit was attempted, then continue safely
CREDIT_UNKNOWN	Credit was requested, but its result is unknown	Check the result or use a safe retry
COMPLETED	Both debit and credit are confirmed	Report completion
REVERSAL_PENDING	Credit is known not to have taken effect; A must be refunded	Perform and confirm the refund
REVERSED	The debit has been refunded	Report that the transfer did not complete

The state name is only part of the record. For example, DEBIT_CONFIRMED does not tell us whether a credit request was sent just before a crash. We also keep operation IDs and confirmed results. Where the record cannot settle what happened, recovery must check the external system or repeat the operation safely.

Saved states and rules for changing them form a persisted state machine. “Persisted” means saved beyond the life of a process. “State machine” means we have defined which changes are allowed. We cannot move to COMPLETED just because a worker reached the end of a method. We need confirmation of both writes. We cannot treat CREDIT_UNKNOWN as a rejection merely because a timer expired.

Why bother with these rules? Because the transfer may outlive the process that started it. A new worker should not need that process's memory to decide what happens next. It reads the saved facts, identifies the unresolved step, and follows the permitted transition. That is resumable processing.

Does every checkpoint prove that the external action happened? Only if we saved a confirmed result. Suppose B receives the money and our worker crashes before recording the reply. Its replacement sees no confirmed credit. That means we have not recorded success. It does not mean B was never credited.

We have found another dual write: perform the external action, then save its result. Saving the result first would create the opposite risk, a record of success for an action we never performed. A durable workflow preserves intent and recorded progress. It cannot supply a reply that never reached us.

This is the kind of uncertainty I discuss in my Two Generals article. An action may succeed while its confirmation is lost. Every workflow step that calls another system must allow for this possibility. Saving a checkpoint does not remove it.

This is why we need idempotency. Repeating the same logical request must not repeat its business effect. It needs support from the system performing that effect. A workflow engine cannot make an arbitrary banking API safe to call twice. Without that support, recovery may need a reliable status check or investigation before continuing.

The workflow also changes our business rules. We now permit a transfer to remain in progress. We still require every debit to have a recorded obligation to finish the transfer or resolve it another way. We still prohibit false completion. The accounting must represent the pending money correctly; a status field alone does not do that work.

How does the workflow keep moving safely?

A restart should not discard accepted work. With at-least-once processing, each accepted item gets processed, provided the system recovers and keeps working. Uncertain results can cause an item to be processed more than once. The workflow therefore needs safe repetition wherever recovery might repeat a step.

Repeated updates to the workflow itself also need care. Confirming a debit twice must not debit the account twice. Applying the same logical transition again should leave the already established result intact. This is an idempotent state transition. The state change and any local database effect it represents need to be protected together.

Now suppose an old worker wakes up and writes “credit pending” after another worker saved “credit confirmed.” We must reject that stale update. One common approach is to update the record only if it still has the version the worker originally read. The check and update happen together. If the version changed, the worker reads the current record before acting again.

That helps us preserve monotonic progress. Here “monotonic” describes what we know. Once a credit is confirmed, an older message cannot erase that fact. If money is later returned, we record the return as another fact. We keep the history of what happened. The account balance can rise or fall while our knowledge moves forward.

What stops two workers from trying to continue the same transfer? A claim assigns the work to one worker. A lease makes that claim expire unless renewed. If the worker crashes, another can take over after the lease expires. The claim must be acquired safely, and updates must check that the worker still owns the work.

A lease does not stop a paused worker from waking up late. That is why ownership checks and safe repetition still matter. The lease helps organize recovery; it does not prove that only one process can ever send a request.

Some transfers will still need reconciliation. We compare our record with the other bank's record and establish what happened. Some cases can be resolved automatically. Others require a person. The workflow must keep those cases visible and preserve enough information to investigate them. “Needs investigation” is a useful state when the alternative is pretending to know the answer.

What can the business safely rely on?

It can rely on the meaning of each state. “Accepted” means the request is durably saved. “In progress” means required work remains. “Completed” means the required results are confirmed. A timeout may justify “checking the result,” but it does not by itself justify “failed.” These labels affect what customers and other services do next.

Some views may update later. The transfer record might be complete while a reporting screen still shows the previous state. With eventual consistency, such copies can catch up after a delay. There must be a working process that carries the updates across. We must choose which screens may lag and which decisions need a current answer. A stale display is different from permanently forgotten work.

What can we achieve at most? We can preserve accepted work, retain confirmed facts, expose uncertainty, and resume when there is a safe next action. Completion still depends on the external systems and the recovery options they provide. If a bank neither tells us what happened nor allows a safe retry, more local checkpoints will not remove that limit.

We also need limits on unfinished work. If a bank stops responding, accepting transfers forever creates a growing backlog and financial exposure. Someone must own unresolved cases. The business needs deadlines for investigation and a point at which it stops accepting more work.

The useful question after any interruption is simple: what do we know, and what can we safely do next? A durable workflow gives that question a place to live. It keeps an unfinished operation from becoming a forgotten one.