DEV Community: Karen Barseghyan

The Dual-Write Problem: Keeping Track of Unfinished Work

Karen Barseghyan — Wed, 09 Sep 2026 17:26:45 +0000

One business operation may need two writes. If they commit separately, the first can finish while the second is still pending. What should the system do with that unfinished state? This question has been a subject of research since the 1970s. It still appears whenever we update a database and send a message, move money between banks, or book a journey through several airlines.

Where exactly is the problem?

Suppose we transfer 10 units from account A to account B. Both accounts start with 100. Ignore fees and other transfers. Our program performs two writes: subtract 10 from A, then add 10 to B.

If the first write is rejected and makes no change, we have no half-completed transfer. That failure needs handling, but let us leave it aside. We are interested in what happens after the first write succeeds.

Point in the operation	A	B	What we have
Before the writes	100	100	No transfer yet
After the first write	90	100	An unfinished transfer
After the second write succeeds	90	110	A completed transfer

The second write may succeed or fail. In both cases, the operation passes through the middle row. If the second write succeeds, the unfinished state lasts for a while. If it fails, that state remains until we do something about it. A crash between the writes has the same practical consequence: somebody must finish or reverse the work.

What is wrong with the middle row? If our rule says that the two balances must always total 200, it violates that rule. Such a rule is called an invariant. It tells us what must remain true as the system works. A correct final row does not make an earlier violation disappear.

We therefore have a choice. We can prevent other operations from seeing the unfinished result, or we can explicitly allow transfers in progress and give that state a clear meaning. In the second design, the balances alone do not tell the whole story. The system must also account for the pending transfer. It must never present that transfer as completed.

Changing the order does not solve this. Crediting B first gives us 210 in the middle. Running the writes together still allows one to finish before the other. Making them faster reduces the delay, but leaves a place where the process can stop. This is the dual-write problem: one business operation has two outcomes that can become permanent separately.

Doesn't a database already handle this?

It does, within the guarantees it offers. If both account changes belong to one database transaction, the database can commit them together or roll them back together. That is atomicity. With suitable isolation, other transactions are also protected from treating our unfinished work as a completed result.

The same underlying difficulty exists inside the database. It writes log records, updates data pages, and may send changes to replicas. These steps do not happen at the same instant. A machine can stop between them. Write-ahead logging, or WAL, gives the database a durable log from which it can recover changes. Replication uses its own rules to decide which copies must acknowledge a change.

The database handles this internal work behind its API. Its transaction and consistency guarantees form the boundary we rely on. We do not have to recover each physical write ourselves. But we must know what that boundary promises: for example, a replica may still return older data if the database permits that.

Now add a call to another bank. Our database cannot roll back that bank's work just because our local transaction failed. We have crossed the boundary of the guarantee.

What solutions do we already have?

When a single transaction includes changes in several databases or other cooperating systems, it is a distributed transaction. The participants must follow one commit or abort decision. We can also choose to let steps commit separately and manage the unfinished work.

Two-phase commit, or 2PC, asks every participant to prepare first. Each promises that it can finish its part and wait for the decision. Once all agree, a coordinator records commit and tells them to finish. Otherwise, it aborts. The difficulty is waiting: if the coordinator disappears at the wrong moment, a prepared participant may not know which decision was made. It cannot safely guess.

Three-phase commit, or 3PC, adds a message between voting and committing. The coordinator tells participants that everyone voted yes. They record that fact and acknowledge it before the final commit decision. Survivors now have more information for recovery. This can let them continue after the coordinator fails, but only with suitable assumptions about delays and failures. It does not solve arbitrary network partitions, where working machines cannot reach each other.

Paxos lets several machines agree on a decision even if some fail. Progress needs a communicating majority. It can help preserve decisions without depending on one coordinator. It does not, by itself, turn two external API calls into one transaction. The systems performing the work still have to cooperate.

A saga accepts separate commits and defines what to do if the larger operation cannot finish. If the first flight is booked and the second is unavailable, cancel the first reservation. This is compensation. It is another operation, so it can also fail or cost money. A cancellation or refund does not make the original action disappear.

A transactional outbox handles the database-and-message case. Save the business change and the message to be sent in one local transaction. A separate worker publishes saved messages. A crash can delay publication, but the pending message remains available for recovery. The worker may send it more than once if it crashes after sending and before recording success.

These ideas can work together. An outbox can deliver work for a saga. A durable workflow can track both the original steps and their compensation. To understand that arrangement, we need to look at what the system remembers.

Why do third-party APIs make this harder?

An international transfer may involve banks that we do not control. A journey may require reservations from several airlines. These systems have their own records, delays, and failure handling. Their APIs may let us submit a request and ask for its status, but offer no way to join our transaction.

There is also a difference between failure and silence. If another bank rejects a transfer, we have a result. If our request times out, we may not. The bank might have completed it while its reply was lost. We must keep that uncertainty in our own record. Otherwise, a replacement worker may repeat a successful operation or abandon one that never happened.

So the practical question changes. We cannot make every external step happen together. Can we remember enough to recover the work safely?

What is a durable, checkpointed workflow?

A workflow describes the steps of an operation and the rules for moving between them. It becomes durable when the information needed to continue is saved in storage that survives the worker's restart. A checkpoint is saved progress. Together, they let another worker continue an operation after the original worker stops.

For our transfer, the record needs an ID, the amount, the two accounts, confirmed results, and the work still unresolved. We create it before starting external work. When a local account change and its progress record belong to the same database, we save them in one transaction. Otherwise, we could debit the account and forget that we owe the next step.

What should that record say? Consider a few possible states:

Saved state	What it means	What can happen next
`CREATED`	The request is saved; no debit is confirmed	Perform or establish the result of the debit
`DEBIT_CONFIRMED`	A was debited; B's credit is not confirmed	Establish whether credit was attempted, then continue safely
`CREDIT_UNKNOWN`	Credit was requested, but its result is unknown	Check the result or use a safe retry
`COMPLETED`	Both debit and credit are confirmed	Report completion
`REVERSAL_PENDING`	Credit is known not to have taken effect; A must be refunded	Perform and confirm the refund
`REVERSED`	The debit has been refunded	Report that the transfer did not complete

The state name is only part of the record. For example, DEBIT_CONFIRMED does not tell us whether a credit request was sent just before a crash. We also keep operation IDs and confirmed results. Where the record cannot settle what happened, recovery must check the external system or repeat the operation safely.

Saved states and rules for changing them form a persisted state machine. “Persisted” means saved beyond the life of a process. “State machine” means we have defined which changes are allowed. We cannot move to COMPLETED just because a worker reached the end of a method. We need confirmation of both writes. We cannot treat CREDIT_UNKNOWN as a rejection merely because a timer expired.

Why bother with these rules? Because the transfer may outlive the process that started it. A new worker should not need that process's memory to decide what happens next. It reads the saved facts, identifies the unresolved step, and follows the permitted transition. That is resumable processing.

Does every checkpoint prove that the external action happened? Only if we saved a confirmed result. Suppose B receives the money and our worker crashes before recording the reply. Its replacement sees no confirmed credit. That means we have not recorded success. It does not mean B was never credited.

We have found another dual write: perform the external action, then save its result. Saving the result first would create the opposite risk, a record of success for an action we never performed. A durable workflow preserves intent and recorded progress. It cannot supply a reply that never reached us.

This is the kind of uncertainty I discuss in my Two Generals article. An action may succeed while its confirmation is lost. Every workflow step that calls another system must allow for this possibility. Saving a checkpoint does not remove it.

This is why we need idempotency. Repeating the same logical request must not repeat its business effect. It needs support from the system performing that effect. A workflow engine cannot make an arbitrary banking API safe to call twice. Without that support, recovery may need a reliable status check or investigation before continuing.

The workflow also changes our business rules. We now permit a transfer to remain in progress. We still require every debit to have a recorded obligation to finish the transfer or resolve it another way. We still prohibit false completion. The accounting must represent the pending money correctly; a status field alone does not do that work.

How does the workflow keep moving safely?

A restart should not discard accepted work. With at-least-once processing, each accepted item gets processed, provided the system recovers and keeps working. Uncertain results can cause an item to be processed more than once. The workflow therefore needs safe repetition wherever recovery might repeat a step.

Repeated updates to the workflow itself also need care. Confirming a debit twice must not debit the account twice. Applying the same logical transition again should leave the already established result intact. This is an idempotent state transition. The state change and any local database effect it represents need to be protected together.

Now suppose an old worker wakes up and writes “credit pending” after another worker saved “credit confirmed.” We must reject that stale update. One common approach is to update the record only if it still has the version the worker originally read. The check and update happen together. If the version changed, the worker reads the current record before acting again.

That helps us preserve monotonic progress. Here “monotonic” describes what we know. Once a credit is confirmed, an older message cannot erase that fact. If money is later returned, we record the return as another fact. We keep the history of what happened. The account balance can rise or fall while our knowledge moves forward.

What stops two workers from trying to continue the same transfer? A claim assigns the work to one worker. A lease makes that claim expire unless renewed. If the worker crashes, another can take over after the lease expires. The claim must be acquired safely, and updates must check that the worker still owns the work.

A lease does not stop a paused worker from waking up late. That is why ownership checks and safe repetition still matter. The lease helps organize recovery; it does not prove that only one process can ever send a request.

Some transfers will still need reconciliation. We compare our record with the other bank's record and establish what happened. Some cases can be resolved automatically. Others require a person. The workflow must keep those cases visible and preserve enough information to investigate them. “Needs investigation” is a useful state when the alternative is pretending to know the answer.

What can the business safely rely on?

It can rely on the meaning of each state. “Accepted” means the request is durably saved. “In progress” means required work remains. “Completed” means the required results are confirmed. A timeout may justify “checking the result,” but it does not by itself justify “failed.” These labels affect what customers and other services do next.

Some views may update later. The transfer record might be complete while a reporting screen still shows the previous state. With eventual consistency, such copies can catch up after a delay. There must be a working process that carries the updates across. We must choose which screens may lag and which decisions need a current answer. A stale display is different from permanently forgotten work.

What can we achieve at most? We can preserve accepted work, retain confirmed facts, expose uncertainty, and resume when there is a safe next action. Completion still depends on the external systems and the recovery options they provide. If a bank neither tells us what happened nor allows a safe retry, more local checkpoints will not remove that limit.

We also need limits on unfinished work. If a bank stops responding, accepting transfers forever creates a growing backlog and financial exposure. Someone must own unresolved cases. The business needs deadlines for investigation and a point at which it stops accepting more work.

The useful question after any interruption is simple: what do we know, and what can we safely do next? A durable workflow gives that question a place to live. It keeps an unfinished operation from becoming a forgotten one.

Low Latency and “Fast Enough”

Karen Barseghyan — Tue, 01 Sep 2026 14:29:40 +0000

What is low latency?
Why is it different from domain to domain?
And what is “fast enough”?

Everybody wants their application to have low latency. This is obvious. The real question is: how low?

Latency is the time between an event and the result we care about. For a user-facing application, it might be the time between a click and a visible response. In a data-processing system, it might be the time between receiving an item and producing a result. In an electronic trading system, it might be the time between receiving market data and sending an order.

The goal is not always to achieve the lowest theoretically possible latency. The goal is to be fast enough for the business case.

To explore this topic, let's divide latency-sensitive flows into three groups. This is a simplified division intended to make the contrast clearer.

1. User experience

If we care about user experience, 100–200 ms is usually acceptable for an ordinary interaction. It is better to be in the 50–100 ms range, though. Once the complete interaction is below roughly 50 ms, reducing it further will usually not make a meaningful difference to most users.

Here we are talking about the time between the user's action and the visible result. It usually includes:

frontend processing;
network time between the frontend and backend;
backend processing;
network time between the backend and its data storage;
processing inside the data storage;
and the same network path in the reverse direction.

This is the simplest flow. In reality the backend might make requests to other backend services, which introduces more processing and network steps.

Just as a reminder:

1 second = 1,000 milliseconds (ms)
1 ms     = 1,000 microseconds (µs)
1 µs     = 1,000 nanoseconds (ns)

Therefore:

1 second = 1,000,000 µs
1 second = 1,000,000,000 ns

For this kind of system, being fast enough usually means that users do not feel that the application is making them wait. Optimizing a 150 ms interaction may improve the experience. Optimizing a 20 ms interaction to 10 ms probably will not change the business result.

2. Backend processes

There are also cases where user experience is not the direct goal. We may have a large distributed system with different kinds of communication, data storage, messaging, synchronous and asynchronous processing, scheduling, batching, and so on.

Here we usually look first at whether the system can continuously keep up with its workload.

For example, imagine a permanent data stream flowing into the system. Our total sustained processing capacity must be higher than the sustained incoming rate. To simplify: this means processing each item more quickly than the interval between incoming items.

Otherwise, we are in trouble.

Of course, we may have a buffer for temporary out-of-control situations. But no buffer can save a system whose processing capacity is constantly below the throughput of the incoming stream. The buffer can only postpone the problem. Eventually it fills and we may end up with system failure and data loss.

Here being fast enough means processing the expected workload continuously without creating a permanently growing queue.

3. Competing systems

The third case is different. Your system competes with other systems, and the business result depends directly on its performance.

Here latency has to be as low as practically possible. You may care about nanoseconds in certain parts of the system, microseconds in others and every avoidable millisecond across the complete flow.

You are fast enough if you are the fastest or at least one of the fastest systems competing for the same opportunity.

This is mostly relevant to certain areas of FinTech, such as latency-sensitive electronic trading and market making. If another system reacts to the same market event before yours, it may take the opportunity before your order arrives. In this case lower latency does not merely make the system feel better. It can directly change the business result.

So, what is low latency?

Low latency is not a specific number.

For one system, 100 ms may be excellent. For another, 5 ms may be dangerously slow. A third system may need removing 100 ns from a critical operation.

The domain changes the meaning of being late:

in a user-facing system, you are late if the user feels the delay
in a backend process, you are late if the system cannot keep up with the workload
in a competing system, you are late if someone else reaches the opportunity first

That is why the real goal is not simply to have “low latency”. The goal is to understand the deadline created by the business case and to be "fast enough" for it.

This is the relationship between low latency and being fast enough.

There is one more important point: a real system does not complete every operation in exactly the same amount of time. Therefore latency is not only a number. It is a distribution. That deserves a separate discussion.

Columnar Data Structure in Java

Karen Barseghyan — Mon, 31 Aug 2026 19:52:12 +0000

What if we need traditional object-oriented API and extra high performance for field-wise operations in one place?

I designed my CPS exactly having this in mind. The article is for describing the idea behind the library.

Imagine you have 10 million product objects, which have the following structure:

class Product {
String name;
int quantity;
int price;
}

You need to calculate the grand total value by summing quantity * price for every product. You would create an ArrayList<Product>, iterate on them one way or another and calculate the sum. And you would be right. Or you need the product with max quantity or lowest price.

Now imagine you are the CPU doing those operations. You would pick the object references from the backing array, go through them one by one, capture field values and then do the computation. In the best case you will cache more of those references. But you still need to chase them to get the data inside. This is called pointer chasing: following object references to reach data stored at other memory locations. It is because Java arrays do not store object data inline. Object arrays store references, while primitive arrays store primitive values directly.

Now imagine you have this class:

class ProductColumnarStore {
String[] names;
int[] quantities;
int[] prices;
int offset;
}

Each offset position represents one logical Product. Now your data is a good candidate for efficient CPU cache utilization during sequential access. And for primitive columns, no pointer chasing in the loop. Just loading new chunks of data from the arrays inside productColumnarStore. You have a contiguous data layout, which is especially good for sequential field-wise operations and SIMD instructions.

Why would you choose this way? It is not convenient to create this structure every time you need. You need to fill it first, then you need to handle bookkeeping of the offset during the iteration and you will not have the product object.

This is where CPS helps you. You design a projection based on your class, it is doing the heavy lifting.

interface ProductProjection {
String name();
int quantity();
int price();
}

Later I will show how and when to use this library.