Two AI Agents Read One Row. How to Stop the Lost Update

A version check is not automatically optimistic concurrency control.

If an agent reads a row, reasons for a few seconds, and then writes a decision, another worker can change that row in the read-to-write gap. The final UPDATE may still return success while silently overwriting newer state.

This is especially easy to miss in agent workflows because the reasoning step is slow, retries are common, and the write often looks like an ordinary CRUD operation.

This post builds a small failure lab and turns the race into a testable contract.

The bug: the check happened too early

A tempting implementation is:

grep -n "version" worker.py

1. Read the row and its version.
2. Check that the version is acceptable.
3. Ask the model what to do.
4. Update the row later.

The check in step 2 says nothing about whether the row is still current in step 4. Two workers can both read version 7, both produce valid decisions, and both write. The second write wins even though its input was stale.

The failure is not necessarily a database error. It is a successful write based on an invalid snapshot.

Make freshness part of the write predicate

Use a monotonic version, and require the version observed by the worker to match the version being replaced:

action = db.execute(
    """
    UPDATE agent_tasks
       SET status = :new_status,
           decision = :decision,
           version = version + 1,
           updated_at = CURRENT_TIMESTAMP
     WHERE task_id = :task_id
       AND version = :expected_version
       AND status = 'ready'
    """,
    {
        "task_id": task_id,
        "expected_version": observed_version,
        "new_status": new_status,
        "decision": decision,
    },
)

if action.rowcount != 1:
    raise StalePlan("the row changed after the agent read it")

The important property is that the read and the model call do not need to be inside one long database transaction. The final compare-and-set write is the fence.

A zero-row update is not a generic retry signal. It means the plan was produced from a snapshot that no longer owns the right to change the row.

Keep the plan tied to the snapshot

Do not pass only a task ID into the write path. Persist the inputs that made the decision:

CREATE TABLE agent_plans (
  plan_id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  observed_version INTEGER NOT NULL,
  input_hash TEXT NOT NULL,
  decision_json TEXT NOT NULL,
  state TEXT NOT NULL CHECK (state IN ('proposed', 'applied', 'stale', 'unknown')),
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The input hash should cover the normalized state the model actually saw, not just the database primary key. If the prompt included policy, tool availability, or a selected artifact, include their versions too.

At apply time, verify both the row version and the plan state. A plan that was already applied must not be applied a second time, even if a client retries after a timeout.

A minimal failure-injection test

You can reproduce the race without an LLM. Replace the reasoning step with a barrier so both workers pause after reading the same version:

async def worker(barrier, task_id):
    row = await read_task(task_id)
    await barrier.wait()  # both workers now hold the same snapshot
    decision = {"owner": current_worker_id()}
    return await apply_plan(
        task_id=task_id,
        observed_version=row.version,
        decision=decision,
    )

The expected result is exactly one successful apply and one stale-plan result. If both calls report success, the write fence is missing or the test is not exercising the real update path.

Run the lab with these cases:

Case	Expected result
Two workers read version 7	One applies; one is stale
Worker retries the same plan	No second business effect
Row changes while the model is thinking	Apply is rejected
Policy version changes	Plan is rejected or re-planned
Database response is lost after commit	Outcome becomes UNKNOWN and is reconciled
Worker restarts after apply	Recovery does not apply the plan again

Treat timeout after write as UNKNOWN

A client timeout does not prove that the UPDATE failed. The database may have committed before the connection broke.

Give every apply attempt a stable effect key, such as task_id plus plan_id, and record it in a durable effect ledger. On a timeout:

1. Look up the effect key.
2. Compare the stored plan and row version.
3. Mark the outcome applied, stale, or UNKNOWN.
4. Reconcile UNKNOWN before allowing another attempt.

Never solve an ambiguous write by blindly rerunning the model and applying its new answer. That can turn one uncertain side effect into two different decisions.

What to monitor

A useful dashboard separates these signals:

• stale-plan rejects, by tool and workflow;
• plans that waited longer than their snapshot budget;
• duplicate apply attempts by effect key;
• UNKNOWN outcomes and reconciliation age;
• policy or tool-contract changes between read and apply;
• successful writes that had no matching plan record.

The last metric should be zero. If it is not, some path can mutate agent state without the same evidence and fencing rules.

Checklist

Before trusting an agent that writes shared state, verify:

• The model decision carries the exact version and policy inputs it observed.
• The final write uses an atomic compare-and-set predicate.
• A zero-row update becomes a typed stale-plan result, not a generic retry.
• Every business effect has a stable idempotency key.
• Lost responses are reconciled instead of replayed blindly.
• A barrier test proves two concurrent readers cannot both apply version 7.
• Restarts and partial failures are included in the test matrix.

A version column is useful, but the real boundary is the version check at the write. Without that fence, an agent can be perfectly correct about an old snapshot and still corrupt the current one.