DEV Community

Zira
Zira

Posted on

Your AI Agent Has Two Clocks. Reconcile Them Before It Duplicates Work

#devops #agents #automation #testing

Long-running agents often use time in two incompatible ways:

  • wall-clock time for leases, schedules, and expiry timestamps
  • monotonic time for measuring elapsed work inside one process

If those clocks are mixed, a restart or clock correction can make a live worker look expired, or make an expired lease look valid. The failure is not theoretical: it can relaunch a browser job, resend a mutation, or let two workers believe they own the same run.

This post shows a small design and a reproducible test plan.

The rule: persist instants, measure durations locally

Persist an absolute UTC instant for cross-process decisions. Use a monotonic clock only for elapsed measurements that never leave the process.

from datetime import datetime, timezone, timedelta
from time import monotonic


def now_utc() -> datetime:
    return datetime.now(timezone.utc)


def lease_expiry(ttl_seconds: int) -> datetime:
    return now_utc() + timedelta(seconds=ttl_seconds)


def expired(expires_at: datetime, now: datetime | None = None) -> bool:
    now = now or now_utc()
    return now >= expires_at


started = monotonic()
# measure local work with monotonic() - started
Enter fullscreen mode Exit fullscreen mode

Never persist a monotonic value as a lease deadline. Its origin is process-specific, so another worker cannot interpret it safely after a restart.

Make ownership a compare-and-swap, not a timestamp check

A worker should not claim a run because expires_at < now() was true in a prior read. Claim it with one conditional write that changes the lease token:

UPDATE runs
SET owner_id = :worker,
    lease_token = :new_token,
    lease_expires_at = :new_expiry,
    version = version + 1
WHERE run_id = :run
  AND status IN ('QUEUED', 'RUNNING')
  AND (lease_expires_at IS NULL OR lease_expires_at <= :now)
  AND version = :expected_version;
Enter fullscreen mode Exit fullscreen mode

Require exactly one affected row. Zero means another worker won, the run changed, or the clock input was stale. Do not continue to the browser or tool call after zero rows.

Every side effect should carry the lease token and a stable request key. A stale worker can then be rejected at the dispatch boundary even if it continues running briefly:

UPDATE side_effects
SET state = 'DISPATCHED'
WHERE request_key = :request_key
  AND state = 'INTENT_RECORDED'
  AND lease_token = :current_lease_token;
Enter fullscreen mode Exit fullscreen mode

This does not make an external API idempotent by itself. It prevents your own control plane from authorizing a stale owner twice. The destination still needs an idempotency key or a reconciliation path for ambiguous outcomes.

Restart reconciliation

On startup, do not blindly resume every row marked RUNNING. Reconcile in this order:

  1. Read the run and its current version.
  2. Compare the persisted UTC expiry against a trusted current UTC value.
  3. Check whether a dispatch intent exists for the next side effect.
  4. If the last result is UNKNOWN, reconcile with the destination before retrying.
  5. Otherwise, atomically acquire a fresh lease with a new token.
  6. Resume only from the last durable checkpoint.

A useful state split is:

QUEUED -> LEASED -> RUNNING -> CHECKPOINTED -> SUCCEEDED
                         |                       |
                         +-> UNKNOWN -----------+
                         +-> FAILED
Enter fullscreen mode Exit fullscreen mode

UNKNOWN is deliberately not FAILED. A process can crash after the remote system accepted a request but before your journal was updated. Retrying from UNKNOWN without reconciliation is how a restart becomes a duplicate side effect.

Five failure fixtures

Build these into a disposable test harness, not a production incident:

  1. Move wall time forward past the lease while the worker is paused. A stale worker must fail the dispatch guard.
  2. Move wall time backward during a renewal. The worker must not extend a lease using an unchecked local timestamp.
  3. Kill the process after writing intent but before sending. Recovery should send once.
  4. Kill it after the remote side effect but before recording the response. Recovery must classify UNKNOWN and reconcile.
  5. Start two workers against the same version. Exactly one conditional claim may succeed.

Record run_id, version, lease_token, request_key, persisted timestamps, and the outcome of every compare-and-swap. Without those fields, a duplicate is difficult to explain after the fact.

Operator checklist

Before calling an agent runtime restart-safe, verify that it can answer:

  • Which worker owns this run now?
  • When does that lease expire in UTC?
  • Which version did the worker claim?
  • What was the last durable checkpoint?
  • Was the next side effect only planned, or was it dispatched?
  • If dispatch is UNKNOWN, what evidence reconciles it?
  • Can an old worker still pass the dispatch guard?

For always-on browser or OpenClaw deployments, place the runtime where its durable database, browser-profile policy, and restart procedure can be tested together. A managed option such as always-on OpenClaw hosting on Ampere may be relevant when you need a persistent runtime surface, but the lease and reconciliation checks remain your responsibility.

The goal is not to eliminate every crash. It is to make a crash produce a bounded, explainable state transition instead of a second payment, duplicate message, or conflicting browser session.

If you build agent runtimes, follow for practical tests and control-plane patterns that turn “it usually works” into evidence you can inspect.

Top comments (0)