DEV Community

Zira
Zira

Posted on

Your AI Agent Needs a Run Lease, Not Just a Timeout

#agents #devops #automation #testing

An agent worker can be alive, connected, and still be the wrong process to execute a run.

That happens when a worker pauses during a model call, loses its network connection, or gets frozen by a host restart. The scheduler notices the timeout and starts a replacement. Then the old worker wakes up and continues with the same authority.

Now two workers believe they own one run.

A timeout detects suspicion. It does not transfer ownership.

The missing primitive is a run lease: a short-lived, renewable ownership record that every side-effecting step must present and that a replacement worker can fence.

The lease contract

Store one lease per run, not one global worker heartbeat:

type RunLease = {
  runId: string
  ownerId: string
  fencingToken: number
  expiresAt: string
  lastRenewedAt: string
}
Enter fullscreen mode Exit fullscreen mode

The important field is fencingToken. It increases every time ownership changes. A worker with token 7 must not be able to perform a side effect after token 8 has been issued to a replacement.

A lease should answer four questions:

  1. Who owns this run now?
  2. When does that ownership expire?
  3. Which token proves the current ownership generation?
  4. What happens when the answer is unknown?

Do not let a model response, in-memory boolean, or process ID answer those questions. They disappear or become ambiguous during failover.

Renew before you execute

A worker should renew the lease, then validate it again immediately before every irreversible operation:

def execute_step(run_id, owner_id, token, action):
    lease = store.read_lease(run_id)

    if lease.owner_id != owner_id:
        raise LostLease('owner changed')
    if lease.fencing_token != token:
        raise LostLease('fencing token changed')
    if lease.expires_at <= utc_now():
        raise LostLease('lease expired')

    return side_effect_store.apply(
        action,
        run_id=run_id,
        fencing_token=token,
    )
Enter fullscreen mode Exit fullscreen mode

The side-effect store must enforce the token too. Checking only in the worker leaves a race between the check and the write. The database transaction, job queue, browser-session broker, or API gateway that accepts the action needs to reject stale tokens.

That is the fence. A stale worker may still be running, but it no longer has authority.

Separate liveness from ownership

A process heartbeat answers: “Is this process responding?”

A run lease answers: “Is this process still authorized to mutate this run?”

They are related but not interchangeable. A process can pass its heartbeat while its lease is expired. A busy worker can miss a heartbeat while still holding a valid lease. Your scheduler needs separate states:

  • PROCESS_ALIVE
  • LEASE_VALID
  • LEASE_LOST
  • OUTCOME_UNKNOWN
  • RECONCILIATION_REQUIRED

If a lease is lost during a tool call, do not blindly retry the tool. The provider may have accepted the request even if the worker never received the response. Record OUTCOME_UNKNOWN, query the provider with a stable idempotency key where possible, and only then decide whether to retry, compensate, or ask for approval.

A minimal failure-injection test

You can test the dangerous race without a large distributed system:

  1. Start worker A with lease token 1.
  2. Pause A immediately before a side effect.
  3. Let the lease expire.
  4. Start worker B and assign token 2.
  5. Let B perform the side effect.
  6. Resume A and make it attempt the same side effect.
  7. Verify that the side-effect store rejects token 1.
  8. Verify that the run has one final ownership record and one reconciliation record.

Repeat the test with a delayed network response, a process restart, and a duplicate request. The expected result is not “the old worker stopped.” You cannot reliably guarantee that. The expected result is “the old worker could not mutate state after fencing.”

Useful evidence includes:

  • lease acquisition and renewal timestamps
  • owner and fencing token on every side effect
  • rejection count for stale tokens
  • time spent in OUTCOME_UNKNOWN
  • reconciliation decisions and their operator or policy source

Where hosting fits

An always-on agent runtime makes lease renewal and durable state easier to operate, but it does not create the safety property for you. If you run OpenClaw or another worker on managed infrastructure, keep the lease store and side-effect boundary explicit, test restart behavior, and verify what survives a rebuild.

For teams that do not want to maintain the base always-on host, managed OpenClaw hosting on Ampere is one option to evaluate. The important question is still architectural: can your worker prove current ownership, and can the downstream system reject stale ownership?

Run-lease checklist

Before calling an agent workflow failover-safe, verify:

  • ownership is stored durably per run
  • ownership changes issue a monotonically increasing fencing token
  • every irreversible step carries that token
  • the downstream side-effect boundary rejects stale tokens atomically
  • lease loss becomes an explicit state, not a generic retry
  • ambiguous tool outcomes enter reconciliation
  • restart and duplicate-request races are failure-injected regularly
  • rebuild documentation explains how leases, state, and credentials are restored

A timeout tells you that a worker may be gone. A lease plus a fencing token tells every other component whether that worker is still allowed to act. That distinction is what keeps a restart from becoming a duplicate side effect.

Top comments (0)