Your AI Agent Restarted Cleanly. Why Did It Run the Job Twice?

A clean process restart is not the same thing as a safe job restart.

An agent can recover its queue, reload its state, and report healthy while still replaying a side effect that already happened before the crash. The dangerous gap is usually between “the worker sent the request” and “the durable state store recorded the result.”

This article turns that gap into a small, testable protocol.

The false promise of exactly once

Most agent runtimes can usually provide one of these, not all three:

• At-least-once execution: a pending item is retried when its result is missing.
• At-most-once execution: a worker marks the item before acting, so it may be lost after a crash.
• Exactly-once business effect: the external system deduplicates a stable operation key.

The third property is not something a process restart can create by itself. It requires cooperation from the side-effect boundary.

Treat every externally visible operation as a state machine:

task_created -> effect_reserved -> effect_dispatched -> outcome_recorded

If the worker dies between effect_dispatched and outcome_recorded, the state is UNKNOWN, not FAILED. Retrying blindly is how duplicate emails, payments, tickets, browser mutations, or deployments happen.

Put the idempotency key at the boundary

Generate one stable key when the logical operation is created. Do not generate a new key for every retry.

from dataclasses import dataclass
from uuid import UUID

@dataclass(frozen=True)
class Effect:
    operation_id: UUID
    kind: str
    target: str
    payload_hash: str

    @property
    def idempotency_key(self) -> str:
        return f"agent:{self.operation_id}:{self.kind}:{self.target}"


def dispatch(effect: Effect, provider):
    # The provider must atomically return the existing result for a reused key.
    return provider.execute(
        target=effect.target,
        payload_hash=effect.payload_hash,
        idempotency_key=effect.idempotency_key,
    )

The provider contract matters more than the client helper. Verify at least these cases:

1. The same key and same payload return the original result.
2. The same key with a different payload is rejected.
3. A timed-out request can be looked up by key.
4. A key cannot be reused for a different target or operation type.
5. Retention lasts longer than the maximum retry and reconciliation window.

If the provider has no idempotency support, keep a durable effect ledger and add a reconciliation adapter. A ledger alone cannot undo a side effect that happened before the crash, but it can stop safe replays and route ambiguous ones for lookup or approval.

Separate recovery decisions from worker liveness

After restart, the worker should not ask only “is this job incomplete?” It should ask “what is the last authoritative fact about this effect?”

A useful recovery table looks like this:

Last durable state	Provider lookup	Recovery action
RESERVED	no dispatch record	dispatch with the original key
DISPATCHED	confirmed result	record the result, do not replay
DISPATCHED	not found	retry only if the provider guarantees lookup completeness
UNKNOWN	confirmed result	reconcile and continue
UNKNOWN	ambiguous or unavailable	keep UNKNOWN and require policy/approval

A fresh heartbeat is not evidence that the effect completed. Likewise, an old lease is not evidence that the effect is safe to replay. Recovery needs a fencing token or ownership lease so a paused pre-crash worker cannot write a conflicting result after a replacement worker takes over.

For a broader work-path probe, see why an AI agent health check should probe the work path, but keep health semantics separate from effect reconciliation.

A restart test you can run locally

Build a fake provider that persists results by idempotency key, then inject a crash at each transition:

1. Create operation with key K.
2. Reserve K in the local ledger.
3. Send K to the fake provider.
4. Kill the worker before recording the provider result.
5. Restart a replacement worker.
6. Reconcile K by lookup.
7. Assert that the provider observed one logical effect, not two.

Repeat the test with:

• the provider returning a timeout after accepting K;
• the local ledger unavailable during recovery;
• a stale worker resuming with an old fencing token;
• a duplicate message arriving from the queue;
• a payload mismatch under the same key;
• the provider lookup API temporarily unavailable;
• a human approval required for an UNKNOWN browser or payment action.

Record both effect attempts and confirmed business effects. Counting only successful worker runs hides the exact failure mode you are trying to prevent.

Hosting does not solve the protocol

An always-on runtime can reduce process churn, but it does not define idempotency, provider lookup, lease fencing, or UNKNOWN handling. If you need a managed place to run an OpenClaw or browser-enabled worker continuously, managed OpenClaw hosting on Ampere is one option to evaluate. The operational contract still belongs in your application and external-effect providers.

Production checklist

Before calling an agent restart-safe, verify:

• Every logical effect gets one stable idempotency key.
• The key binds operation type, target, and payload identity.
• Reuse with a different payload is rejected.
• UNKNOWN is a first-class state, not an automatic retry.
• Provider lookup is possible after client timeouts.
• Replacement workers fence stale workers.
• Key retention covers the full replay and reconciliation window.
• Crash injection runs between dispatch and local outcome recording.
• Metrics distinguish attempts, provider effects, reconciliations, and unresolved UNKNOWN cases.

A worker that restarts without losing its queue is only recovering execution state. A trustworthy agent also recovers the truth about what happened outside the process.