Your Agent Queue Is Not a Workflow Until It Tracks Execution State

An AI agent can receive the same job twice even when your queue is behaving correctly.

A worker may acknowledge a message after starting a browser action. The process can crash before recording the result. A visibility timeout can expire while the first worker is still running. A retry then starts a second worker, and both may send an email, place an order, rotate a credential, or publish a post.

The queue delivered a message. It did not prove that the side effect happened exactly once.

This distinction matters for coding agents and OpenClaw-style automations because the model is usually the least deterministic component in the loop. The surrounding runtime needs an explicit contract for what is queued, what is executing, what is externally visible, and what is still unknown.

Start with two ledgers, not one status field

A single status = done column cannot explain a crash between an external side effect and your database update. Track delivery and execution separately:

Concern	Example states	Question answered
Delivery	queued, leased, acknowledged, dead-lettered	Did a worker receive this attempt?
Execution	not_started, running, succeeded, failed, unknown	What do we know about the work itself?
Side effect	not_attempted, submitted, confirmed, ambiguous	What can we prove about the external system?

A minimal record can look like this:

CREATE TABLE job_runs (
  job_id TEXT NOT NULL,
  attempt INTEGER NOT NULL,
  delivery_state TEXT NOT NULL,
  execution_state TEXT NOT NULL,
  side_effect_state TEXT NOT NULL,
  idempotency_key TEXT NOT NULL,
  lease_expires_at TIMESTAMP,
  result_ref TEXT,
  updated_at TIMESTAMP NOT NULL,
  PRIMARY KEY (job_id, attempt)
);

The important state is unknown. It is not a nicer spelling of failure. It means the worker lost the ability to observe whether the external action completed.

Make the side effect idempotent before making the worker clever

Retries are normal. Design the effect so a retry can safely use the same key:

1. Derive one stable idempotency key from the logical job, not from the attempt number.
2. Persist the key before calling the external service.
3. Send the key to the service when its API supports idempotency.
4. If the service has no idempotency support, create a reconciliation record and make the action conditional where possible.
5. Never treat a timeout as proof that nothing happened.

For an API that supports an idempotency header, the request should use the same value across retries:

KEY="invoice-$JOB_ID"
curl -X POST https://billing.example/invoices \\
  -H "Idempotency-Key: $KEY" \\
  -H 'Content-Type: application/json' \\
  -d @invoice.json

If the first request times out, the retry asks the provider for the result associated with invoice-$JOB_ID, rather than creating a second invoice.

Use leases as crash detection, not as cancellation

A lease says “this worker currently owns an attempt until time T.” It does not say the work stopped when T passed.

A safe worker loop is:

claim job -> write running + lease
perform bounded step
renew lease only while ownership is valid
record confirmed result, or mark unknown on observation loss
reconcile unknown before retrying side effects

Every external call should check whether the lease is still valid before starting. After the call, write the result using a compare-and-set condition on the lease owner. A late worker must not overwrite the result of a newer attempt.

For browser automation, add a second identity check. The browser profile, account, and target should be bound to the run record. If a recovered worker finds a different profile or an expired login, stop and mark the run unknown or blocked; do not “try once more” against an unverified identity.

Reconcile unknown work explicitly

When execution is unknown, recovery needs a read path. Examples:

• Query the provider by idempotency key.
• Look up a sent-message ID in the mail provider.
• Inspect the target system for a unique job marker.
• Compare the last durable checkpoint with the external audit log.
• Ask for human approval when the action is irreversible and cannot be queried.

A reconciliation result should be one of:

• confirmed: the effect exists, so do not replay it.
• not_found: replay is safe under the action’s conditions.
• conflict: stop and escalate; the evidence disagrees.
• unavailable: keep the run unknown and retry reconciliation later.

This is also why a durable state store matters for an always-on agent. If the process, host, or container is replaced, the new worker must recover the ledger, leases, keys, and reconciliation queue. For teams that do not want to operate that layer themselves, managed OpenClaw hosting on Ampere can be a relevant deployment option when the requirement is an always-on runtime with operational recovery handled outside the agent prompt. It does not remove the need for idempotency or reconciliation.

For the related authorization boundary, see Your AI Agent Needs a Tenant Fence at Every Hop. Tenant identity and execution identity solve different problems: one prevents cross-tenant work, while the other prevents uncertain work from being replayed blindly.

Run this failure-injection test

Before trusting a workflow, inject failures at each boundary:

Injection point	Expected durable result	Safe recovery
Crash before external call	running or not_started	Reclaim after lease expiry
Crash during external call	unknown	Reconcile by key or marker
Crash after external success, before local write	unknown	Confirm externally, then close
Duplicate delivery	second attempt	Same idempotency key, no duplicate effect
Lease expiry with slow first worker	stale owner	Compare-and-set rejects late write
State-store restore	recovered ledger	Resume reconciliation before new side effects

Test the crash points with a real process kill, not only a mocked exception. A mocked exception usually skips the timing window that causes the production duplicate.

The practical invariant

The invariant I want is simple:

No retry may create an unbounded new side effect while the previous attempt is unknown.

That invariant pushes the complexity to the right places: stable keys, durable state, bounded leases, external lookup, and an explicit human stop for irreconcilable actions. The model can still plan the work, but the runtime decides whether the work is safe to start again.

If your agent has only a queue depth metric and a green worker count, you are measuring delivery capacity, not workflow correctness. Add execution state, side-effect state, and an unknown-work queue before adding more parallel workers.