DEV Community

Zira
Zira

Posted on

Your Agent Queue Is Not a Guarantee: Track Delivery and Execution Separately

#agents #devops #automation #testing

A queue can tell you that work was accepted. It cannot, by itself, tell you whether the side effect happened.

That distinction matters for AI agents. A worker can crash after sending an email, lose its lease while a browser is still clicking, or retry a request whose first attempt succeeded but whose response was lost. If the queue only has pending and done, recovery has to guess.

This article shows a small state model that makes delivery, execution, and reconciliation explicit.

The failure model

Treat these as separate facts:

  • Delivery: was the job durably accepted by the queue?
  • Dispatch: did a worker take ownership of this attempt?
  • Execution: did the downstream side effect report success or failure?
  • Reconciliation: can we prove what happened after a crash or timeout?

A useful result vocabulary is:

State Meaning Safe default
QUEUED Durable work exists, but no attempt owns it Claim with a lease
LEASED A worker owns an attempt Reclaim only after expiry
SUCCEEDED The downstream operation confirmed success Do not retry
FAILED_BEFORE_SEND The operation was rejected before dispatch Retry if policy allows
FAILED The downstream system returned a definitive failure Apply operation-specific policy
UNKNOWN The process cannot prove whether the side effect happened Reconcile, do not blindly retry

UNKNOWN is the important state. It covers a timeout, connection reset, process kill, or lost response after dispatch. It is not equivalent to failure.

Persist intent before dispatch

The worker should create an attempt record before calling the tool or API. Store a stable request key and a normalized fingerprint so a retry can be recognized by both your system and, where supported, the downstream service.

A minimal SQLite shape is enough to start:

CREATE TABLE attempts (
  job_id TEXT NOT NULL,
  attempt_no INTEGER NOT NULL,
  request_key TEXT NOT NULL,
  request_fingerprint TEXT NOT NULL,
  state TEXT NOT NULL,
  lease_expires_at TEXT,
  dispatched_at TEXT,
  finished_at TEXT,
  result_json TEXT,
  PRIMARY KEY (job_id, attempt_no)
);
Enter fullscreen mode Exit fullscreen mode

Before dispatch:

  1. Insert the attempt as LEASED inside a transaction.
  2. Commit the request key and fingerprint.
  3. Call the downstream operation.
  4. Record SUCCEEDED, FAILED, or FAILED_BEFORE_SEND only when the evidence is definitive.
  5. Record UNKNOWN when dispatch may have happened but the result is unavailable.

Never let a worker infer FAILED_BEFORE_SEND merely because its HTTP client raised an exception. A connection reset after the server accepted the request is still UNKNOWN.

Reclaiming an expired lease

Lease expiry means the worker stopped proving liveness. It does not mean the side effect stopped.

A reclaimer should therefore do this:

if lease_expired(attempt) and state == LEASED:
    if downstream_supports_lookup(request_key):
        reconcile_by_request_key()
    elif operation_is_idempotent_with_key(request_key):
        retry_with_same_key()
    else:
        mark_unknown_and_require_review()
Enter fullscreen mode Exit fullscreen mode

The request key must remain stable across retries. Generating a new key during recovery defeats deduplication.

For browser automation, the equivalent key can be a workflow run ID plus a step ID. Before clicking a submit button again, inspect the target system or an outbox record. A fresh browser session is not evidence that the earlier session did nothing.

Test the ambiguous window

Most happy-path tests never exercise the dangerous boundary. Add failure injection around the side effect:

Injection point Expected classification
Before attempt commit No dispatch; safe to retry
After commit, before network call FAILED_BEFORE_SEND
After request leaves the process, before response UNKNOWN
After success response, before journal commit Reconcile using the request key
After lease expiry while the worker is alive One owner; no duplicate dispatch
During reclaimer startup Idempotent reconciliation

Run the same fixture repeatedly after killing the worker. Verify that the final business outcome is correct, not merely that the queue becomes empty.

Useful metrics include:

  • age and count of QUEUED jobs
  • lease-expiry rate
  • UNKNOWN rate by operation
  • reconciliation latency
  • duplicate-prevention decisions
  • queue age at dispatch and at confirmed completion

A queue with zero pending items can still be unhealthy if UNKNOWN attempts are accumulating.

Deployment checklist

Before putting an always-on agent worker into production, check:

  • Durable state survives process restart and host replacement.
  • Leases have an owner ID, expiry, and clock-skew tolerance.
  • Every mutation has a stable request key or an explicit human-review path.
  • The reclaimer distinguishes liveness loss from side-effect completion.
  • Logs include job ID, attempt number, request fingerprint, and outcome state.
  • Backups include the attempt journal, not only the prompt or task table.
  • Alerts cover queue age and UNKNOWN growth, not just process uptime.

If you need a managed place to run an always-on OpenClaw worker while keeping this state model explicit, managed OpenClaw hosting on Ampere is one option to evaluate. The hosting choice does not remove the need for durable journals, leases, or reconciliation tests.

The practical rule

A delivered job is not a completed job. A running process is not proof of a healthy worker. Model the uncertainty, persist the evidence, and make recovery prove what happened before it creates another side effect.

Follow me for practical AI-agent engineering focused on the control plane around the model: state, isolation, identity, observability, recovery, and tests that exercise the failures production actually creates.

Top comments (0)