An agent run can complete successfully while the message that reports it never arrives. The reverse is also possible: a retry delivers a message twice even though the tool call ran once.
That is not one failure. It is two state machines:
- Execution state: did the agent dispatch the tool, and did the provider confirm the outcome?
- Delivery state: did every downstream consumer receive the result, and did they acknowledge it?
Treating them as one status = done field creates the most dangerous recovery behavior: rerunning work because delivery is unknown.
The smallest useful state model
Keep an execution record and a delivery record linked by the same stable operation ID.
operation_id: op_01J...
execution: INTENT_RECORDED | DISPATCHED | SUCCEEDED | FAILED | UNKNOWN
result_ref: provider-request-123
delivery: PENDING | SENT | ACKED | DEAD_LETTERED
attempt: 2
The important value is UNKNOWN. It means the worker lost contact after dispatch and cannot safely infer whether the side effect happened. It is not a failure and it is not permission to retry blindly.
A compact relational shape is enough to start:
CREATE TABLE agent_operations (
operation_id TEXT PRIMARY KEY,
tool_name TEXT NOT NULL,
request_hash TEXT NOT NULL,
execution_state TEXT NOT NULL,
provider_ref TEXT,
result_ref TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE deliveries (
operation_id TEXT NOT NULL,
destination TEXT NOT NULL,
state TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT,
PRIMARY KEY (operation_id, destination)
);
Do not put a complete provider response in every queue message. Store a durable result reference, then make delivery retryable and independently observable.
Recovery rules
1. Retry delivery, not execution
If execution is SUCCEEDED and delivery is PENDING, replay the delivery using the existing result. Never call the tool again merely because a webhook timed out.
If execution is UNKNOWN, pause the operation and reconcile with the provider. Prefer a provider lookup keyed by provider_ref, or an idempotency-key lookup if the provider supports one. Only create a new execution attempt after policy explicitly says the original side effect is impossible or safely repeatable.
2. Make the consumer idempotent too
An idempotency key protects the producer, not necessarily the consumer. The receiver should persist the last accepted key before applying a non-idempotent change, or use a transaction/outbox pattern that makes the check and change atomic.
A useful contract is:
POST /events
Idempotency-Key: op_01J...
HTTP/1.1 202 Accepted
{ "operation_id": "op_01J...", "delivery": "accepted" }
The response acknowledges receipt, not successful execution. Keep those meanings separate in logs and dashboards.
3. Recheck policy during reconciliation
A request that was allowed at dispatch time may be forbidden by the time an unknown outcome is investigated. Reconciliation should re-evaluate:
- tenant and actor identity
- tool and destination permissions
- credential validity and scope
- expiration or cancellation
- whether repeating the side effect is safe
A successful provider lookup does not override a current policy decision. It only tells you what happened.
A failure-injection test you can run
Build a test harness with a fake tool that records every received idempotency key. Inject a process kill at each boundary:
| Boundary | Expected recovery |
|---|---|
| Before dispatch | No provider call; retry execution once |
| After dispatch, before response | Reconcile; do not blind-retry |
| After confirmed success, before result write | Recover provider result, then mark success |
| After result write, before delivery enqueue | Transactional outbox creates delivery |
| After delivery send, before ACK write | Replay with consumer deduplication |
| After policy revocation | Refuse new execution; preserve the audit trail |
The assertion is not just “the final status is green.” Count provider side effects and verify that one logical operation produces at most one non-idempotent change.
What to monitor
At minimum, graph these separately:
- execution age by state, especially
UNKNOWN - delivery age by destination
- reconciliation success and refusal counts
- duplicate keys seen by providers and consumers
- dead-letter volume
- time between execution confirmation and delivery acknowledgement
A healthy process heartbeat proves almost nothing about this state. A worker can be alive while its durable queue is stuck, its result store is unavailable, or its consumer acknowledgements are being dropped.
For always-on agents, the hosting choice matters because the runtime must preserve durable state and restart with the same identity boundaries. If you want to compare that operational surface with a managed option, see managed OpenClaw hosting on Ampere. The link is secondary to the design: verify state persistence, restart behavior, backups, credential scope, and rebuild steps before trusting any provider.
A practical acceptance checklist
Before calling an agent workflow reliable, answer yes to all of these:
- Can I distinguish execution success from delivery acknowledgement?
- Can I reconcile an interrupted tool call without guessing?
- Does every side-effecting operation have a stable idempotency key?
- Can the consumer deduplicate the same event?
- Are result records durable before a worker acknowledges the job?
- Does reconciliation recheck current authorization?
- Can I prove, from logs, whether a retry was execution or delivery?
- Have I killed the worker at every boundary in the table above?
The model may decide what to do, but the control plane must decide what is safe to repeat. That separation is what turns a crash from a mystery into a recoverable state transition.
If this kind of operational detail is useful, follow for more articles on agent runtimes, durable state, and failure testing.
Top comments (0)