DEV Community

Zira
Zira

Posted on

Your Agent Crashed Mid-Tool Call. Can You Prove What Happened?

#testing #agents #devops

An agent that restarts cleanly is not necessarily an agent that recovers safely.

The dangerous case is an external tool call that is accepted by the tool but not acknowledged by the agent process. After a crash, a blind retry can send the email twice, create two tickets, charge twice, or rerun a deployment. A blind skip can lose work.

The fix is not “add more retries.” Give every side effect an interruption ledger and make recovery a reconciliation step.

The three states you need

Do not model a tool call as only success or failure. At minimum, record:

  • INTENT_RECORDED: the agent decided to call a tool and stored the normalized arguments.
  • DISPATCHED: the request was sent, with a provider or transport request ID if available.
  • OUTCOME_CONFIRMED: the tool returned a durable result that can be queried or verified later.

The crash window is between DISPATCHED and OUTCOME_CONFIRMED. That is not a failure state. It is UNKNOWN.

Here is a minimal SQLite schema:

CREATE TABLE tool_attempt (
  attempt_id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  tool_name TEXT NOT NULL,
  args_hash TEXT NOT NULL,
  idempotency_key TEXT NOT NULL UNIQUE,
  state TEXT NOT NULL CHECK (state IN (
    'INTENT_RECORDED', 'DISPATCHED', 'OUTCOME_CONFIRMED',
    'FAILED_RETRYABLE', 'UNKNOWN', 'RECONCILED'
  )),
  provider_request_id TEXT,
  result_ref TEXT,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);
Enter fullscreen mode Exit fullscreen mode

The unique idempotency key matters more than the retry loop. It should represent the logical side effect, not a randomly generated attempt. For example, invoice:tenant-7:2026-08-11:send is stable across a process restart.

Write the intent before dispatch

The safe order is:

  1. Normalize and validate tool arguments.
  2. Insert the intent and idempotency key in a transaction.
  3. Dispatch the tool request using that key.
  4. Persist the provider request ID immediately when returned.
  5. Confirm the outcome through the provider’s lookup API or a durable receipt.

Do not mark the row successful because the HTTP request returned 200. If the response was lost after the remote side effect happened, your local state is still UNKNOWN.

A worker loop can make that explicit:

INTENT_RECORDED -> DISPATCHED -> OUTCOME_CONFIRMED
                         |
                         +-> UNKNOWN -> reconcile -> CONFIRMED | FAILED | MANUAL_REVIEW
Enter fullscreen mode Exit fullscreen mode

For providers without idempotency support, the reconciliation policy should be more conservative. Query by a provider-side correlation field, compare a deterministic payload hash, or require manual review. Never “just retry” an ambiguous payment, deletion, message, or deployment.

Recovery after a crash

On startup, query attempts left in DISPATCHED or older in-flight states. For each row:

  • Provider says it exists and matches the hash: mark it confirmed.
  • Provider says it does not exist and the operation is idempotent: retry with the same key.
  • Provider is unavailable or the result is ambiguous: keep UNKNOWN and back off.
  • Arguments, policy version, or tenant changed: do not reuse the attempt. Create a new linked attempt and require review.

The last rule prevents a recovery worker from silently applying yesterday’s authority to today’s task.

A failure-injection test you can run

You do not need a large distributed test platform. Wrap the tool adapter with crash points:

A. kill after intent commit, before dispatch
B. kill after dispatch, before local acknowledgment
C. kill after provider response, before outcome commit
D. duplicate the worker message
E. change authorization while the attempt is UNKNOWN
F. make the provider lookup return a timeout
Enter fullscreen mode Exit fullscreen mode

For every case, assert:

  • no non-idempotent side effect occurs twice;
  • every attempt ends as CONFIRMED, FAILED, or MANUAL_REVIEW;
  • the operator can see the original arguments hash and request ID;
  • a retry never widens credentials or crosses a tenant boundary;
  • UNKNOWN work is visible in the queue and has an owner;
  • the recovery process is safe to run twice.

This is a better deployment gate than a green happy-path demo because it tests the boundary where process failure meets external reality.

Hosting is part of the recovery design

An always-on agent also needs a runtime where its ledger, state, and logs survive process replacement. If you want the server setup handled rather than assembling that control plane yourself, managed OpenClaw hosting on Ampere is one option to evaluate. The important question is still the same: can you inspect durable state, restart safely, and reconcile interrupted work?

The checklist

Before calling an agent production-ready, verify:

  • side effects have stable idempotency keys;
  • intent is durable before dispatch;
  • UNKNOWN is a first-class state;
  • recovery queries the external system before retrying;
  • policy and credential scope are rechecked on recovery;
  • backups include the ledger, not just configuration;
  • a forced crash test produces an auditable result.

Model quality determines what an agent may attempt. The interruption ledger determines whether a crash turns that attempt into a duplicate, a lost action, or a recoverable fact.

Top comments (0)