DEV Community

Zira
Zira

Posted on

Your AI Agent Needs a Side-Effect Class, Not Just Max Attempts

#agents #devops #automation #testing

An agent retry loop is not a reliability policy.

It is only a counter.

The dangerous question is not “how many times should this tool call run?” It is:

If the previous attempt may have changed the world, what evidence lets us safely try again?

A model can retry a failed HTTP request, a shell command, a browser click, or an MCP call. Those actions do not share the same failure semantics. Treating them as interchangeable is how an agent sends duplicate emails, creates duplicate tickets, charges twice, or repeats a deployment after the first one already succeeded.

This article shows a small policy that makes retryability explicit.

Classify the side effect before dispatch

Give every tool operation a side-effect class:

Class Meaning Default after timeout
PURE No external mutation. Safe to repeat. Retry automatically
IDEMPOTENT Repeating with the same key converges to one result. Retry with the same key
REPLAYABLE Repeat is safe only after checking a durable operation record. Reconcile first
UNKNOWN The effect may have happened, but the caller cannot prove it. Stop and reconcile
NON_RETRYABLE Repeating can create an unacceptable new effect. Require approval

Do not infer this class from the tool name. A method called create_issue might be replayable if the provider supports an idempotency key, or non-retryable if it does not. A GET can still be unsafe if it triggers a workflow behind a badly designed endpoint.

Store the classification in the tool contract, not in the prompt:

{
  "tool": "create_deployment",
  "effect": "IDEMPOTENT",
  "idempotency_key": "required",
  "reconcile": "provider_lookup_by_key",
  "approval": "not_required"
}
Enter fullscreen mode Exit fullscreen mode

The executor should reject a dispatch that has no effect policy. “The model probably knows this is safe” is not a control.

Separate attempt count from effect evidence

A retry budget answers “how much work may we spend?” It does not answer “did the world change?” Keep both in durable state:

operation_id: op_01J...
attempt: 2
side_effect: IDEMPOTENT
idempotency_key: deploy:repo-a:commit-91c2
phase: DISPATCHED
provider_id: null
last_error: connection reset
Enter fullscreen mode Exit fullscreen mode

Useful phases are:

  1. PLANNED: the agent proposed an operation.
  2. AUTHORIZED: policy and credentials passed a fresh check.
  3. DISPATCHED: the request left your boundary.
  4. CONFIRMED: the provider returned a durable result.
  5. UNKNOWN: the process lost the response or timed out after dispatch.
  6. RECONCILED: a lookup proved the final state.

A process restart must not turn DISPATCHED into NOT_STARTED. If you already persist run ownership, use the same fencing approach described in run leases for AI agents so an old worker cannot resume an operation while the recovery worker is reconciling it.

Make the retry gate boring and deterministic

A retry decision should be explainable without asking the model:

def retry_decision(op, now):
    if op.phase == "CONFIRMED":
        return "STOP"
    if op.phase == "UNKNOWN":
        return "RECONCILE"
    if op.side_effect == "PURE" and op.attempt < 3:
        return "RETRY"
    if op.side_effect == "IDEMPOTENT" and op.attempt < 3:
        return "RETRY_SAME_KEY"
    if op.side_effect == "REPLAYABLE":
        return "RECONCILE"
    return "APPROVAL"
Enter fullscreen mode Exit fullscreen mode

The important detail is that the idempotency key remains stable. Never generate a new key merely because the worker restarted. A new key converts a retry into a new operation.

For providers without idempotency support, create your own operation record before dispatch and include a unique client token where the provider allows it. If neither is possible, classify the operation as NON_RETRYABLE or UNKNOWN after a lost response. Do not “try once more just in case.”

Test the failure window, not only the happy path

A useful harness kills the worker at each boundary:

  • before the request is built
  • after authorization but before dispatch
  • immediately after dispatch
  • after the provider commits but before the response is read
  • after the response is stored but before the agent continues
  • during reconciliation

For each injected crash, assert:

  • the operation has one stable identity
  • a stale worker cannot dispatch after recovery takes ownership
  • the retry gate chooses RECONCILE for ambiguous outcomes
  • the provider is queried by the original key or client token
  • no duplicate side effect is accepted as success
  • the final audit record explains why the operation was retried, stopped, or approved

A compact metric set helps expose policy drift: unknown_outcomes, reconciliations, duplicate_rejections, approval_interruptions, and retry_by_effect_class. A falling retry count is not automatically good. It may mean the agent is hiding failures or classifying everything as UNKNOWN.

Where hosting fits

If this worker must stay available for scheduled OpenClaw jobs or browser automation, a managed runtime such as always-on OpenClaw hosting on Ampere can reduce the operational work of keeping the process online. It does not decide whether a side effect is safe, preserve your operation ledger, or eliminate credential and prompt-injection risk. Those controls belong in the application and its recovery tests.

The practical rule is simple: retry computation freely, retry mutations only with evidence. Once an agent records the effect class, stable operation identity, and reconciliation path before dispatch, “retry” becomes a controlled state transition instead of a hopeful loop.

If you build agent runtimes, what side-effect class is hardest to verify in your system?

Top comments (0)