Your AI Agent Needs a Circuit Breaker, Not Just Retries

Retries are useful when a request failed before the model or tool saw it. They are dangerous when the failure is ambiguous.

A long-running coding agent or OpenClaw worker can receive a timeout after the upstream endpoint accepted the request. Blindly retrying may duplicate a tool call, amplify an outage, or turn a small latency spike into a queue collapse.

The missing control is a circuit breaker with explicit outcome states and a bounded retry budget.

The failure model

Treat each model or tool request as one of four outcomes:

• SUCCEEDED: a valid response was received and recorded.
• FAILED_BEFORE_SEND: the request was rejected locally and was never dispatched.
• FAILED: the endpoint returned a definitive error before any side effect.
• UNKNOWN: the client cannot prove whether the remote endpoint accepted it.

Only the first three are safe to handle automatically. UNKNOWN needs reconciliation or operator policy, especially when a request can trigger a mutation.

This distinction also changes what a retry means. A retry of a pure model completion may be acceptable after UNKNOWN; a retry of deploy, send_email, or browser_click may not be.

A small state machine

Use three breaker states per upstream, not one global switch:

type BreakerState = "CLOSED" | "OPEN" | "HALF_OPEN"

interface UpstreamHealth {
  state: BreakerState
  consecutiveFailures: number
  openedAt?: number
  probeInFlight: boolean
}

• CLOSED: requests flow normally.
• OPEN: fail fast for a cooldown period. Do not spend model or tool budget adding more load.
• HALF_OPEN: allow one bounded probe. A successful probe closes the breaker; a failure reopens it.

Keep health separate for the model endpoint, browser service, and each high-risk tool provider. A broken screenshot service should not stop a text-only planning step, and a broken model endpoint should not cause the worker to hammer every dependency.

Classify errors before counting them

Do not increment the breaker for every exception. Count transport failures, connection resets, 408/429 responses, and 5xx responses according to your policy. Treat authentication failures, invalid schemas, and deterministic 4xx errors as configuration or code failures instead.

A useful event record is small and replayable:

upstream: local-model
request_id: run-42-step-7
attempt: 2
outcome: UNKNOWN
error_class: read_timeout
elapsed_ms: 30000
breaker_state: CLOSED
policy_version: llm-retry-v3

The policy version matters. When you change timeout or retry thresholds, you should be able to explain why two runs made different decisions.

Put the budget in one place

A retry loop inside every tool is how systems accidentally multiply attempts. Make the orchestrator own the total budget:

def call_with_budget(operation, *, max_attempts=3, deadline_s=90):
    started = monotonic()
    for attempt in range(1, max_attempts + 1):
        if monotonic() - started >= deadline_s:
            return Result("FAILED", "deadline_exhausted")

        if breaker_is_open(operation.upstream):
            return Result("FAILED_BEFORE_SEND", "breaker_open")

        mark_dispatch_intent(operation.request_id, attempt)
        result = dispatch(operation, timeout=per_attempt_timeout(attempt))

        if result.received_valid_response:
            record_success(operation.request_id, result)
            return Result("SUCCEEDED", result.value)

        if result.transport_ambiguous:
            record_unknown(operation.request_id, result.error)
            return Result("UNKNOWN", result.error)

        record_definitive_failure(operation.request_id, result.error)
        if not retryable(result.error):
            return Result("FAILED", result.error)

        sleep(backoff(attempt))

    return Result("FAILED", "attempt_budget_exhausted")

The important line is mark_dispatch_intent before sending. It gives reconciliation a durable record to inspect after a crash or timeout. For mutations, attach an idempotency key and require the remote side to expose a status lookup where possible.

Backoff is not a recovery strategy

Exponential backoff reduces synchronized load. It does not prove that the original request was harmless. Use jitter, cap the delay, and stop at a deadline:

wait = min(30, 0.5 * (2 ** (attempt - 1))) + random.uniform(0, 0.25)

For 429 responses, honor Retry-After when present. For a half-open probe, use a cheap read-only request rather than replaying the failed mutation.

Add a queue admission check

When the breaker is open, decide what happens to new work explicitly:

1. Reject low-priority tasks with a visible reason.
2. Queue only work with an expiry time.
3. Continue local, read-only steps that do not depend on the unhealthy upstream.
4. Require approval before replaying an UNKNOWN mutation.

A queue without an expiry becomes a second failure. Persist created_at, expires_at, priority, dependency, and replay policy for each item.

Test the states you actually fear

A happy-path unit test will not exercise this control plane. Inject failures at these boundaries:

• connection refused before bytes leave the worker;
• request accepted, response delayed past the client timeout;
• 429 with and without Retry-After;
• five failures followed by a successful half-open probe;
• worker crash after dispatch intent but before the response is journaled;
• breaker opening while a high-priority task is queued;
• two workers attempting the same half-open probe;
• schema-valid response with an application-level error;
• clock skew around cooldown and queue expiry.

For each fixture, assert the final outcome, number of remote attempts, breaker state, queue state, and audit record. A useful invariant is:

No automatic retry may occur for an UNKNOWN side effect unless the operation declares itself idempotent or reconciliation proves it was not applied.

A practical rollout checklist

Before putting an agent behind an always-on process supervisor, verify:

• each upstream has an independent breaker;
• attempts are bounded by both count and deadline;
• UNKNOWN is distinct from FAILED;
• dispatch intent is durable before sending;
• mutations have idempotency keys or a reconciliation endpoint;
• queued work expires and reports why it was dropped;
• half-open probes are serialized;
• logs include request ID, attempt, outcome, policy version, and breaker state;
• failure injection checks remote attempt count, not just local return values.

Retries are a transport convenience. A circuit breaker is an operational contract. The difference becomes visible when the worker restarts, the endpoint is overloaded, or a timeout hides whether the outside world saw the request.