DEV Community

Zira
Zira

Posted on

Your AI Agent Is Restarting in a Loop. Stop Treating It Like a Health Check

A process that is alive is not necessarily an agent that is healthy.

A supervisor can report “running” while the worker is crashing before it registers, restoring a corrupt state file, losing its credential lease, or replaying the same outbound action after every restart. Restarting harder does not fix those failure domains. It can make them worse by hiding the first useful error and multiplying side effects.

This article gives you a small restart-loop protocol: classify the failure, preserve the evidence, quarantine unsafe work, and only then decide whether to restart.

1. Separate process liveness from agent readiness

Use at least three states:

  • PROCESS_UP: the process accepts a local health request.
  • READY: configuration, durable state, credentials, and dependencies passed startup checks.
  • EXECUTING: the agent is allowed to claim new work.

Do not use a single /healthz endpoint for all three. A process can be up while its state database is locked or its credential lease has expired.

A useful readiness response should identify the failed domain without exposing secrets:

domain              status    evidence
runtime             ok        pid=1842, build=2026.08.13
state               fail      sqlite_busy_after=3s
credentials         unknown   lease_version=41
outbound_delivery   paused    pending=2, unknown=1
admission           blocked   reason=readiness_failed
Enter fullscreen mode Exit fullscreen mode

The important part is the last line. A failed readiness check should stop new work, not necessarily kill the process.

2. Preserve the first failure, not just the last restart

Supervisors often produce a misleading story:

  1. start worker
  2. worker fails during recovery
  3. supervisor restarts worker
  4. logs rotate or the same error is buried
  5. repeat

Record a restart episode before attempting another start. Give it an ID and attach:

  • boot ID and process ID
  • config hash and state-schema version
  • last durable run position
  • credential version, never the credential value
  • dependency readiness results
  • the first exception and its timestamp
  • whether a tool call or delivery was UNKNOWN at shutdown

A compact event model is enough:

auto_restart_episode(episode_id, boot_id, config_hash, state_position, readiness_failures, first_error, unknown_side_effects)
Enter fullscreen mode Exit fullscreen mode

If the first error is not durable, you do not have a restart policy. You have a loop with a counter.

3. Add a restart budget and a quarantine state

A bounded restart budget prevents a broken worker from consuming all CPU, API quota, or queue capacity. For example:

MAX_RESTARTS = 5
WINDOW = 10 minutes
recent = episodes within WINDOW
if len(recent) >= MAX_RESTARTS: return QUARANTINED
else: delay = min(300, 2 ** len(recent))
Enter fullscreen mode Exit fullscreen mode

The exact numbers are workload-specific. The invariant is not: after repeated failure, the worker must stop claiming new work and require a human or an automated rollback decision.

Quarantine should be explicit and observable:

  • no new admissions
  • leases expire or are returned safely
  • in-flight work becomes UNKNOWN unless its outcome is confirmed
  • outbound delivery retries pause
  • a diagnostic bundle is retained
  • recovery can start from a known-good configuration

Do not automatically delete the state directory as a “fix.” That may erase the only evidence needed to distinguish corrupted state from a bad deployment.

4. Test the four common restart-loop causes

Bad configuration

Deploy a config with an invalid tool or model endpoint. Verify that the worker enters NOT_READY, records the config hash, and does not claim work.

State incompatibility

Restore a state snapshot from an older schema. Verify that migration is explicit, reversible, and logged. A migration that partially writes before crashing needs a recovery position, not another blind restart.

Expired credentials

Revoke or expire the credential lease while the worker is stopped. On boot, it should report CREDENTIALS_NOT_READY, avoid tool calls, and request a fresh scoped lease.

Unknown side effect

Kill the process after dispatch but before the provider response. On restart, the run must remain UNKNOWN until the provider is queried or an operator makes a documented reconciliation decision. It must not blindly replay the action.

A useful failure matrix looks like this:

failure                  expected startup state    allowed action
bad config NOT_READY diagnostic only
state migration error QUARANTINED rollback or repair
expired credential NOT_READY lease renewal
unknown tool outcome READY, admission paused reconcile first
Enter fullscreen mode Exit fullscreen mode



  1. Make recovery evidence part of the deployment

Before putting an always-on agent on a VPS or managed runtime, prove that you can answer:

  • Which code and config produced this boot?
  • Which state was durable before the crash?
  • Which work was executing, queued, delivered, or UNKNOWN?
  • Which credentials survived, and what is their blast radius?
  • Can I rebuild the runtime without copying an opaque machine image?
  • Can I pause admissions without taking down diagnostics?

If you want a managed place to run an always-on OpenClaw or browser workload, managed OpenClaw hosting on Ampere is one option to evaluate. It does not remove prompt-injection, credential, or application-level recovery risk; you still need the state, admission, and reconciliation checks above.

The practical rule

A restart is an action, not a diagnosis.

First preserve the failure episode. Then classify process liveness, readiness, execution state, delivery state, and credential state separately. Quarantine after a bounded number of attempts. Reconcile UNKNOWN side effects before replaying anything.

That turns “the container keeps restarting” from a vague uptime problem into a testable recovery contract.

Top comments (0)