Your AI Agent Health Check Is Lying: Probe the Work Path, Not Just the Process

A process can be alive while an AI agent is unable to do useful work.

That happens when the event loop is running but the state store is read-only, the model provider is unreachable, the queue is wedged, or the worker has lost the credentials it needs. A green process check hides all of those failures.

The fix is not a single expensive end-to-end test on every probe. Use a layered health contract that tells the scheduler and operator exactly which failure domain is broken.

Four health signals

1. Liveness: should this process be restarted?

Liveness should answer one narrow question: is the process making progress through its own event loop?

It should not call an LLM, mutate a ticket, or contact every dependency. A cheap monotonic timestamp or watchdog can detect a deadlock. If liveness fails, a supervisor may restart the process.

2. Readiness: should this worker receive new runs?

Readiness checks the dependencies required before accepting work:

• Can the worker read and write its durable state store?
• Is its queue consumer connected?
• Is its credential lease valid and unexpired?
• Is the configured model or tool endpoint reachable?
• Is the worker draining or fenced by an operator?

A not-ready worker should stop receiving new runs. That is different from killing it.

3. Semantic canary: can one safe run cross the real path?

A semantic canary exercises the orchestration path with a no-side-effect task. It should create a short-lived run, load the same policy and tool registry as production, perform a deterministic assertion, and record evidence. It must not send email, edit a repository, or click a browser button.

For example, the canary can require the agent runtime to return a structured result from a fixed input:

{
  "run_id": "health-canary-2026-08-18T07:00Z",
  "kind": "semantic_canary",
  "expected": "STATE_STORE_AND_TOOL_REGISTRY_OK",
  "observed": "STATE_STORE_AND_TOOL_REGISTRY_OK",
  "side_effects": "none"
}

The important property is not that a model answered a question. It is that the request crossed state loading, policy evaluation, tool registration, execution, and evidence writing without authorizing an external side effect.

4. Dependency health: which boundary is failing?

Expose dependency-level details separately so an operator can distinguish:

• state store unavailable
• queue lagging or disconnected
• model provider timing out
• credential lease rejected
• tool schema mismatch
• evidence writer failing

Do not collapse these into a generic 503. A scheduler needs a safe admission decision; a human needs a repair target.

A small readiness contract

One useful response shape is:

{
  "status": "degraded",
  "accepting_work": false,
  "checks": {
    "process": "ok",
    "state_store": "ok",
    "queue": "ok",
    "credential_lease": "expired",
    "model_provider": "unknown"
  },
  "observed_at": "2026-08-18T07:00:02Z"
}

The unknown state matters. A timeout is not proof that the provider is down, and it is not permission to retry an unknown tool call. Keep the run state and outbound effect state separate, then reconcile them using the provider lookup or an operator review. My earlier guide on adding progress tokens instead of relying on heartbeats covers the related distinction between process liveness and meaningful work progress.

Failure-injection checklist

Test the health contract by breaking one boundary at a time:

1. Freeze the event loop and verify liveness fails.
2. Make the state store read-only and verify readiness rejects new work.
3. Disconnect the queue and verify no duplicate consumer starts.
4. Expire the credential lease while a worker is idle.
5. Delay the model provider until the check becomes unknown.
6. Return an incompatible tool schema from the registry.
7. Kill the evidence writer after execution but before the canary record commits.
8. Restart the watchdog and confirm it does not erase an in-flight run.

Record the expected scheduler action for every case: restart, drain, stop admission, retry a read-only probe, or escalate for reconciliation.

Hosting does not replace the contract

An always-on runtime can make process supervision and dependency monitoring easier to operate, but it does not decide what ready means for your agent. If you run OpenClaw or another long-lived agent, managed always-on OpenClaw hosting on Ampere is one deployment option to evaluate. You still own the health contract, credential scope, safe canary design, and unknown-outcome handling.

The practical rule

Use liveness to decide whether to restart. Use readiness to decide whether to admit work. Use semantic canaries to prove the safe path works. Use dependency details to decide what to repair.

If your dashboard shows only “agent process: up,” it is measuring the easiest thing to fake.