A process heartbeat can tell you that an agent runtime is still responding. It cannot tell you that the run is making progress.
That distinction matters in long-lived coding agents, browser automations, and OpenClaw-style workers. A loop can keep answering health checks while holding a lease, retrying the same tool call, waiting on a dead child process, or consuming tokens without advancing the workflow.
The fix is small: make progress an explicit, durable signal and monitor it separately from process liveness.
Two signals, two questions
Use a heartbeat for: “Is the process alive enough to answer?”
Use a progress token for: “Has this run advanced since the last observation?”
A progress token is not a timestamp that updates on every loop iteration. It should change only when the agent crosses a meaningful state boundary, such as:
- a plan step was accepted
- a tool call reached a terminal outcome
- a file diff was validated
- a child task returned a result
- an outbound delivery was confirmed
A useful record looks like this:
{
"run_id": "run_8f2",
"state": "EXECUTING",
"progress_seq": 17,
"progress_kind": "tool_outcome_confirmed",
"progress_at": "2026-08-18T03:10:00Z",
"lease_token": 42
}
The sequence is more important than the wall-clock time. Clocks can move backward, workers can replay old messages, and a noisy loop can refresh a timestamp without doing useful work.
Define what counts as progress
Write the state machine before writing the watchdog. For example:
QUEUED -> PLANNING -> EXECUTING -> VERIFYING -> SUCCEEDED
| |
v v
BLOCKED UNKNOWN
Then define an allowed progress event for each state:
| State | Progress event | Not progress |
|---|---|---|
| PLANNING | plan version committed | another model-token callback |
| EXECUTING | tool outcome recorded | tool request still retrying |
| VERIFYING | verifier result persisted | verifier process heartbeat |
| BLOCKED | blocker reason and next action saved | repeated “waiting” logs |
| UNKNOWN | reconciliation attempt recorded | silently restarting the worker |
This prevents the watchdog from rewarding activity that only looks busy.
Store progress with the run, not in memory
An in-memory counter disappears when the worker restarts. Persist the progress record with the run state, and update it transactionally with the state transition that it describes.
A minimal SQL shape is:
UPDATE agent_runs
SET state = :next_state,
progress_seq = progress_seq + 1,
progress_kind = :kind,
progress_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE run_id = :run_id
AND lease_token = :current_lease_token;
The lease check matters. Without it, a paused worker can wake up after replacement and overwrite the new worker’s progress. The update should affect exactly one row. A zero-row update means the worker lost ownership and must stop.
Keep the progress event immutable in an append-only table if debugging and incident reconstruction matter:
CREATE TABLE run_progress (
run_id TEXT NOT NULL,
progress_seq INTEGER NOT NULL,
kind TEXT NOT NULL,
detail_json TEXT NOT NULL,
recorded_at TEXT NOT NULL,
PRIMARY KEY (run_id, progress_seq)
);
The current row is useful for monitoring. The event history is what lets you answer “where did it stop?” after a crash.
Watch for stale progress, not just dead processes
A watchdog can classify a run using both signals:
- No heartbeat: treat the worker as unavailable and begin ownership recovery.
- Heartbeat is fresh but progress is stale: mark the run STALLED.
- Progress advances: keep the lease and continue observing.
- Progress is UNKNOWN after an external side effect: reconcile before retrying.
Do not immediately kill every stalled run. First capture the last progress kind, lease token, current tool, retry count, and child-process status. Then choose a bounded recovery action:
- re-poll a provider if the run is waiting on an external result
- terminate and replace a child process that has no valid lease
- pause the run when the same progress sequence repeats past a retry budget
- route to manual review when the outcome of a side effect is ambiguous
For an always-on OpenClaw or browser-agent deployment, a managed runtime such as always-on agent hosting on Ampere can reduce the operational work of keeping the process available. It does not decide what progress means, recover an unknown side effect, or replace a run-level lease and evidence trail. Those contracts still belong in the application.
A failure-injection test matrix
Test the difference between liveness and progress deliberately:
| Injection | Expected result |
|---|---|
| health endpoint stays up while the main loop repeats | watchdog marks STALLED |
| worker pauses after dispatch but before outcome recording | outcome becomes UNKNOWN and is reconciled |
| old worker resumes after lease replacement | stale lease update affects zero rows |
| progress event is duplicated | sequence or idempotency constraint rejects it |
| progress store is unavailable | run stops safely instead of claiming advancement |
| watchdog restarts during recovery | persisted state makes recovery idempotent |
Record evidence for each test: last heartbeat, last progress sequence, lease token, state transition, recovery decision, and final disposition. A green health check is not proof that the run completed.
Practical checklist
Before calling an agent “healthy,” verify that you can answer:
- What meaningful event increments progress?
- Is progress durable across a worker restart?
- Is the update fenced by the current lease or generation?
- Can a noisy retry loop fake advancement?
- What happens when progress stops but heartbeats continue?
- Which states require reconciliation instead of retry?
- Can the watchdog restart without duplicating recovery?
- Can an operator reconstruct the last confirmed boundary?
Heartbeats are still useful. They answer the process-liveness question. Progress tokens answer the workflow-liveness question. Production agents need both, with separate thresholds and separate recovery actions.
Top comments (0)