An agent can produce thousands of log lines and still leave you unable to answer the only question that matters during an incident: what did this run actually do?
A useful agent trace is not a transcript. It is a durable run ledger that lets an operator reconstruct intent, authority, tool dispatch, external results, and recovery decisions without trusting the model’s final summary.
This matters for browser agents, coding agents, MCP clients, and OpenClaw workflows. A model can say "the deployment completed" when the request timed out after the provider accepted it, or say "I used the production profile" when the browser worker was attached to a different profile. Logs alone do not make those states distinguishable.
Define the run contract first
Give every run a stable identity before the first model call:
- run_id: immutable execution identity
- parent_run_id: the triggering run, if any
- tenant_id and actor_id: who authorized the work
- policy_version: the rules in effect at dispatch time
- workspace_id or browser-profile identity
- request_hash: normalized input, with secrets removed
- started_at and deadline_at
Do not use a conversation ID as the run ID. One conversation can contain retries, handoffs, and resumed work. Those need separate runs or attempt IDs if you want to distinguish a new execution from a replay.
A minimal event should look more like this than a free-form log line:
run_id: r_2026_08_09_001
attempt: 2
sequence: 17
phase: tool_dispatch
tool: deploy
arguments_hash: sha256:...
policy_version: deploy-v4
decision: allowed
outcome: unknown
recorded_at: 2026-08-09T10:14:02Z
Keep the actual arguments in a separately protected store when they contain sensitive data. The ledger should retain enough metadata to correlate the call without turning observability into a credential leak.
Record state transitions, not just messages
For each model turn and tool call, record a small state transition:
- PLANNED: the model requested an action.
- AUTHORIZED: policy evaluation allowed this exact tool and normalized argument set.
- DISPATCHED: the runtime handed the request to the external system.
- SUCCEEDED or FAILED: the external result is known.
- UNKNOWN: the runtime lost certainty after dispatch.
- RECONCILED: a later read confirmed the external state.
The important transition is UNKNOWN. Treating every timeout as failure causes duplicate mutations; treating every timeout as success hides lost work. The ledger should make "we do not know yet" a visible operational state with an owner and a reconciliation deadline.
This is also where authorization belongs. A plan made under policy-v3 should not silently execute under policy-v4. Recheck the policy at dispatch and record both versions if they differ.
Make the ledger useful during a restart
Write the dispatch intent before sending the side effect. Then use a transaction or equivalent durable boundary so a crash cannot leave an untraceable request:
action_intent(run_id, attempt, request_hash, state=AUTHORIZED)
commit()
dispatch_to_external_system()
record(SUCCEEDED | FAILED | UNKNOWN)
On restart, query unfinished intents before creating new ones:
reconcile(ledger, external_api):
for intent in ledger.unfinished():
if intent.state != UNKNOWN:
continue
result = external_api.lookup_by_request_key(intent.request_key)
if result.found:
ledger.mark_reconciled(intent, result.remote_id)
elif result.not_found and intent.expired:
ledger.mark_failed(intent, reason=expired_without_evidence)
else:
ledger.keep_unknown(intent)
The lookup needs a stable request key or another idempotency mechanism. A timestamp and a natural-language description are not enough to prove that two requests are the same.
For browser work, the equivalent check might be a remote order ID, a sent-message marker, or a server-side audit record. Do not infer success from a changed local page or from the model’s summary.
Four queries an operator should be able to run
If the data model is right, incident response becomes a query problem:
- Which external calls were authorized but never dispatched?
- Which calls are UNKNOWN past their reconciliation deadline?
- Which runs used a browser profile, credential scope, or policy version different from the intended one?
- Which side effects have no parent run or request key?
If answering these requires grepping prose, the ledger is not doing its job.
Test observability as a failure mode
Add these fixtures to CI or a disposable staging environment:
- crash before the dispatch-intent commit
- crash after dispatch but before the response is recorded
- duplicate delivery of the same event
- policy change between planning and dispatch
- clock skew around the reconciliation deadline
- malformed or partial provider responses
- log-store outage while the side effect succeeds
- worker restart with a stale browser or credential identity
For each fixture, assert both the external result and the evidence result. A system that performs the correct mutation but cannot prove which run performed it is not fully observable.
Track operational metrics such as time to reconcile UNKNOWN work, percentage of side effects with a request key, orphaned events, missing policy versions, and time to reconstruct a failed run. Avoid using token counts or log volume as substitutes for these measures.
A small rollout checklist
Before trusting an always-on agent:
- create the run ID before planning
- persist intent before mutation
- attach actor, tenant, policy, tool, and profile identity
- record normalized argument hashes without copying secrets into logs
- distinguish FAILED from UNKNOWN
- provide a read-based reconciliation path
- make duplicate delivery safe or detectable
- test telemetry loss separately from runtime failure
- verify that an operator can reconstruct one run from the ledger alone
More logs can help with debugging, but they cannot repair an ambiguous state model. Start with a run ledger, make uncertainty explicit, and test whether the evidence survives the same crashes your agent is expected to survive.
Top comments (0)