An agent can print thousands of log lines and still leave you unable to answer the one question that matters after a failure: what did it actually do?
A useful audit trail is not the same thing as verbose logging. Logs describe observations. An audit trail must let you prove the order of important decisions, detect missing evidence, and distinguish an attempted side effect from a confirmed one.
This article shows a small evidence contract you can add around an agent runtime. It is deliberately boring: append-only events, stable IDs, a hash chain, redaction at write time, and tests that intentionally remove or reorder records.
Start with an evidence contract
For every run, record an immutable run ID and a monotonically increasing sequence number. Wall-clock timestamps are useful for humans, but they are not safe ordering keys when workers have skewed clocks.
A minimal event looks like this:
{
"run_id": "run_01J...",
"seq": 42,
"event": "TOOL_OUTCOME_CONFIRMED",
"tool": "github.create_issue",
"request_key": "rk_7f...",
"policy_digest": "sha256:...",
"credential_version": 8,
"result": "provider_id:issue_123",
"prev_hash": "sha256:...",
"event_hash": "sha256:..."
}
Keep the event vocabulary small. For a tool call, I normally need at least:
- INTENT_RECORDED: the runtime accepted a proposed action.
- DISPATCHED: a request crossed the provider boundary.
- OUTCOME_CONFIRMED: the provider returned a verifiable result.
- OUTCOME_UNKNOWN: the process lost certainty after dispatch.
- RECONCILED: a later lookup resolved UNKNOWN.
Do not write raw prompts, cookies, bearer tokens, or full tool responses into this ledger. Redact before persistence, not in a dashboard query. A dashboard is too late if the original evidence already contains a credential.
Use a hash chain to catch silent gaps
A hash chain does not make the log truthful. It makes tampering or omission detectable when the verifier has a trusted checkpoint.
For each event, canonicalize the fields, then calculate:
event_hash = SHA256(canonical_event_without_event_hash + prev_hash)
Store a signed or separately protected checkpoint at run completion and at regular intervals. The verifier should reject:
- a sequence number that goes backwards or skips without an explicit gap event
- a prev_hash that does not match the prior record
- an event whose recomputed hash differs
- two events claiming the same sequence number
- a completion record without a terminal outcome for each dispatched effect
If your storage is eventually consistent, do not pretend a missing record is proof of absence. Mark the range as INCOMPLETE and reconcile it from the source or replica that owns the checkpoint.
Separate execution evidence from delivery evidence
A runtime may successfully finish a task while failing to deliver the notification. Record those as separate state machines. OUTCOME_CONFIRMED for an API call does not prove that an email, webhook, or chat message was delivered.
Use a stable request key for every externally visible effect. On retry, look up the request key and provider message ID before sending again. If the provider cannot answer, preserve UNKNOWN rather than guessing. This is how the audit trail prevents a recovery loop from becoming a duplicate-send loop.
Add evidence tests, not only happy-path tests
The test suite should mutate the ledger and verify that the verifier fails closed. Here is a compact test matrix:
| Fault injection | Expected result |
|---|---|
| Delete event 42 | INCOMPLETE or chain failure, never VERIFIED |
| Swap events 41 and 42 | sequence or hash failure |
| Change policy digest | policy mismatch |
| Insert a fake confirmation | unknown request key or signature failure |
| Remove the final checkpoint | run remains UNSEALED |
| Replace a credential version | authorization mismatch |
| Truncate after DISPATCHED | effect remains UNKNOWN and enters reconciliation |
Run these tests against the actual serialization and storage path. A unit test over an in-memory object proves very little if production code serializes JSON differently, strips fields, or writes records out of order.
What to expose to operators
Give operators a run timeline with four separate indicators:
- evidence completeness: are all expected sequence ranges present?
- integrity: does the hash chain verify to a trusted checkpoint?
- authorization: do policy and credential versions match the dispatch decision?
- effect certainty: which actions are confirmed, unknown, or only intended?
Do not collapse these into a single green check. A run can have an intact ledger and an UNKNOWN external effect. It can also have a confirmed effect but incomplete logs. Those states require different recovery actions.
For always-on OpenClaw or browser-agent deployments, the hosting choice is secondary to this contract. A managed runtime such as managed OpenClaw hosting on Ampere can reduce the amount of infrastructure you operate, but it does not replace redaction, scoped credentials, durable evidence, or reconciliation.
A practical acceptance checklist
Before calling your agent observable, prove that:
- every run and external effect has a stable ID
- ordering uses a sequence or logical clock, not timestamps alone
- sensitive fields are redacted before persistence
- the verifier detects deletion, mutation, duplication, and reordering
- checkpoints are protected separately from the event store
- execution completion and outbound delivery have separate evidence
- UNKNOWN is a durable state with a reconciliation path
- a clean rebuild can verify the ledger without trusting the crashed process
The goal is not more logs. The goal is a bounded answer to: what was intended, what crossed a boundary, what was confirmed, and what still needs reconciliation? Developers building agents that survive restarts, retries, and partial outages will get more value from that answer than from another wall of debug output.
Top comments (0)