Zira

Posted on Aug 17

Your Agent Audit Trail Is Not Evidence Until You Can Verify It

#testing

An agent log can tell you what the process claims happened. It does not prove that the record is complete, ordered, or unchanged.

That distinction matters after a failed deployment, a suspicious tool call, or a customer asking who approved an external side effect. A timestamped JSON file is useful for debugging. It is weak evidence if a worker can overwrite it, two processes can emit the same sequence number, or a restart can silently drop the events between decision and dispatch.

The practical fix is not “log more.” Give the audit stream a small verification contract.

Define the contract before choosing a storage engine

For each event, require:

a run ID and tenant ID
a monotonic sequence allocated by one authority
an event type and schema version
the actor, tool, resource, and policy decision
a server-side timestamp plus the previous event hash
a stable event ID for deduplication
a signature or MAC over the canonical event bytes

Wall-clock time is for humans. Sequence and hash links are for ordering and integrity. Do not use a model-generated timestamp or event order as the source of truth.

A minimal canonical record looks like this:

event = {
  "run_id": "run_7f2",
  "seq": 42,
  "event_id": "evt_42",
  "type": "TOOL_DISPATCHED",
  "tool": "github.create_issue",
  "resource": "repo:acme/api",
  "policy": "allow:ticket-bot",
  "occurred_at": "2026-08-17T08:00:00Z",
  "prev_hash": "sha256:..."
}

canonical = json.dumps(event, sort_keys=True, separators=(",", ":"))
event["hash"] = sha256(canonical.encode()).hexdigest()

Store the hash after the event is accepted, not before. In production, sign or MAC the canonical bytes with a key unavailable to the agent process. Hashing detects accidental changes; a separate key boundary helps detect who could have rewritten the stream.

Make the verifier independent of the agent

The agent should emit events, but it should not be the only component that decides whether its own history is valid. Run a small verifier as a separate job or service that checks:

sequence numbers are strictly increasing within a run
every event points to the previous accepted hash
event IDs are unique
required transitions are legal
signatures or MACs validate
checkpoints agree with the preceding segment

For example, a tool call should normally have a transition like:

PROPOSED -> POLICY_ALLOWED -> DISPATCHED -> OUTCOME_CONFIRMED

A timeout is not confirmation. If the process dies after dispatch, record OUTCOME_UNKNOWN and reconcile with the provider using the stable event ID. Never “repair” the stream by inventing a success event just to make the state machine look tidy.

Checkpoints make long histories practical

A hash chain lets you detect a changed or missing event, but verifying millions of events on every query is expensive. Add signed checkpoints every N events or at each durable state transition:

run ID
last sequence
last event hash
verifier version
checkpoint signature

Keep checkpoints in storage with a different write path from the live event stream. A checkpoint is not a backup and does not prove that the provider received an outbound request. It only gives you an independently verifiable boundary for the local history.

For an always-on OpenClaw or browser-agent deployment, this is one reason to treat the runtime’s durable state and its audit evidence as separate restore items. A managed runtime such as managed always-on agent hosting on Ampere can help with the hosting layer, but it does not make an audit trail trustworthy automatically. You still need scoped credentials, an append-only policy, a verifier, and a tested restore path.

A reproducible failure-injection test

Do not declare the audit trail complete because the happy path looks good. Run these tests against a disposable environment:

Injection	Expected result
Drop event 17	verifier reports a sequence or hash gap
Reorder events 20 and 21	verifier rejects the chain
Duplicate event ID	verifier reports a duplicate, not two tool calls
Mutate the tool resource	signature or MAC validation fails
Kill the worker after DISPATCHED	run becomes OUTCOME_UNKNOWN
Restore an older checkpoint	verifier reports rollback or fork
Rotate the signing key	old events remain verifiable; new events use the new key

The important assertion is not merely “an alert fired.” Capture the exact invalid range, run ID, last trusted checkpoint, and whether any external side effect needs reconciliation.

What to monitor in production

Track evidence health separately from agent health:

audit_events_accepted_total
audit_verification_failures_total
audit_unknown_outcomes
time since last verified checkpoint
event-ingest lag
duplicate event IDs
runs with a missing terminal transition

A green process health check can coexist with a broken audit stream. Alert on verification lag and unknown outcomes even when the worker is still responding.

The rule of thumb

Logs explain. An audit trail constrains and proves.

If you cannot identify the last trusted event, detect a missing or reordered record, distinguish an unknown external outcome from a confirmed one, and verify the evidence after restoring it, you have observability but not reliable history.

Build the verifier and the failure-injection test before you need the evidence. That is the part that survives a restart, a compromised worker, and an uncomfortable question about what the agent actually did.

DEV Community