Your agent run passed. Can you prove it was allowed?

Short version: Most agent teams can show a run. Few can show, in one re-runnable command, whether that run was allowed under a policy. I added a policy-as-code audit to agent-lab-trust: it checks cost and call caps, the declared artifact contract, and forbidden markers, then emits a canonical audit_hash. On 13 archived runs, the default contract passed 0/13 and a declared GenMentor contract passed 13/13. The data did not change. The contract did.

The gap

Dashboards answer "what happened". Governance needs "what was allowed, and by which rule". A run can succeed and still violate a budget, write an unexpected artifact, or carry a marker that should never ship.

The audit is deliberately boring:

agent-lab-trust audit <run-root> --policy policy.json

A policy can declare:

• max_cost_usd and max_calls per run;
• required_artifacts plus required_artifacts_mode: all|any;
• forbidden_markers.

The output is a finding list and a canonical audit_hash. deletion-proof --output <path> writes the deletion evidence used in the same policy flow.

The demonstration

I ran the audit over 13 archived GenMentor runs.

• Trust-layer default contract (output/structured.json or results/structured.json): 0/13 passed, missing_artifact ×13, cost_exceeded ×1.
• Declared GenMentor contract (archive.json and summary.json, mode all, cap 5.00): 13/13 passed, no findings.

Audit hashes:

• default structured contract: 2c122c4ff2c0d5eed11a0fc23b4ff717002864dbd941402e8fe18d6d17c96ce6
• GenMentor contract: 368b75a06c0f5c205436d287881cfddcbabd94b7ad34a860e852fba59dad7ab5

The one cost finding is real: replay-8of8-20260920-a recorded 3.9529266 under the trust-layer cap 0.60. Under the GenMentor policy cap 5.00, it passes. Caps are part of the contract too.

The lesson is not "the audit is noisy". It is that an audit without an explicit run-family contract silently embeds one family's format. The policy has to name the contract, the caps, and the markers.

Reproduce

docker run --rm ghcr.io/janzong/agent-lab-trust:rc2

or bash scripts/reproduce.sh. Expected: 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, deletion audit_hash 4f0193abbd49a0f9.

The :rc2 tag points at the latest published build; SERIES.md pins the verification digest sha256:2e178e63fff30e70ac68501cb17e5316097875932d1d7085e669ce4f8d5a105f.

What this does not prove

• no real governance deployment;
• no external reproduction yet;
• no real decision changed yet;
• the 13 runs are a private synthetic family, not published data.

The series

This is piece 3 of a three-part line: write it down → test it → govern it.

1. agent-charters: what people actually tell agents.
2. agent-lab-trust: whether an agent run is valid and reproducible.
3. this audit: whether a run was allowed under a declared policy.

Index: https://github.com/janzong/agent-lab-trust/blob/main/SERIES.md

Call

Run the audit on a run family you own. Tell me where the policy and the artifacts disagree. I am also looking for 3 independent reproductions of the trust layer: https://github.com/janzong/agent-lab-trust/issues/1

Comments

[Mihai Perdum]

Janz, the policy-vs-data split is the right frame. Same 13 runs. 0/13 under one contract, 13/13 under another - the data didn't move, the contract did. One question: does audit_hash cover the policy.json content itself, or only the findings list? If it's the latter, nothing stops someone editing the policy after a failing run and re-running to a clean hash, with no trace the contract moved. Pinning the policy alongside the run artifacts, not just declaring it, seems like the harder but more honest version of this.

[Janz]

Direct answer, then the change: before your comment, the hash payload was {runs, findings, passed, policy_keys, required_artifacts, required_artifacts_mode}. It covered the policy's key names and the artifact contract, but not the values of max_cost_usd, max_calls, or forbidden_markers. Measured on the fixture, two policies differing only in max_cost_usd (0.60 vs 100.0) both passed and produced the same audit_hash d39ac5fe…. The gap you described was real.

I changed it. The audit result now embeds the full canonical policy object plus a policy_sha256, and audit --output <path> writes that pinned result next to the run artifacts. The same two policies now produce different hashes: 79b25b9a… for 0.60 and 385f8ed6… for 100.0. The fixture audit hash is now b304f294… with policy_sha256 cda4f1bb…. The artifact contract still changes the hash as before (2a3ce600… for the GenMentor-style contract).

The failure scenario is now covered both ways: a failing run changes findings, and a clean-but-loosened policy changes policy_sha256 and therefore the audit hash. The qualitative result on the 13 real runs is unchanged: 0/13 under the structured contract, 13/13 under the declared GenMentor contract.

Thank you for pushing on the harder version. It was the right call.

[Janz]

You pushed the policy/data split, and pinning exists because of that thread — so the honest next step is letting someone else's machine decide whether the pin reproduces. There's a one-click path now (Actions → reproduce → Run workflow, Target D in the linked issue: github.com/janzong/agent-lab-trust... — no local setup, or one pinned docker command as Target C). If you ever want to spend a minute on it, I'd take the result; if not, no follow-up from me.

[Mihai Perdum]

Thanks for turning it into something anyone can run. Can't promise a run soon so don't wait on me, but a pin that reproduces on a stranger's machine is the right test for it.

[Janz]

Agreed — and no pressure at all. The pinned digest and the reproduction issue stay put whether anyone runs it today, next month, or never. I'd rather have a reproducible artifact that is rarely checked than a persuasive claim that cannot be checked.

If you do ever run it, the Actions path is the lowest-effort option; if not, the thread already did its job by forcing the pin to be exact.