Short version: Most agent teams can show a run. Few can show, in one re-runnable command, whether that run was allowed under a policy. I added a po...
For further actions, you may consider blocking this person and/or reporting abuse
Janz, the policy-vs-data split is the right frame. Same 13 runs. 0/13 under one contract, 13/13 under another - the data didn't move, the contract did. One question: does audit_hash cover the policy.json content itself, or only the findings list? If it's the latter, nothing stops someone editing the policy after a failing run and re-running to a clean hash, with no trace the contract moved. Pinning the policy alongside the run artifacts, not just declaring it, seems like the harder but more honest version of this.
Direct answer, then the change: before your comment, the hash payload was
{runs, findings, passed, policy_keys, required_artifacts, required_artifacts_mode}. It covered the policy's key names and the artifact contract, but not the values ofmax_cost_usd,max_calls, orforbidden_markers. Measured on the fixture, two policies differing only inmax_cost_usd(0.60 vs 100.0) both passed and produced the sameaudit_hashd39ac5fe…. The gap you described was real.I changed it. The audit result now embeds the full canonical
policyobject plus apolicy_sha256, andaudit --output <path>writes that pinned result next to the run artifacts. The same two policies now produce different hashes:79b25b9a…for 0.60 and385f8ed6…for 100.0. The fixture audit hash is nowb304f294…withpolicy_sha256cda4f1bb…. The artifact contract still changes the hash as before (2a3ce600…for the GenMentor-style contract).The failure scenario is now covered both ways: a failing run changes findings, and a clean-but-loosened policy changes
policy_sha256and therefore the audit hash. The qualitative result on the 13 real runs is unchanged: 0/13 under the structured contract, 13/13 under the declared GenMentor contract.Thank you for pushing on the harder version. It was the right call.
You pushed the policy/data split, and pinning exists because of that thread — so the honest next step is letting someone else's machine decide whether the pin reproduces. There's a one-click path now (Actions → reproduce → Run workflow, Target D in the linked issue: github.com/janzong/agent-lab-trust... — no local setup, or one pinned docker command as Target C). If you ever want to spend a minute on it, I'd take the result; if not, no follow-up from me.
Thanks for turning it into something anyone can run. Can't promise a run soon so don't wait on me, but a pin that reproduces on a stranger's machine is the right test for it.
Agreed — and no pressure at all. The pinned digest and the reproduction issue stay put whether anyone runs it today, next month, or never. I'd rather have a reproducible artifact that is rarely checked than a persuasive claim that cannot be checked.
If you do ever run it, the Actions path is the lowest-effort option; if not, the thread already did its job by forcing the pin to be exact.