An agent patch is not certified when the property run is green, if that run's fixture digest also sits in the prompt cohort. Shared bytes make the result a rehearsal. The audit below blocks on digest overlap, on sealed bytes inside the prompt bundle, and on any sealed property that is missing or not passing. A clean result does not merge the patch. It only marks the patch hold-for-human.
The split is the method. Prompt fixtures may be shown to a generator. Verdict fixtures may not. Those two sets have to be digest-disjoint before a flake note is parsed.
What a matching digest fails to prove
A hash match proves byte identity, not holdout coverage. If the same file is both the example in the agent context and the input to the property checker, a pass shows that the candidate handled a case it was allowed to read. It does not show how the candidate behaves on a case it was not allowed to read.
The failure mode is ordinary. A generator is given fixtures/prompt/orders_tiny.json, edits production code, and the CI job points the same property at that file because the path was already wired. The report is green. The file under fixtures/sealed/ never ran, and a conventional summary does not record that omission.
Renaming the path is not a partition. A copy with a new name and the same SHA-256 is still the same fixture. A paraphrase in the prompt can also leak content when the digest string is absent. The checker below catches shared digests, raw sealed bytes in the bundle, and the digest string itself. It does not catch a careful paraphrase.
Properties that the report is allowed to cite
Declare property ids in the manifest, then implement those ids as predicates over a fixture. This sample conserves refund totals. It is an illustration of the predicate shape, not a domain oracle, and it does not use the seed.
def prop_refund_total_conserved(order: dict) -> bool:
if "balance" not in order or "payments" not in order:
return False
refunded = sum(item["amount"] for item in order.get("refunds", []))
paid = sum(item["amount"] for item in order["payments"])
return 0 <= refunded <= paid and order["balance"] == paid - refunded
A predicate that returns true for every input will not be caught by the digest audit. Treat that as out of scope for this script. The runner must still emit one sealed row per declared property id. Emitting only the prompt row leaves the sealed status missing, and missing is not a pass.
Artifact: manifest, bundle, and a local audit
The reproducible artifact is a manifest, a byte-for-byte prompt bundle, a property report, and one script. No network call is required. This script is a local file audit you can run yourself. It was not executed against a live model endpoint for this write-up, and it states no pass rate.
cohort_manifest.json:
{
"sealed": ["fixtures/sealed/orders_v3.json", "fixtures/sealed/refunds_v3.json"],
"prompt": ["fixtures/prompt/orders_tiny.json"],
"properties": ["prop.refund_total_conserved", "prop.status_machine_closed"]
}
property_report.json must carry the cohort as a field:
{
"results": [
{"property": "prop.refund_total_conserved", "cohort": "sealed", "status": "pass", "seed": 17},
{"property": "prop.status_machine_closed", "cohort": "sealed", "status": "pass", "seed": 17},
{"property": "prop.refund_total_conserved", "cohort": "prompt", "status": "pass", "seed": 3}
],
"flake_note": null
}
Hash the fixtures before you trust the report, and keep the pre-generation digest outside the generator workspace:
sha256sum fixtures/sealed/orders_v3.json fixtures/sealed/refunds_v3.json fixtures/prompt/orders_tiny.json | tee sealed_and_prompt.sha256
python3 audit_cohorts.py cohort_manifest.json . prompt_bundle.bin property_report.json
echo "audit_exit=$?"
Exit 1 means blocked. Exit 0 means hold-for-human. Exit 2 means the invocation was wrong. None of those codes is an automatic merge.
#!/usr/bin/env python3
"""Audit digest disjointness for agent-patch fixtures. Local files only."""
import hashlib
import json
import sys
from pathlib import Path
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
def main() -> int:
if len(sys.argv) != 5:
print(
"usage: audit_cohorts.py manifest.json repo_root prompt_bundle.bin property_report.json",
file=sys.stderr,
)
return 2
manifest_path, root = Path(sys.argv[1]), Path(sys.argv[2])
bundle_path, report_path = Path(sys.argv[3]), Path(sys.argv[4])
for path in (manifest_path, bundle_path, report_path):
if not path.is_file():
print(f"missing file: {path}", file=sys.stderr)
return 2
manifest = json.loads(manifest_path.read_text())
bundle = bundle_path.read_bytes()
report = json.loads(report_path.read_text())
for key in ("sealed", "prompt", "properties"):
if key not in manifest:
print(f"manifest missing {key}", file=sys.stderr)
return 2
reasons, leaks = [], []
sealed, prompt = {}, {}
if not manifest["sealed"] or not manifest["properties"]:
reasons.append("empty-sealed-cohort")
for cohort_name, bucket in (("sealed", sealed), ("prompt", prompt)):
for rel in manifest[cohort_name]:
path = root / rel
if not path.is_file():
print(f"missing fixture: {rel}", file=sys.stderr)
return 2
bucket[rel] = sha256_file(path)
if set(sealed.values()) & set(prompt.values()):
reasons.append("shared-digest")
for rel, digest in sealed.items():
raw = (root / rel).read_bytes()
if digest.encode() in bundle:
leaks.append({"fixture": rel, "via": "digest-string"})
elif raw in bundle:
leaks.append({"fixture": rel, "via": "raw-bytes"})
if leaks:
reasons.append("sealed-bytes-in-prompt-bundle")
sealed_status = {}
allowed = set(manifest["properties"])
for row in report.get("results", []):
prop = row.get("property")
if prop not in allowed:
reasons.append("unknown-property")
continue
if row.get("cohort") == "sealed":
sealed_status[prop] = row.get("status")
if any(sealed_status.get(prop) != "pass" for prop in manifest["properties"]):
reasons.append("sealed-property-not-green")
note = report.get("flake_note")
if note:
prop = note.get("property")
digest = note.get("fixture_digest")
if note.get("cohort") != "sealed" or digest not in sealed.values():
reasons.append("flake-note-not-sealed-keyed")
if sealed_status.get(prop) != "pass":
reasons.append("flake-note-without-sealed-pass")
verdict = "blocked" if reasons else "hold-for-human"
print(json.dumps({
"verdict": verdict,
"reasons": sorted(set(reasons)),
"sealed_digests": sealed,
"prompt_digests": prompt,
"leaks": leaks,
}, indent=2, sort_keys=True))
return 1 if verdict == "blocked" else 0
if __name__ == "__main__":
raise SystemExit(main())
If the sealed list or the property list is empty, the script adds empty-sealed-cohort and blocks. An empty holdout is not a quiet success.
JSON fixtures need a canonical byte policy before hashing. Unstable key order makes two equivalent files look disjoint, which weakens the overlap check. This helper is a proposal for JSON only. Write the canonical bytes to disk, then point both the manifest and the script at those bytes. Do not hash canonical bytes in one place and raw bytes in another.
def canonical_json_bytes(path: Path) -> bytes:
payload = json.loads(path.read_text())
return json.dumps(
payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False
).encode()
Illustration of the output contract, not a recorded run: with disjoint digests, no sealed bytes in the bundle, both sealed properties pass, and flake_note null, the printed verdict is hold-for-human and the process exits 0. Any overlap prints blocked and exits 1.
Numbered workflow
1. Freeze sealed digests before generation
Commit sealed_and_prompt.sha256 before the agent process starts. Recompute the same command after the patch. A changed sealed line is blocked, even when properties pass. The generator does not get a vote on oracle inputs.
Store that digest file on the CI host, not in a workspace the generator can edit. A digest computed only after generation can be replaced in the same commit as the fixture.
2. Archive the prompt bundle as bytes
Save the request body, attached files, and tool results that contained fixture data. A chat transcript that drops attachments is not a bundle. If the archive step failed, stop. Missing evidence is not a clean audit.
3. Run the same property ids on both cohorts
Prompt-cohort rows are diagnostic. They show whether the draft is in the right shape. Only sealed rows can clear sealed-property-not-green. Reuse the same property id strings so a renamed check cannot drop coverage without a reason code.
Keep the seed on each row. A new seed is a new observation. It does not convert a missing sealed row into a pass. The sample predicate above ignores the seed because it is deterministic. A sampling property must record the seed it actually used.
This emitter covers only prop.refund_total_conserved. Against the sample manifest, which also names prop.status_machine_closed, the audit stays blocked until you add that second predicate and a sealed row for it.
def emit_report(manifest: dict, root: Path, seed: int) -> dict:
rows = []
for cohort in ("sealed", "prompt"):
for rel in manifest[cohort]:
order = json.loads((root / rel).read_text())
ok = prop_refund_total_conserved(order)
rows.append({
"property": "prop.refund_total_conserved",
"cohort": cohort,
"status": "pass" if ok else "fail",
"seed": seed,
})
return {"results": rows, "flake_note": None}
4. Key any flake note to a current sealed digest
A flake note is optional. It is valid only when three fields agree: cohort is sealed, fixture_digest equals a digest in the current sealed set, and the cited property already has sealed status pass. A note about the prompt cohort, a note with no digest, or a note whose property did not pass is blocked.
{
"flake_note": {
"property": "prop.status_machine_closed",
"cohort": "sealed",
"fixture_digest": "<sha256 of fixtures/sealed/orders_v3.json>",
"observation": "one timeout at seed 17 on the sealed host"
}
}
The note is bound to that digest, not to the diff. When the sealed file changes, the digest no longer matches, and the audit blocks until a human writes a new note. This script does not accept a waiver file shipped inside the patch.
5. Run the audit on the sealed host, then read the diff
Run audit_cohorts.py on the CI host that holds the sealed files. Do not run the certifying copy on the host that generated the draft. hold-for-human means the partition held and the declared sealed properties passed. It does not mean the patch is correct.
Open the diff next, including test files. This script does not detect edits to test sources. A green sealed cohort can still hide a weakened assertion in a file the manifest never names.
How to read the verdict rows
| Sealed properties | Digest overlap or bundle leak | Flake note | Script verdict |
|---|---|---|---|
all pass, cohort non-empty |
none | null, or sealed-keyed to a current digest | hold-for-human |
any missing or not pass
|
any | any | blocked |
all pass
|
shared digest or sealed bytes in the bundle | any | blocked |
all pass
|
none | prompt cohort, or digest not in the sealed set | blocked |
| empty sealed list or empty property list | any | any | blocked |
Row one is the only non-block, and it is still not a merge bit. Rows two through five should stay separate in the log. Folding them into one flaky label hides a contaminated fixture inside a timing story.
Where draft generation fits
Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option can host draft generation and the prompt-cohort run, when those two options are actually available in your workspace.
They are not a sealed runner. Do not mount the sealed directory into that server, and do not paste sealed bytes into the context that will be stored in the prompt bundle. Availability does not change the verdict table. This article claims no quota, hardware shape, model id, or duration for that access. Read those details from the current product surface if your budget depends on them.
If a free MonkeyCode server is already where drafts are produced, bind that workspace to the prompt cohort only, and run audit_cohorts.py on the CI host that stores the sealed files.
Limitations
Byte search misses base64, compression, chunked uploads, and paraphrases that keep meaning without keeping bytes. Filenames and JSON schemas in the prompt still teach structure. A two-file sealed set can be digest-disjoint and still be a thin holdout. SHA-256 will not notice that a sealed file is a trivial subset of a prompt file with a different digest.
The script does not execute properties. A forged property_report.json that says pass is trusted unless CI writes that report on the sealed host after the workspace is locked. Huge fixtures are loaded fully into memory for the byte search, so this form is a poor fit for multi-gigabyte corpora. No generator pass rate is stated here, because none was measured for this text.
Who should skip this
Skip it when you cannot archive the real prompt bundle. A rule that demands evidence you do not retain will only manufacture clean logs. Skip it when the domain has a single fixture that must be shown to the agent. There is no sealed cohort left to hash.
Skip it in pipelines that already require an independent verification group, a formal proof, or signed runner attestation. A local SHA-256 check is weaker than those controls and is not a replacement. Skip it if the goal is to let a green rehearsal clear a queue. That goal contradicts the verdict table.
Freeze sealed digests first. Archive the prompt as bytes. Refuse any flake note that is not keyed to a current sealed digest. The patch can wait for a person.
Top comments (0)