Give Every Agent Patch a Receipt, Not a Verdict

Our CI box said green at 11:04. The same commit failed on my laptop at 11:06.

The difference was one stale fixture file. Nothing in the report told me that. The suite only told me a boolean.

Agent-written patches make this worse. A patch arrives with a green check attached. I have no idea what the check actually proved. So I stopped reading exit codes and started collecting receipts.

A verdict is one bit. A receipt is a record.

Consider a warehouse audit. "Stock is fine" is a verdict. "Counted 412 pallets at 09:00, seals intact, two pallets moved since Tuesday" is a receipt.

A test run can produce the second kind. It just needs to write down what it saw. Concretely: the commit it ran against, the list of tests it collected, and the hashes of every fixture file before and after the run.

That is enough to answer three questions a boolean cannot. Did this run match the diff I am reviewing? Did the suite rewrite its own inputs? Did the set of executed tests change?

The manifest plugin

I keep this small module outside conftest.py so it can be imported by tooling too. No dependencies beyond the standard library.

# tests/_evidence.py
import hashlib, json, pathlib, subprocess, time

ROOT = pathlib.Path(__file__).resolve().parents[1]
EVID = ROOT / ".evidence"

def sha256(p: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(p.read_bytes())
    return h.hexdigest()

def snapshot(bases) -> dict:
    out = {}
    for base in bases:
        for p in sorted(pathlib.Path(ROOT / base).rglob("*")):
            if p.is_file():
                out[str(p.relative_to(ROOT))] = sha256(p)
    return out

def head_sha() -> str:
    return subprocess.run(
        ["git", "rev-parse", "HEAD"], capture_output=True, text=True
    ).stdout.strip()

def dirty_paths() -> list:
    raw = subprocess.run(
        ["git", "status", "--porcelain"], capture_output=True, text=True
    ).stdout
    return [line[3:] for line in raw.splitlines() if line.strip()]

The hooks are short. They run at session start, after collection, and at session end.

# tests/conftest.py
import json
import pytest
from tests._evidence import EVID, dirty_paths, head_sha, snapshot

WATCH = ["tests/fixtures", "tests/conftest.py", "tests/_evidence.py"]
FROZEN = set(json.loads((EVID.parent / "flake-ledger.json").read_text())["frozen"])

def pytest_sessionstart(session):
    session.ev = {"fixtures_before": snapshot(WATCH)}

def pytest_collection_modifyitems(session, config, items):
    ids = sorted(i.nodeid for i in items)
    session.ev["collected"] = ids
    session.ev["frozen_leak"] = sorted(set(ids) & FROZEN)

def pytest_sessionfinish(session, exitstatus):
    ev = session.ev
    ev["fixtures_after"] = snapshot(WATCH)
    ev["mutated"] = sorted(
        k for k, v in ev["fixtures_after"].items()
        if ev["fixtures_before"].get(k) != v
    )
    ev["exitstatus"] = int(exitstatus)
    EVID.mkdir(exist_ok=True)
    (EVID / "manifest.json").write_text(json.dumps(
        {"head": head_sha(), "dirty": dirty_paths(), **ev},
        indent=2, sort_keys=True,
    ))

The ledger is a plain file with one rule inside it. Frozen tests are quarantined, not deleted.

{
  "frozen": [
    "tests/test_upload.py::test_retry_budget",
    "tests/test_sync.py::test_conflict_merge"
  ],
  "rule": "no frozen node may appear in a run that gates a patch"
}

Exclude them with --deselect flags in addopts, not with skip markers. A skipped test still shows up in collection and pollutes the comparison. Deselected tests do not.

Two runs on two boxes

One run is an opinion. Two runs on separate machines is evidence. I run the same wrapper twice, on CI and on a clean box.

#!/usr/bin/env bash
set -euo pipefail
export PYTHONDONTWRITEBYTECODE=1
rm -rf .evidence && mkdir -p .evidence
status=0
python -m pytest -p no:cacheprovider -q --timeout=120 || status=$?
stamp=$(date -u +%Y%m%dT%H%M%SZ)
mv .evidence/manifest.json ".evidence/run-$stamp.json"
exit "$status"

The comparison is twenty lines. It reports drift, not pass or fail.

# tools/drift.py
import json, pathlib, sys

a, b = (json.loads(pathlib.Path(p).read_text()) for p in sys.argv[1:3])
keys = ["head", "collected", "mutated", "frozen_leak", "exitstatus"]
drift = {k: {"first": a.get(k), "second": b.get(k)} for k in keys if a.get(k) != b.get(k)}
print(json.dumps({"clean": not drift, "drift": drift}, indent=2))

Run it after both boxes finish.

python tools/drift.py .evidence/run-a.json .evidence/run-b.json

A clean report looks like this. Anything else is a reason to read the diff again.

{ "clean": true, "drift": {} }

On a two-core box my last pair of runs took 38 seconds total. The receipt cost less than the argument it ended.

The gate table

I turn the manifest into five gates. Each one can block a patch on its own.

Gate	Check	Blocks when
Identity	manifest.head equals the SHA under review	Patch was rebased or the run is stale
Clean tree	dirty lists only files inside the patch	Agent left untracked scratch files
Fixture integrity	mutated is empty	The suite rewrote its own inputs
Collection parity	collected matches the baseline set	Tests silently vanished or appeared
Quarantine	frozen_leak is empty	A known-flaky test slipped back into the gate

The rules are boring on purpose. Boring rules survive deadlines.

Where this fits with an agent workflow

I use MonkeyCode's free model access to draft property-based assertions from the diff. I paste a hunk, ask for invariants, and keep the two or three that a human could defend. Then I use the free server option for the second box, so the clean-box run does not sit on my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both of those are convenience, not proof. The model proposes properties. The manifest decides whether the run that tested them can be trusted. Availability is operator-supplied here, and I make no claim about model quality or server specifications.

Limits I hit

Hashing catches writes, not meaning. A fixture can be rewritten with identical bytes and still change behavior through its filename. So add the path to the hash map, not just the content.

Collection parity breaks on legitimate refactors. Renaming a test file produces a large diff in collected. Accept it in the same commit that renames tests, and nowhere else.

Ledger rot is real. A test frozen in March may be stable by June. Review the ledger quarterly or it becomes a graveyard. Two of my eight entries came back last quarter.

Network tests still win. If the clean box and CI see different upstream services, exit codes will diverge for reasons no manifest can explain. Mark those tests and run them outside the gate.

Who should skip this

Small projects with one test file do not need a manifest. Neither do teams whose suites never touch files on disk. The overhead is real: one more artifact to store and one more script to maintain.

Skip it too if you cannot pin a commit. A receipt for an unknown revision is just paper.

The narrow case is the one that matters. An agent hands you a patch, a green check, and no evidence. Ask for the receipt instead of the verdict.

If you try the two-box run, start with the manifest only. Add the gate table after your first false green.