A Green Run Is a Claim, Not Evidence: Re-deriving Agent Patch Test Runs

A green test run is a claim about one environment at one moment. An evidence packet is the derivation that lets someone else reach the same conclusion without you. If an agent patch merges with a green badge and nothing else, you shipped a screenshot.

This post builds the smallest packet I trust, a verifier that re-derives it, and a sampling policy so the check stays cheap. The artifact is stdlib Python plus pytest; nothing here depends on a hosted service.

The three ways a green run silently stops being true

Agent patches drift more than human ones because the loop is fast: the patch is edited, rebased, or regenerated between the test run and the merge. Three things break, and all three break quietly.

1. Tree drift. Someone changes the patch (or a lockfile) after the run. The badge still renders.
2. Selection drift. Collection changes — a marker moved, a conftest gained a skip, a new test file appeared — so "the suite" is no longer the set of node IDs that ran.
3. Environment drift. Interpreter patch version, hash seed, timezone, PYTHONPATH, native library versions. Any of these can flip a pass to a fail with zero source changes.

Signal	What it catches	Capture cost	Re-derive cost
Whole-tree SHA-256	tree drift, lockfile edits	one pass over files	same
Hash of sorted collected node IDs	selection drift, silent skips	one --collect-only	same
Env whitelist (hash seed, TZ, locale, path)	environment drift	free	free
Interpreter + platform string	version skew	free	free
Full stdout/stderr of the run	everything, at high storage cost	large	large

The first four rows are cheap enough to keep for every patch. The last row is what people usually store, and it is the least useful of the five.

Step 1 — Capture the packet at run time

Capture inside the same process that runs the tests, not from a shell wrapper someone can edit later.

# evidence_packet.py — stdlib only
import hashlib, json, os, platform, subprocess, sys, time

ALLOW_ENV = ("PYTHONHASHSEED", "LANG", "TZ", "PYTHONPATH")
SKIP_DIRS = {".git", "__pycache__", ".venv", ".mypy_cache", "node_modules"}


def tree_digest(root: str) -> str:
    h = hashlib.sha256()
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
        for name in sorted(filenames):
            path = os.path.join(dirpath, name)
            rel = os.path.relpath(path, root)
            h.update(rel.encode())
            with open(path, "rb") as fh:
                h.update(hashlib.sha256(fh.read()).digest())
    return h.hexdigest()


def collect(test_path: str) -> list[str]:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", test_path, "--collect-only", "-q"],
        capture_output=True, text=True,
    )
    if proc.returncode not in (0, 5):  # 5 = no tests collected
        raise SystemExit(f"collection failed: {proc.stderr[-400:]}")
    return sorted(ln.strip() for ln in proc.stdout.splitlines() if "::" in ln)


def build(root: str, test_path: str) -> dict:
    node_ids = collect(test_path)
    return {
        "schema": 1,
        "captured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "tree": tree_digest(root),
        "tests_digest": hashlib.sha256("\n".join(node_ids).encode()).hexdigest(),
        "node_count": len(node_ids),
        "env": {k: os.environ.get(k, "") for k in ALLOW_ENV},
        "interpreter": f"{platform.python_version()} {platform.platform()}",
        "argv": [sys.executable, "-m", "pytest", test_path, "-p", "no:randomly", "-q"],
    }

Three decisions matter here. The digest skips caches so it is stable across machines. The node-ID hash sorts before hashing so collection order cannot mask a change. The argv pins the random plugin off, which removes one source of run-to-run variance that a packet should not have to explain.

Step 2 — Re-derive instead of trusting

Verification is not "read the JSON and agree". It is: rebuild the packet from the current tree, diff the cheap fields, and only then re-run the suite.

def verify(packet: dict, root: str, test_path: str) -> dict:
    live = build(root, test_path)
    drift = [k for k in ("tree", "tests_digest", "node_count", "interpreter")
             if live[k] != packet[k]]
    if drift:
        return {"verdict": "STALE", "drifted": drift}
    proc = subprocess.run(packet["argv"], capture_output=True, text=True,
                          env={**os.environ, **packet["env"]})
    return {"verdict": "REPRODUCED" if proc.returncode == 0 else "FAILED",
            "returncode": proc.returncode,
            "tail": (proc.stdout or proc.stderr)[-400:]}

STALE is the verdict that earns its keep. In practice it means one of: the patch was touched after the run, a test was added or deselected, or the interpreter moved. Those are three different bugs with the same symptom, and the drifted list tells you which one you are looking at before you spend a single core-second re-running tests.

Step 3 — Sample re-derivation, do not gate on it

Re-deriving every patch is a tax you will stop paying within a month. Sample instead, weighted by risk:

Patch class	Re-derive	Reason
Touches conftest.py, markers, or CI config	always	selection drift surface
Touches dependency locks or interpreter pin	always	environment drift surface
Generated by an agent with >1 edit round	always	tree drift surface
Test-only addition, no prod code	nightly batch	low blast radius
Docs, types, comments	never	no evidence claim worth keeping

The policy is the artifact. A team that cannot state which patches get re-derived is running on vibes with a YAML file attached.

Step 4 — Property probes that survive a re-derivation

A packet proves a run happened as described. It says nothing about whether the run was worth anything. Pair it with a small number of invariant probes that hold across trees, so a stale packet fails loudly rather than politely.

• Idempotence probe. Apply the agent's change twice; the second application must be a no-op or must error, never silently double-apply.
• Boundary probe. The patched function returns the same value at its documented edges as the pre-patch implementation, or the diff explains why not.
• Absence probe. After the patch, the old failure mode is still reachable when the fix is disabled — if it is not, you changed the fixture, not the behavior.

Each probe is one assertion, not a test suite. Three probes that fail for the right reason beat forty that pass for none.

Where free access actually helps

Two pieces of this workflow are off the critical path: the nightly batch re-derivation, and triage of the drifted list when a batch reports a dozen STALE verdicts at once. Those map cleanly onto availability-limited resources rather than your merge gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is a reasonable fit for the triage half — classifying a batch of drift records into "rebased", "collection changed", "interpreter moved" — because the output is a short label per record and a mislabel only costs a human glance. The free server option fits the batch half: run the verifier on a schedule, write packets and verdicts to an artifact store, and keep the merge gate on your own runners. Neither belongs inline in pre-merge; treat capacity as best-effort and design the workflow so a skipped night weakens your sampling, not your gate.

# nightly, off the critical path
python evidence_packet.py verify --root . --tests tests/ --since 24h \
  --out artifacts/evidence/$(date -u +%Y%m%dT%H%M%SZ).json

Limitations and who should skip this

• Slow builds. If one full re-derivation costs more than about ten minutes, sampling shrinks to near zero and the packet becomes decoration. Fix build time first.
• Regulated audit trails. This packet is reproducibility evidence, not a signed attestation. It does not satisfy frameworks that require cryptographic signing, retention policy, or third-party attestation.
• Single-developer prototypes. One author, one day, no merge queue: the tree digest alone is enough; the rest is ceremony.
• Non-deterministic suites. If your suite is genuinely non-deterministic, REPRODUCED versus FAILED is noise. Stabilize or quarantine first; a packet cannot launder a flaky suite into a trustworthy one.

The habit worth keeping is narrow and durable: when a patch reports green, ask what would have to be true for that green to be false, and store the four cheap fields that answer it. If you want to try the batch half on free capacity, the nightly command above is the whole integration surface.