Throw Away the Agent Recap. Merge Only a Hashable Proof Bundle.

The recap is not evidence. If a merge gate can be satisfied by a paragraph the model wrote about itself, you already lost.

I do not care that the agent said “all tests passed.” I care that a directory on disk can be hashed, parsed, and rejected without asking the model a follow-up. Recaps are cheap. Proof is a folder.

Why this now? The industry is loud about vibe coding versus engineering. Fine. I am not here to argue the slogan. I am here to make pretending fail in CI.

This is a from-zero workflow. Each stage has a command and a verification step. Examples below are a proposed local method. Label them that way. I am not attaching fake timings, fake pass rates, or a model name I cannot stand behind.

What you are building

A proof bundle: a small, boring directory the agent does not get to narrate.

proof/
  done.json          # you wrote this before any prompt
  junit.xml          # produced by your test runner, not by the model
  git_stat.txt       # git diff --stat against the base ref
  git_name_status.txt
  forbidden.txt      # scan output; empty file means clean
  env_fingerprint.txt
  PROOF.sha256       # hash of the files above, written last

The model may write code. It may not write junit.xml. It may not write PROOF.sha256. If those files appear in a diff from the agent, the job dies. That is the whole trick.

Need a remote box because your laptop is a mess? Same verifier. Same bundle. I will get to that after the local path works.

Stage 0 — Pin the base ref

Do not start from “whatever HEAD feels like.”

git rev-parse --abbrev-ref HEAD
git status --porcelain
git rev-parse HEAD > /tmp/agent-base.sha
cat /tmp/agent-base.sha

Verify: git status --porcelain is empty. If it is not, stop. You cannot hash a proof bundle against a dirty tree and then pretend the hash means anything.

Why so strict? Because later you will ask git diff for a budget. A dirty tree lies.

Stage 1 — Write done.json before the prompt

I write the contract first. Always. The prompt comes after, and the prompt is allowed to be sloppy because the contract is not.

{
  "job_id": "fix-invoice-timezone",
  "base_sha_file": "/tmp/agent-base.sha",
  "must_pass": ["tests/test_invoices.py"],
  "min_tests": 4,
  "max_failures": 0,
  "max_files_changed": 6,
  "max_diff_lines": 180,
  "forbidden_globs": ["secrets/**", ".env", ".env.*", "**/*.pem"],
  "junit_path": "proof/junit.xml",
  "agent_must_not_touch": [
    "proof/junit.xml",
    "proof/PROOF.sha256",
    "proof/done.json"
  ]
}

Put it in proof/done.json. Commit it if you want the contract in history. I do.

Verify:

python -c "import json,pathlib; json.loads(pathlib.Path('proof/done.json').read_text())"
test -f proof/done.json && echo done.json_ok

If that one-liner throws, you do not have a contract. You have a mood.

Stage 2 — Fingerprint the toolchain

Agents love to “just pip install” their way out of a red test. Do not let them. Record what you believe the job is running on.

mkdir -p proof
{
  echo "python=$(python --version 2>&1)"
  echo "pip=$(pip --version 2>&1)"
  echo "pytest=$(pytest --version 2>&1 | head -n 1)"
  echo "base=$(cat /tmp/agent-base.sha)"
  echo "uname=$(uname -s -m)"
} > proof/env_fingerprint.txt
cat proof/env_fingerprint.txt

Verify: every line has a value. If pytest= is empty, install the runner yourself. The agent does not get to choose the runner after the fact. That is how recaps get born.

Stage 3 — Produce a JUnit file you trust

Run the tests you named. Redirect your runner into proof/junit.xml. Not a screenshot. Not a quote block.

pytest tests/test_invoices.py --junitxml=proof/junit.xml -q
ls -l proof/junit.xml

Yes, this may be red. Good. A red proof bundle is still a proof bundle. A green recap with no XML is a story.

Verify:

python - <<'PY'
import pathlib, xml.etree.ElementTree as ET
p = pathlib.Path("proof/junit.xml")
assert p.is_file() and p.stat().st_size > 0, "missing junit"
root = ET.parse(p).getroot()
# pytest may use <testsuites> or <testsuite>
suites = [root] if root.tag == "testsuite" else list(root)
assert suites, "no testsuite nodes"
print("junit_ok", "suites", len(suites))
PY

If parsing fails, your runner did not write JUnit. Fix the runner. Do not “ask the model to confirm.”

Stage 4 — Capture git reality, not git poetry

BASE=$(cat /tmp/agent-base.sha)
git diff --stat "$BASE" -- > proof/git_stat.txt
git diff --name-status "$BASE" -- > proof/git_name_status.txt
cat proof/git_name_status.txt

Verify:

test -s proof/git_stat.txt && test -f proof/git_name_status.txt && echo git_proof_ok

Empty name-status after a “huge refactor” recap? Then nothing changed, or you hashed the wrong base. Which one is it? Check the sha file before you argue with the model.

Stage 5 — Scan forbidden paths yourself

The agent will not volunteer that it touched .env. You have to look.

python - <<'PY'
from pathlib import Path
import json, fnmatch
done = json.loads(Path("proof/done.json").read_text())
changed = Path("proof/git_name_status.txt").read_text().splitlines()
paths = []
for line in changed:
    parts = line.split("\t")
    if len(parts) >= 2:
        paths.append(parts[-1])
hits = []
for p in paths:
    for g in done["forbidden_globs"]:
        if fnmatch.fnmatch(p, g):
            hits.append(f"{p} matches {g}")
Path("proof/forbidden.txt").write_text("\n".join(hits))
print("forbidden_hits", len(hits))
PY
cat proof/forbidden.txt

Verify: proof/forbidden.txt exists. Zero bytes means clean. Any bytes means fail closed. Do not negotiate with the glob.

Stage 6 — The verifier that ignores English

This is the artifact. One script. No LLM client. No “summarize the job.” It reads files or it dies.

Save as scripts/verify_proof_bundle.py:

#!/usr/bin/env python3
"""Fail closed on a proof/ directory. Proposed local gate; run it yourself."""
from __future__ import annotations

import hashlib
import json
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

PROOF = Path("proof")
DONE = PROOF / "done.json"
HASH_NAME = "PROOF.sha256"
HASHABLE = [
    "done.json",
    "junit.xml",
    "git_stat.txt",
    "git_name_status.txt",
    "forbidden.txt",
    "env_fingerprint.txt",
]


def die(msg: str) -> None:
    print(f"FAIL: {msg}", file=sys.stderr)
    raise SystemExit(1)


def load_done() -> dict:
    if not DONE.is_file():
        die("missing proof/done.json")
    try:
        data = json.loads(DONE.read_text())
    except json.JSONDecodeError as exc:
        die(f"done.json is not JSON: {exc}")
    required = (
        "job_id",
        "must_pass",
        "min_tests",
        "max_failures",
        "max_files_changed",
        "max_diff_lines",
        "junit_path",
    )
    for key in required:
        if key not in data:
            die(f"done.json missing {key}")
    return data


def parse_junit(path: Path) -> tuple[int, int]:
    if not path.is_file() or path.stat().st_size == 0:
        die(f"missing junit at {path}")
    root = ET.parse(path).getroot()
    nodes = [root] if root.tag.endswith("testsuite") else list(root)
    tests = 0
    fails = 0
    for node in nodes:
        tests += int(node.attrib.get("tests") or 0)
        fails += int(node.attrib.get("failures") or 0)
        fails += int(node.attrib.get("errors") or 0)
    return tests, fails


def count_diff(name_status: Path, stat_text: Path) -> tuple[int, int]:
    files = [ln for ln in name_status.read_text().splitlines() if ln.strip()]
    # git --stat footer looks like: " 3 files changed, 40 insertions(+), 2 deletions(-)"
    lines = 0
    for ln in stat_text.read_text().splitlines():
        if "changed" in ln and ("insertion" in ln or "deletion" in ln or "files changed" in ln):
            for token in ln.replace(",", "").split():
                if token.isdigit() and "file" not in ln.split(token)[0][-6:]:
                    # Count numeric tokens that are not the file count when possible.
                    pass
            nums = [int(t) for t in ln.replace(",", "").split() if t.isdigit()]
            if nums:
                # files, insertions, deletions — sum edits; fall back to file count only
                lines = sum(nums[1:]) if len(nums) > 1 else 0
    return len(files), lines


def main() -> None:
    done = load_done()
    for name in HASHABLE:
        p = PROOF / name
        if not p.is_file():
            die(f"missing {p}")

    tests, fails = parse_junit(Path(done["junit_path"]))
    if tests < int(done["min_tests"]):
        die(f"tests {tests} < min_tests {done['min_tests']}")
    if fails > int(done["max_failures"]):
        die(f"failures {fails} > max_failures {done['max_failures']}")

    files_changed, diff_lines = count_diff(
        PROOF / "git_name_status.txt", PROOF / "git_stat.txt"
    )
    if files_changed > int(done["max_files_changed"]):
        die(f"files {files_changed} > max_files_changed {done['max_files_changed']}")
    if diff_lines > int(done["max_diff_lines"]):
        die(f"diff_lines {diff_lines} > max_diff_lines {done['max_diff_lines']}")

    forbidden = (PROOF / "forbidden.txt").read_text().strip()
    if forbidden:
        die("forbidden paths:\n" + forbidden)

    digest = hashlib.sha256()
    for name in HASHABLE:
        digest.update(name.encode())
        digest.update(b"\0")
        digest.update((PROOF / name).read_bytes())
        digest.update(b"\0")
    expected = digest.hexdigest()
    hash_file = PROOF / HASH_NAME
    if not hash_file.is_file():
        hash_file.write_text(expected + "\n")
        print(f"WROTE {hash_file} {expected}")
        raise SystemExit(0)
    got = hash_file.read_text().strip()
    if got != expected:
        die(f"hash mismatch\n expected {expected}\n got      {got}")
    print(f"PASS job_id={done['job_id']} tests={tests} files={files_changed} hash={expected[:12]}")


if __name__ == "__main__":
    main()

Run it twice. First write. Second verify.

python scripts/verify_proof_bundle.py
python scripts/verify_proof_bundle.py

Verify: first run prints WROTE. Second run prints PASS. If the second run writes again, you are hashing a moving target. What changed between the two invocations? Find that file. Do not shrug.

Stage 7 — Ban the agent from the proof files

After the agent proposes a diff, re-check name-status against the contract.

python - <<'PY'
from pathlib import Path
import json, sys
done = json.loads(Path("proof/done.json").read_text())
banned = set(done["agent_must_not_touch"])
changed = []
for ln in Path("proof/git_name_status.txt").read_text().splitlines():
    parts = ln.split("\t")
    if len(parts) >= 2:
        changed.append(parts[-1])
hits = [p for p in changed if p in banned]
if hits:
    print("FAIL agent touched proof files:", hits)
    sys.exit(1)
print("proof_files_untouched")
PY

Verify: proof_files_untouched. If this fails, the recap is worse than useless. The agent forged the exam.

Re-run pytest yourself. Overwrite proof/junit.xml. Recompute PROOF.sha256 by deleting it and running the verifier again. The model does not get a vote.

rm -f proof/PROOF.sha256
pytest tests/test_invoices.py --junitxml=proof/junit.xml -q
python scripts/verify_proof_bundle.py
python scripts/verify_proof_bundle.py

Stage 8 — Optional remote run, local merge

Laptop fans screaming? Use a remote shell for the agent attempt, then copy the repo back and run stages 3–7 on a machine you control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode here only as a place that offers free model access and a free server option for that remote attempt. I am not attaching model names, quotas, hardware, or a promise that the free tier lasts forever. I cannot. Treat those two availability claims as “exists today, re-check before you depend on them.”

The important part is architectural, not branded: the remote box may run the model. Your merge machine still hashes the proof bundle. If you cannot scp or pull the tree and re-run scripts/verify_proof_bundle.py cold, you do not have a gate. You have a demo.

One check after a remote hop:

# proposed: after you fetch the job tree back
git rev-parse HEAD
python scripts/verify_proof_bundle.py

If the hash file traveled with the agent’s diff, delete it. You write the hash. Nobody else does.

Decision table (keep this next to the script)

Signal	Looks like	Action
Recap says green, no junit.xml	Story	Fail
JUnit green, forbidden.txt non-empty	Leak	Fail
JUnit green, files_changed over budget	Scope creep	Fail
Agent diff includes proof/PROOF.sha256	Forgery	Fail
Verifier PASS twice, hash stable	Boring	Eligible to merge
Verifier PASS, then hash flips	Flaky tree	Do not merge

Print the table. Tape it to the PR template. Recaps do not get a row.

Limitations, and who should not bother

This does not prove the product is correct. It proves the job met a contract you were willing to write in JSON. Bad contracts rubber-stamp bad code. If min_tests is 1 and must_pass points at a trivial file, congratulations: you automated theater.

JUnit parsing is only as honest as the runner. A custom harness that writes <testsuite tests="99" failures="0"/> by hand will sail through. So pin the runner in env_fingerprint.txt and refuse unknown pytest versions if you need that tightness.

git diff --stat line counts are a budget, not a quality score. A one-line change can still delete production data. Pair this with whatever path allowlist or git apply --check gate you already use. This article is not that gate.

Skip this workflow if you are pairing live and reading every hunk. Skip it if the repo cannot run tests without production credentials. Skip it if you need the model to “just ship.” That last group is exactly who should not hold merge buttons.

What I actually read at merge time

Not the recap. Not the chat. Three files: proof/git_name_status.txt, proof/junit.xml, PROOF.sha256. Then the verifier stdout.

If those four disagree, I already know which one I throw away. The English.

Want a remote attempt on a free server with free model access? Fine. Bring the tree back and make this script print PASS twice. That is the only invitation I have.