Reachability Receipts: Prove an Agent Patch's Green Build Came From Tests That Touched It

A green build is a fact about your test suite, not about the patch. If no executed test line intersects the lines the patch changed, the pipeline measured the repository's steady state and reported it as approval. The receipt format below records that intersection explicitly, and it closes the most common abuse of a flaky-test freeze: freezing a test that never touched the patch and then counting the green result as evidence.

The Three Fields a Receipt Needs

Every merge decision in this workflow reduces to three sets.

1. Changed lines — the exact new-side line ranges of the patch, not a diffstat.
2. Per-test executed lines — which lines each test actually ran.
3. The freeze set — tests currently quarantined as flaky, with an expiry.

Everything else is bookkeeping. The verdict comes from intersecting the first two, then asking the third whether that intersection is trustworthy.

Step 1: Derive Changed Line Ranges, Not Counts

git diff --stat counts insertions and hides where they landed. A patch can add forty lines and still touch none of the functions your properties cover. Use zero context so each hunk is exactly the changed region.

git fetch origin main --depth=1
git diff -U0 --no-color origin/main...HEAD > patch.diff
grep -c '^@@' patch.diff

The third command tells you how many changed regions you are dealing with. If that number is zero on a non-empty patch, your merge base is wrong and the rest of this workflow is measuring nothing.

Step 2: Collect Per-Test Line Hits

Line-level execution is the only signal here that cannot be faked by a passing suite. coverage.py records which test covered which line when dynamic contexts are enabled.

# .coveragerc
[run]
dynamic_context = test_function

coverage run -m pytest -q
coverage json --show-contexts -o coverage-contexts.json
python tools/contexts_to_trace.py coverage-contexts.json > trace.jsonl

A minimal adapter from that JSON to the JSONL the receipt consumes:

# tools/contexts_to_trace.py
import json, sys

doc = json.load(open(sys.argv[1]))
by_test = {}
for path, filedata in doc['files'].items():
    for ln, contexts in (filedata.get('contexts') or {}).items():
        for ctx in contexts:
            by_test.setdefault(ctx, set()).add((path, int(ln)))

for test_id, hits in sorted(by_test.items()):
    print(json.dumps({'test_id': test_id, 'hits': sorted(map(list, hits)), 'status': 'unknown'}))

Two caveats matter. Contexts prove coverage, not outcome, so the status field above is a placeholder that must be merged from your JUnit XML before the receipt is trusted. And --show-contexts depends on your installed coverage version; check coverage --version on the runner before you build a gate on it. If your runner cannot emit per-test hits at all, stop here rather than approximating.

Step 3: The Receipt Script

This is a reference implementation. I have not executed it against your repository, and the sample output later in this article is fabricated to show the shape only.

#!/usr/bin/env python3
# tools/reachability_receipt.py
import argparse, hashlib, json, re, sys
from pathlib import Path

HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@')

def changed_ranges(diff_text):
    ranges, current = {}, None
    for line in diff_text.splitlines():
        if line.startswith('+++ '):
            path = line[4:].strip()
            path = path[2:] if path.startswith('b/') else path
            current = None if path == '/dev/null' else path
            if current:
                ranges.setdefault(current, [])
            continue
        m = HUNK.match(line)
        if m and current:
            start = int(m.group(1))
            count = int(m.group(2) or 1)
            if count:
                ranges[current].append((start, start + count - 1))
    return ranges

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--diff', required=True)
    ap.add_argument('--trace', required=True)
    ap.add_argument('--freeze')
    ap.add_argument('--require-property', action='append', default=[])
    args = ap.parse_args()

    ranges = changed_ranges(Path(args.diff).read_text())
    changed = {(f, ln) for f, rs in ranges.items() for a, b in rs for ln in range(a, b + 1)}

    freeze = {}
    if args.freeze:
        raw = json.loads(Path(args.freeze).read_text())
        freeze = raw if isinstance(raw, dict) else {t: {} for t in raw}

    tests = []
    for raw_line in Path(args.trace).read_text().splitlines():
        if not raw_line.strip():
            continue
        rec = json.loads(raw_line)
        test_id = rec['test_id']
        hits = {(f, int(ln)) for f, ln in rec.get('hits', [])}
        touched = hits & changed
        if test_id in freeze:
            verdict = 'frozen-blocking' if touched else 'frozen-irrelevant'
        elif touched:
            verdict = 'evidence'
        else:
            verdict = 'no-reach'
        tests.append({'test_id': test_id, 'status': rec.get('status', 'unknown'),
                      'touched': len(touched), 'verdict': verdict})

    properties = []
    for name in args.require_property:
        support = [t['test_id'] for t in tests
                   if t['verdict'] == 'evidence' and name in t['test_id'] and t['status'] == 'pass']
        properties.append({'property': name, 'supporting_tests': support, 'ok': bool(support)})

    payload = {
        'changed_lines': len(changed),
        'files': {f: sorted(rs) for f, rs in sorted(ranges.items())},
        'tests': sorted(tests, key=lambda t: t['test_id']),
        'properties': properties,
    }
    blob = json.dumps(payload, sort_keys=True, separators=(',', ':')).encode()
    receipt = {'payload': payload, 'digest': hashlib.sha256(blob).hexdigest()}
    json.dump(receipt, sys.stdout, indent=2, sort_keys=True)
    sys.stdout.write('\n')

    blocked = any(t['verdict'] == 'frozen-blocking' for t in tests)
    ok = all(p['ok'] for p in properties) and not blocked and len(changed) > 0
    return 0 if ok else 1

if __name__ == '__main__':
    sys.exit(main())

The digest covers a canonicalised payload, so two runs over the same diff, trace, and freeze produce the same hash. That is what lets a reviewer ask a specific question later: which evidence did this merge actually rest on?

Step 4: Read the Verdicts Against a Fixed Table

python tools/reachability_receipt.py \
  --diff patch.diff \
  --trace trace.jsonl \
  --freeze freeze.json \
  --require-property proration_total \
  --require-property refund_idempotency > receipt.json
echo $?

Verdict	Meaning	Merge action
evidence	Executed a changed line and reported a result	Counts toward the property
no-reach	Passed, touched nothing the patch changed	Ignore; never count as support
frozen-blocking	In the freeze set and touched changed lines	Do not merge; evidence is contaminated
frozen-irrelevant	In the freeze set, touched nothing changed	Keep frozen; it must not gate this patch

The fourth row is the one teams usually get wrong. A flaky test that never touched the patch should stay frozen without blocking anything, because freezing it was a scheduling decision, not a statement about this change. The third row is the opposite: once a known-flaky test intersects the patch, no one can separate the flake from the regression on those lines.

An illustrative receipt fragment, fabricated for shape:

{
  "payload": {
    "changed_lines": 12,
    "properties": [
      {"ok": true, "property": "proration_total", "supporting_tests": ["tests/test_billing.py::test_proration_total"]},
      {"ok": false, "property": "refund_idempotency", "supporting_tests": []}
    ],
    "tests": [
      {"status": "pass", "test_id": "tests/test_billing.py::test_proration_total", "touched": 4, "verdict": "evidence"},
      {"status": "pass", "test_id": "tests/test_misc.py::test_config_parses", "touched": 0, "verdict": "no-reach"},
      {"status": "failed", "test_id": "tests/test_rates.py::test_fx_cache", "touched": 2, "verdict": "frozen-blocking"}
    ]
  },
  "digest": "0000000000000000000000000000000000000000000000000000000000000000"
}

That exit code is non-zero for two independent reasons: a declared property has no supporting test, and a frozen test touched the patch.

Step 5: Self-Test the Gate Before You Trust It

A gate that rejects nothing is decoration. Run these three cases against a scratch branch before wiring the script into CI.

1. Reformat-only patch. Whitespace changes across one file, no behaviour change. Expected: changed_lines above zero, evidence count zero, exit non-zero.
2. Single-line behaviour patch. One condition inverted in a function a property covers. Expected: at least one evidence test, exit zero when the property is declared.
3. Freeze scoping. Add the flaky test to freeze.json, run twice — once on a patch that touches its lines, once on a patch that does not. Expected: frozen-blocking appears only in the first run.

Freeze entries should carry an expiry so quarantine does not become permanent:

{"tests/test_rates.py::test_fx_cache": {"class": "flake", "expires": "2026-11-01"}}

The expiry date is not enforced by the script above. Reject any freeze entry whose expiry has passed in a separate check, otherwise the freeze set becomes an allowlist for failures.

Limitations and Who Should Not Use This

• Execution is not falsifiability. A line hit means the line ran, not that an assertion could have failed. Pair this with mutation testing or a tautology classification step; a reachability receipt alone will happily certify an assertion-free test.
• Rename and move handling. With -U0, moved code can appear as a full rewrite, inflating the changed set and producing false evidence verdicts.
• Cost. Dynamic contexts slow the suite and produce large trace files. On a large monorepo, restrict tracing to the packages the patch touches.
• Language coverage. This depends on a line-level tracer. Compiled or out-of-process code needs a different mechanism, and none of the script above applies.
• Do not use it if your repository has one meaningful test file, if your runner cannot emit per-test hits, or if the receipt is destined to become a checkbox nobody reads. Manual review of a handful of tests is cheaper and more honest.

Where MonkeyCode Fits

One part of this workflow is genuinely tedious: naming the properties for a patch you did not write. The account operator states that MonkeyCode offers free model access, which I used as a drafting step — propose one property name per changed function, then a human accepts, renames, or deletes each one. The --require-property list must be human-approved before it becomes a gate; a model-generated property that always passes is exactly the tautology problem this workflow exists to catch.

The receipt job is also a good fit for a free server option, per the operator, because it is short, I/O-heavy, and produces a JSON artifact rather than a service. Running it off the developer machine keeps the triage out of the local loop and makes the digest reproducible from a clean checkout.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Remove every product mention above and the workflow still stands. That is the test I apply before writing this section at all.

Closing

The receipt does not make a patch correct, and it does not replace property design. It does make one failure mode impossible to paper over: a green build whose tests never reached the changed lines. Start with a single patch, produce one no-reach verdict, and decide whether you want that verdict gating your merges. If you want to prototype the triage step on a free server before committing CI minutes, the receipt script is small enough to run there first.