Coverage Is a Claim, Not Evidence: A Hunk-Level Ledger for Agent Patches

A green suite tells you the tests passed. It does not tell you which changed lines made them pass. Agent patches exploit exactly that gap: the suite goes green, the merge happens, and nobody can say which assertion actually witnessed the new behavior. This post builds a small ledger that grades every added line by the strength of its witness and refuses the lines that have none.

The artifact is about 80 lines of Python. It reads a diff, joins it against per-test coverage contexts, and fails closed on anything it cannot classify. Below are the steps, the code, a four-case calibration set, and the cases where you should skip it.

1. The failure mode is unfalsifiable coverage

File-level coverage answers "was this file imported?" It rarely answers "did the patch's new branch get executed by a test that would notice if the branch were wrong?" Those are different questions.

Agents make the gap wider because they generate tests and code in the same motion. A generated test that touches the new line satisfies coverage while asserting almost nothing. The gate sees a number go up and calls it evidence.

So the unit of accounting should not be the file or the pull request. It should be the added line, and the witness attached to it.

2. Step one: extract hunks, not files

Start with the diff. --unified=0 gives you exact added-line ranges with no context noise.

# evidence_ledger.py (part 1)
import json, re, subprocess, sys
from datetime import date
from pathlib import Path

BASE = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")


def changed_lines():
    diff = subprocess.run(
        ["git", "diff", "--unified=0", "--no-color", f"{BASE}...HEAD"],
        check=True, capture_output=True, text=True).stdout
    out, path = {}, None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            path = line[len("+++ b/"):]
        elif line.startswith("@@") and path:
            m = HUNK.match(line)
            start, n = int(m.group(1)), int(m.group(2) or 1)
            if n:                      # deletions are not new surface
                out.setdefault(path, set()).update(range(start, start + n))
    return out

Deliberate limitation: pure deletions produce no rows. Removal is checked by the suite as a whole; the ledger is for new surface.

3. Step two: collect per-test coverage contexts

You need to know which test executed line 42, not just that line 42 ran. pytest-cov can record that with dynamic contexts, and coverage json can emit them.

pytest --cov=src --cov-context=test
coverage json --show-contexts -o coverage.json

Each line then carries context strings like tests/test_pricing.py::test_rounds|run. Strip the phase suffix before matching.

# evidence_ledger.py (part 2)
def contexts(path="coverage.json"):
    files = json.loads(Path(path).read_text())["data"]["files"]
    idx = {}
    for src, blob in files.items():
        for lineno, ctxs in blob.get("contexts", {}).items():
            for ctx in ctxs:
                idx.setdefault(src, {}).setdefault(int(lineno), set()).add(ctx.split("|")[0])
    return idx

If your package reports absolute paths, add a [paths] section to .coveragerc so source paths match the diff paths. Ten minutes of path normalization saves an hour of debugging empty joins.

4. Step three: grade the witness, not the count

A witness is only as good as its class. Keep the registry in a reviewed file, not in a code comment.

Witness class	Typical source	Weight	Rationale
property	Hypothesis / stateful invariant	1.0	Fails on inputs nobody wrote down
test	Deterministic unit or integration test	1.0	Pins one observed behavior
quarantined	Test on the flaky registry	0.0	Its result carries no information
unknown	Not listed in the registry	0.0	Unclassified tests drift silently

The important row is quarantined. If a line is covered only by a frozen flaky test, it has no witness — the green is noise. Downgrade it rather than accepting it. Fixture-backed tests are legitimate witnesses, but the ledger does not check that their fixture state contract holds; pair it with whatever fixture-lock check your repo already runs.

# evidence_ledger.py (part 3)
WEIGHT = {"property": 1.0, "test": 1.0, "quarantined": 0.0}


def grade(src, lines, ctx, registry):
    rows = {}
    for n in sorted(lines):
        tests = ctx.get(src, {}).get(n, set())
        best = max((WEIGHT.get(registry.get(t, "unknown"), 0.0) for t in tests), default=0.0)
        rows[n] = (best, sorted(tests))
    return rows

5. Step four: gate, with expiring allowances

Fail closed, but leave a pressure valve. An orphan line is allowed only through a written grant with an owner and an expiry date. Expired grants revert to failures automatically.

# evidence_ledger.py (part 4)
REGISTRY, ALLOW = Path("evidence.json"), Path("allow_orphans.json")


def main():
    registry = json.loads(REGISTRY.read_text())
    allow = json.loads(ALLOW.read_text()) if ALLOW.exists() else {}
    ctx, today = contexts(), date.today().isoformat()
    orphans, tolerated = [], []
    for src, lines in sorted(changed_lines().items()):
        if not src.endswith(".py") or "/tests/" in src:
            continue
        for n, (weight, tests) in grade(src, lines, ctx, registry).items():
            if weight > 0:
                continue
            key, grant = f"{src}:{n}", allow.get(f"{src}:{n}")
            if grant and grant["expires"] >= today:
                tolerated.append(key)
            else:
                orphans.append((key, tests or ["<none>"]))
    for key, tests in orphans:
        print(f"ORPHAN {key}  witnessed_by={tests}")
    print(f"orphans={len(orphans)} tolerated={len(tolerated)}")
    return 1 if orphans else 0


if __name__ == "__main__":
    sys.exit(main())

Output is deliberately boring: one line per orphan, then a count. Boring output gets read in CI logs.

6. Step five: buy evidence where it is cheapest

An orphan hunk usually needs one of two things: a narrow unit test, or a property that pins the invariant the hunk was supposed to preserve. Properties are often cheaper and catch more, so draft them first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A workable loop uses MonkeyCode's free model access to propose candidate invariants from the hunk itself, and the free server option to run the ledger plus the property harness without provisioning a separate box. Treat that as availability, not as a performance claim: I have not benchmarked the models here, and generated properties are proposals that a human must review before they enter the registry. Verify current terms before you depend on them.

# tests/test_pricing_properties.py — candidate drafted from a diff hunk
from decimal import Decimal
from hypothesis import given, strategies as st
from pricing import round_half_up


@given(st.decimals(min_value=Decimal("-1000"), max_value=Decimal("1000"), places=2))
def test_rounding_never_decreases_with_input(amount):
    assert round_half_up(amount) <= round_half_up(amount + Decimal("0.01"))

One caveat matters more than the rest: a property that also passes on the parent commit is not a witness, it is decoration. Confirm each new property fails against the pre-patch tree before registering it as property.

7. Calibrate the ledger with four known outcomes

Before trusting the gate, prove it can fail. Build four throwaway branches and record the expected verdict.

Case	Patch	Expected
A	Adds a line covered by a test witness	exit 0
B	Adds a line with no covering test	exit 1, one ORPHAN
C	Adds a line covered only by a quarantined test	exit 1, weight downgraded
D	Case B plus an allowance that expired yesterday	exit 1

If case C passes, your registry is misclassified. If case D passes, your date comparison is wrong — a common bug when one side is a string and the other a date.

8. Limitations, and who should not use this

Dynamic coverage contexts slow the suite; budget for it on large repositories. The script above assumes Python and a coverage.py JSON report, and it says nothing about deleted behavior, API compatibility, or whether the test witnesses you registered are themselves meaningful. It also cannot see config, SQL, or template changes that shift runtime behavior without touching .py lines.

Skip this if you have no coverage instrumentation, if your diffs routinely touch hundreds of files, or if nobody will own the allowance file — an unowned allowance list becomes a permanent allowlist within a month. For a first cut, ship Step 1 alone and print the hunk map for a week. You will learn more about your gate from that log than from any policy document.

If you build the whole ledger, the loop I would start with is: extractor, contexts, registry, gate — in that order, one pull request each. MonkeyCode's free model access and free server option are a reasonable place to run the property-drafting loop if you do not already have a sandbox for it; the extractor is worth shipping either way.