Attribute Coverage to Hunks, Not Files: An Ownership Map for Agent Patches

A patch can show 100% coverage on its changed lines and still be untested. The number that matters is not how many changed lines executed, but which test would fail if the change were wrong — and whether that test was added by the patch or inherited from the repository.

This post builds a hunk ownership map: per-test coverage contexts mapped onto diff hunks, classified as OWNED, INCIDENTAL, or ORPHAN, with a fixture-reach audit and a stability precondition. The artifact is a script you run in your own repository.

The failure mode in one example

An agent adds a retry wrapper and a backoff cap. It also touches a helper that an old smoke test happens to exercise. Line coverage on the diff looks excellent.

What actually happened: the only test touching lines 41–48 is tests/test_client.py::test_get_ok, written two years ago, which never asserts anything about retries. Lines 52–55 have no covering test at all. Only lines 60–63 are protected by a test the patch itself introduced.

Hunk	Instrumented lines	Owner	Would fail on wrong behavior?
src/retry.py:41-48	executed	pre-existing smoke test	no
src/retry.py:52-55	not executed	none	no
src/retry.py:60-63	executed	new test in patch	yes

File-level coverage hides all three behind one percentage.

Step 1 — Collect per-test coverage contexts

coverage.py records a context per measurement, and pytest-cov can set that context to the running test id. This is the whole trick; without contexts you only get file-level truth.

# .coveragerc  (or [tool.coverage.run] in pyproject.toml)
# [run]
# relative_files = True

pytest --cov=src --cov-context=test
coverage json --show-contexts -o coverage.json

relative_files = True matters: the JSON keys must be repo-relative so they line up with git diff paths. If you use pytest-xdist, expect per-worker data files; run coverage combine before coverage json if contexts look empty. Context support and the --show-contexts JSON option have version floors, so confirm both against the coverage.py and pytest-cov versions you are actually running.

Step 2 — Map hunks to lines

Parse the diff with zero context lines. You want added-line ranges, not readability.

git diff -U0 origin/main...HEAD

-U0 produces @@ -a,b +c,d @@ headers where c,d is exactly the added block in the new file. Deleted-only hunks appear as +c,0 and must be skipped — they have no lines to own.

Step 3 — Classify every hunk

The script below joins the two inputs. Run it as a local gate; it is a starting point, not a packaged tool, and its behavior depends on your coverage.py version.

#!/usr/bin/env python3
"""hunk_ownership.py — map patch hunks to the test contexts that executed them."""
import json, re, subprocess, sys
from collections import defaultdict

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")
NEW_TEST = re.compile(r"^(?:tests?|spec)/.*\.py$")


def changed_hunks(rev_range):
    out = subprocess.run(["git", "diff", "-U0", rev_range],
                         capture_output=True, text=True, check=True).stdout
    hunks, path = defaultdict(list), None
    for line in out.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
        elif line.startswith("@@"):
            m = HUNK.match(line)
            if not m:
                continue
            start = int(m.group(1))
            count = int(m.group(2)) if m.group(2) is not None else 1
            if count:                      # skip deleted-only hunks
                hunks[path].append((start, start + count - 1))
    return dict(hunks)


def added_test_files(rev_range):
    out = subprocess.run(["git", "diff", "-U0", "--diff-filter=A",
                          "--name-only", rev_range],
                         capture_output=True, text=True, check=True).stdout
    return {p for p in out.split() if NEW_TEST.match(p)}


def coverage_index(cov_path):
    data = json.load(open(cov_path))
    idx = {}
    for fname, fdata in data["files"].items():
        for lineno, contexts in (fdata.get("contexts") or {}).items():
            idx[(fname, int(lineno))] = set(contexts)
    return idx


def classify(hunks, cov, new_tests):
    rows = []
    for path, ranges in hunks.items():
        for lo, hi in ranges:
            owners = set()
            for line in range(lo, hi + 1):
                owners |= cov.get((path, line), set())
            owned = any(o.startswith(t) for o in owners for t in new_tests)
            kind = "ORPHAN" if not owners else ("OWNED" if owned else "INCIDENTAL")
            rows.append((path, f"{lo}-{hi}", kind, sorted(owners)))
    return rows


if __name__ == "__main__":
    rev = sys.argv[1] if len(sys.argv) > 1 else "origin/main...HEAD"
    rows = classify(changed_hunks(rev), coverage_index("coverage.json"),
                    added_test_files(rev))
    for path, span, kind, owners in rows:
        print(f"{kind:11} {path}:{span}  owners={','.join(owners) or '-'}")
    sys.exit(1 if any(r[2] != "OWNED" for r in rows) else 0)

Illustrative report shape (synthetic values, not a measurement of any real repository):

INCIDENTAL  src/retry.py:41-48  owners=tests/test_client.py::test_get_ok
ORPHAN      src/retry.py:52-55  owners=-
OWNED       src/retry.py:60-63  owners=tests/test_retry.py::test_backoff_caps

Step 4 — Audit fixture reach before you trust OWNED

Many INCIDENTAL verdicts come from a shared fixture that makes almost any test execute almost any code path. Count how many tests depend on each fixture, then intersect that list with fixtures the patch touched.

import re, subprocess
from collections import defaultdict

out = subprocess.run(["pytest", "--fixtures-per-test", "-q"],
                     capture_output=True, text=True).stdout
reach, current = defaultdict(set), None
for line in out.splitlines():
    m = re.match(r"^([\w/\-]+\.py)::(\S+)", line)
    if m:
        current = f"{m.group(1)}::{m.group(2)}"
    elif current and line.strip().startswith("-- "):
        reach[line.strip()[3:].split()[0]].add(current)

for fixture, users in sorted(reach.items(), key=lambda kv: -len(kv[1])):
    print(f"{len(users):4}  {fixture}")

The --fixtures-per-test layout has shifted across pytest releases, so validate this parse against your version before gating on it. A fixture reachable from most of the suite and modified by the patch is a strong signal that the patch changed test infrastructure, not just production behavior — review it as a test change, and read the diff twice.

Step 5 — Make stability a precondition, not an afterthought

Ownership is only meaningful when the covering test returns the same verdict every time. Before you accept an OWNED hunk, run just its covering files repeatedly with a fixed shuffle seed.

# requires pytest-repeat and pytest-randomly; confirm flags for your versions
pytest -q tests/test_retry.py --count=5 --randomly-seed=1729

If the verdict flips, move the test out of the owner set and treat every hunk it covered as ORPHAN until it is fixed. A test that sometimes passes is not evidence; it is noise with a merge button.

Step 6 — Decide with a fixed table

Hunk class	Coverage on changed lines	Owner	Gate action
OWNED	executed	test added/changed as a test by this patch	accept, record owner ids in the PR
INCIDENTAL	executed	pre-existing test only	block; probe the hunk or add an assertion
ORPHAN	not executed	none	block; require a new test or a written waiver
WAIVED	any	explicit human waiver	accept; waiver expires with the next release tag

INCIDENTAL is the class that pays for the whole exercise. It is where a green CI run and a confident review collide. For those hunks, run a per-hunk behavioral probe — the differential revert technique from the earlier gate design in this series — and promote the hunk to WAIVED only with a recorded reviewer.

Running the matrix somewhere it does not cost CI minutes

Per-test contexts multiply the size of the coverage data and the runtime, and the five-times stability pass multiplies it again. That is an awkward load to put on a metered CI plan.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, per the operator; I have not verified quotas, hardware, or duration, and you should read the current plan page before relying on either. Two uses fit this workflow honestly. First, run the coverage-context matrix and the stability repeats on the free server, so the expensive collection stays off your paid runners. Second, feed coverage.json plus the ownership report to a free model to draft a PR comment and propose candidate property tests for ORPHAN hunks — then review those proposals yourself, because a generated test that asserts the observed behavior of a wrong patch is worse than no test.

Limitations and who should skip this

• Small suites. Under roughly a hundred tests, reading the diff is faster than building the map. The tooling overhead exceeds the savings.
• Generated and compiled code. Line-level coverage is a poor proxy across transpiled output, macros, and template instantiation; hunk boundaries in the source rarely match executable boundaries.
• Context support is not universal. coverage.py contexts are the mature path here; if your language's coverage tool has no per-test context concept, this method degrades to file-level guessing.
• Teams already running per-PR mutation testing. Mutation gives a stronger ownership signal directly. Adding this on top duplicates cost for little new information.
• Parallel collection. With pytest-xdist, context data must be combined before analysis, and per-worker context names can diverge. Budget for that or run serially on the free server.

The weekly loop

1. Generate the patch diff with git diff -U0.
2. Run pytest --cov-context=test and emit coverage.json --show-contexts.
3. Classify every hunk; fail the gate on ORPHAN and INCIDENTAL.
4. Audit fixture reach for any fixture the patch touched.
5. Re-run covering files five times with a fixed seed; demote unstable owners.
6. Record owner ids and waivers in the PR body so the next reader sees the evidence, not the percentage.

Coverage tells you a line ran. Ownership tells you a test would have complained. If you adopt one change from this post, make it the exit code on ORPHAN.