Test Selection Is a Claim: Audit Which Tests an Agent Patch Actually Ran

An agent patch that reports 42 passed has told you that 42 tests ran and passed. It has not told you that your suite passed. When the repository collects 900 tests and the patch executed 42, the other 858 are not evidence — they are an assumption you have not priced yet.

This post makes that scope claim auditable. The artifact is a small evidence bundle plus four gates: a selection/execution ledger, a reachability check from changed lines to owning tests, a fixture read-set closure check, and a hash-frozen quarantine ledger so a patch cannot quietly widen the set of tests it is allowed to skip.

The three counts every patch produces

A test run generates at least three different numbers, and patches usually quote only the last one.

Count	Produced by	What it can prove
collected	collection phase	the tests still exist in the tree
selected	-k, --lf, impacted-test tooling	the scope the patch claims to cover
executed (call phase)	run reports	the tests ran to a verdict
skipped / xfail	run reports	nothing, until triaged

The gate is a containment check: executed ⊆ selected ⊆ collected. If a test executes that was never selected, the harness is lying and no verdict from that run is trustworthy. If a changed line has no owning test in executed, the patch has an unreachable change and needs a written justification.

Step 1 — Record selection and execution separately

Collection and the call phase are different events with different payloads, so write them to different files. This is a reference plugin; adapt the paths and confirm the hooks against your pytest version before relying on it.

# conftest.py — evidence recorder (reference implementation, adapt paths)
import json, os, pathlib

ART = pathlib.Path(os.environ.get("EVIDENCE_DIR", ".evidence"))
ART.mkdir(parents=True, exist_ok=True)
COLLECTED = ART / "collected.jsonl"
EXECUTED  = ART / "executed.jsonl"


def pytest_collection_modifyitems(config, items):
    with COLLECTED.open("w") as fh:
        for item in items:
            fh.write(json.dumps({
                "nodeid": item.nodeid,
                "path": str(item.path),
                "markers": [m.name for m in item.iter_markers()],
            }) + "\n")


def pytest_runtest_logreport(report):
    if report.when != "call":
        return
    with EXECUTED.open("a") as fh:
        fh.write(json.dumps({
            "nodeid": report.nodeid,
            "outcome": report.outcome,
            "skipped": report.skipped,
            "duration": report.duration,
        }) + "\n")

Run the patch's own command, then check containment before reading the pass count:

EVIDENCE_DIR=.evidence pytest -q -m "not slow" --junitxml=.evidence/junit.xml
python - <<'PY'
import json, pathlib
art = pathlib.Path(".evidence")
collected = {json.loads(l)["nodeid"] for l in (art/"collected.jsonl").read_text().splitlines()}
executed  = {json.loads(l)["nodeid"] for l in (art/"executed.jsonl").read_text().splitlines() if l}
outliers = sorted(executed - collected)
print(f"collected={len(collected)} executed={len(executed)} uncollected_executed={len(outliers)}")
for n in outliers[:10]:
    print("  !", n)
PY

Skipped tests belong in executed but must be counted in their own column. A patch that converts a failing test into a skipped test has changed the verdict vocabulary, not the behavior.

Step 2 — Map changed lines to owning tests

The patch's diff is the input; per-test coverage contexts are the index. Record a context per test node, export JSON, then intersect.

coverage run -m pytest -q
coverage json -o .evidence/coverage.json

# reachability.py — reference implementation; verify the JSON schema first.
import json, re, subprocess, sys

diff = subprocess.run(["git", "diff", "-U0", "origin/main...HEAD"],
                      capture_output=True, text=True, check=True).stdout
HUNK = re.compile(r"^\+\+\+ b/(?P<path>.+)$"
                  r"|^@@ -\d+(?:,\d+)? \+(?P<start>\d+)(?:,(?P<count>\d+))? @@")

changed, path = {}, None
for line in diff.splitlines():
    m = HUNK.match(line)
    if not m:
        continue
    if m.group("path"):
        path = m.group("path")
        changed.setdefault(path, set())
    elif path:
        start = int(m.group("start"))
        count = int(m.group("count") or 1)
        changed[path].update(range(start, start + count))

cov = json.load(open(".evidence/coverage.json"))
unowned = []
for path, lines in changed.items():
    contexts = cov["files"].get(path, {}).get("contexts", {})
    for ln in sorted(lines):
        if not contexts.get(str(ln)):
            unowned.append(f"{path}:{ln}")

print(json.dumps({"changed_lines": sum(map(len, changed.values())),
                  "lines_with_no_owning_test": unowned}, indent=2))
sys.exit(1 if unowned else 0)

Two caveats matter. The contexts key is only present when contexts were recorded, and its shape has changed across coverage.py releases — check the actual JSON you produce. And a brand-new line in a brand-new file has no context at all, so treat "new file" as "needs at least one executing test that imports the module," not as an automatic failure.

Step 3 — Fixture read-set closure

Patches fail in two directions here: they rewrite a fixture so a test passes, or they rewrite a fixture and never run the tests that read it. Both are detectable if you keep a fixture registry keyed by fixture name.

pytest --fixtures-per-test -q > .evidence/fixture_usage.txt

# .evidence/fixture_readsets.yaml — maintained by you, not generated
# Only fixtures whose on-disk inputs are stable enough to hash belong here.
db_session:
  reads:
    - tests/fixtures/schema.sql
    - tests/fixtures/seed_users.json
clock:
  reads:
    - tests/fixtures/clock.json

The closure rule: if the patch touched any file in a fixture's reads list, every test declaring that fixture (per fixture_usage.txt) must appear in executed.jsonl. Otherwise the gate blocks and the patch author writes an entry in .evidence/scope_exceptions.yaml naming the test, the reason, and an expiry date. Exceptions are reviewed, not silent.

Step 4 — Freeze the flaky quarantine

This is the highest-value gate, because an agent under pressure has a cheap escape hatch: mark the failing test as skipped. Quarantine is legitimate — flaky tests destroy signal — but the list must not be editable by the patch it protects.

# .evidence/quarantine.yaml — content-hashed in CI
- nodeid: tests/test_ingest.py::test_retry_budget
  reason: timing-dependent under parallel execution
  added: 2026-09-02
  expires: 2026-11-02
  tracker: ISSUE-4412

Change to the ledger	Policy
Add a new quarantined test	Blocked; requires human sign-off
Extend an existing expires	Blocked; requires human sign-off
Remove an entry after N consecutive green runs	Auto-allowed
Edit reason or tracker only	Auto-allowed

Hash the file in CI and compare against the hash stored on the base branch. Additions fail the build. Removals pass. That single rule removes the most common way an agent patch hides a regression it introduced.

Step 5 — Bundle the evidence (optional second reader)

Everything above writes files, so the bundle is mechanical:

File	Producer	Gate it feeds
collected.jsonl, executed.jsonl	recorder plugin	containment
coverage.json	coverage contexts	reachability
fixture_usage.txt	pytest CLI	closure
quarantine.yaml + hash	CI	freeze

Once the bundle exists, a second model can act as an adversarial reader over JSON instead of prose. Ask it fixed questions — "list every changed line with no owning test," "which executed test would fail if this line were reverted?" — and treat its answers as leads, not verdicts.

Running that review step needs a clean environment and a model you are not paying per token for while iterating on the questionnaire. MonkeyCode's free model access and free server option are a low-commitment way to run the audit on a scratch clone, with free limits changing over time, so check current terms before depending on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should not use this

• Reachability is line-level. A changed line inside a covered region can still be logically untested; this gate narrows the search, it does not conclude it.
• The selection audit says nothing about whether an assertion is meaningful, and nothing about whether the collected inventory shrank. Pair it with suite-inventory diffing and falsifiability checks.
• Per-test contexts add runtime and storage. On a large suite, record contexts only on the changed test files.
• Skip this if your full suite runs in under a couple of minutes — then run everything and skip selection entirely. The audit only pays for itself where selection exists.
• Skip it if nobody owns the fixture registry. A stale fixture_readsets.yaml produces blocked patches for the wrong reason, and teams learn to bypass gates that cry wolf.

The cheapest gate to add this week is the quarantine hash, because it needs no coverage tooling and one CI step. The containment check is second. Reachability and fixture closure are worth it only after the first two are boring and automatic.