Which Check Would Have Caught It? A Fault-to-Test Attribution Matrix for Agent Patches

A green suite proves the patch passes. It does not prove any specific check would have failed if the patch were wrong. Those are different claims, and agent-written patches blur them constantly: a new test appears, the suite turns green, and nobody asks which assertion is load-bearing.

This is the audit I run before trusting that green. I inject a small catalog of plausible faults, run each one against each test in isolation, and record which single test — if any — catches it. The output is an attribution matrix, not a coverage number.

1. Detection and attribution are separate measurements

Most patch gates answer one question: does the suite go red when the code is broken? That is detection. Attribution asks the follow-up: red because of which check?

The distinction matters and it is not academic. A fault caught by eleven tests is well covered. A fault caught by exactly one test is one refactor away from escaped. A fault caught by nobody is a gap you have been shipping.

So the artifact has three outputs per fault: escaped, caught once, caught repeatedly. Only the first two demand action.

2. Build the fault catalog from the patch's own diff

Do not invent exotic bugs. Write five to fifteen faults that a reviewer of this patch would plausibly worry about. Store them as git patches so they reapply cleanly and review like code.

faults/
  001-range-off-by-one.patch
  002-swap-arg-order.patch
  003-drop-none-guard.patch
  004-wrong-default-status.patch
  005-cents-to-dollars-rounding.patch

A fault patch should be tiny and reversible:

--- a/src/pricing.py
+++ b/src/pricing.py
@@ -12,3 +12,3 @@ def bucket_total(rows, size):
-    for i in range(0, len(rows), size):
+    for i in range(0, len(rows) - size, size):

Step 1: read the agent's diff and list the invariants it claims to preserve. Step 2: for each invariant, write one fault that breaks it. Step 3: stop at fifteen. A catalog you can read in two minutes gets maintained; a generated mutation set of thousands does not.

3. Run the matrix

The runner below applies one fault at a time, runs the full suite once to measure detection, then runs every collected test in isolation to measure attribution. Reverting goes through git, so a crashed run cannot leave a fault in your tree.

#!/usr/bin/env python3
"""attribution.py - which check catches which fault?"""
from __future__ import annotations

import json
import pathlib
import subprocess
import sys

ROOT = pathlib.Path(__file__).resolve().parent
FAULTS = sorted((ROOT / "faults").glob("*.patch"))
PYTEST = [sys.executable, "-m", "pytest", "-q", "-p", "no:randomly"]


def sh(cmd: list[str]) -> subprocess.CompletedProcess:
    return subprocess.run(cmd, cwd=ROOT, capture_output=True, text=True, timeout=900)


def git(*args: str) -> None:
    subprocess.run(["git", *args], cwd=ROOT, check=True, capture_output=True, text=True)


def collect_tests() -> list[str]:
    out = sh(PYTEST + ["--collect-only", "-q"]).stdout.splitlines()
    return [line.strip() for line in out if "::" in line]


def green(test: str | None) -> bool:
    return sh(PYTEST + ([test] if test else [])).returncode == 0


def main() -> int:
    tests = collect_tests()
    matrix: dict[str, dict[str, bool]] = {}

    for fault in FAULTS:
        git("apply", str(fault))
        try:
            row = {"__full_suite__": green(None)}
            for t in tests:
                row[t] = green(t)
            matrix[fault.stem] = row
            print(f"ran {fault.stem}", file=sys.stderr)
        finally:
            git("checkout", "--", ".")
            git("clean", "-fd", "src", "tests")

    pathlib.Path("attribution.json").write_text(json.dumps(matrix, indent=2))
    return report(matrix)


def report(matrix: dict[str, dict[str, bool]]) -> int:
    tests = [k for k in next(iter(matrix.values())) if k != "__full_suite__"]
    escapes = 0
    print(f"\n{'fault':34} {'suite':6} attribution")
    for fault, row in matrix.items():
        red = [t for t in tests if not row[t]]
        if row["__full_suite__"]:
            escapes += 1
            label = "ESCAPED"
        elif len(red) == 1:
            label = f"single: {red[0]}"
        else:
            label = f"shared by {len(red)} tests"
        print(f"{fault:34} {'red' if not row['__full_suite__'] else 'green':6} {label}")
    return 1 if escapes else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it with a hard budget in mind. Twelve faults against sixty tests is 732 pytest invocations, so scope the test collection to files that touch the patched modules before you widen it.

python -m pytest --collect-only -q | grep -E 'test_(pricing|orders)' > /tmp/scope.txt
python attribution.py 2>/tmp/attribution.log; echo "exit=$?"

The exit code is deliberate: any escaped fault fails the job, so the matrix can sit in CI without a human reading it on happy paths.

4. Read three numbers, not the whole table

Escape rate. Faults where the full suite stayed green. Target is zero, or an explicit written acceptance per fault. Nothing else in the report matters until this is handled.

Sole detectors. Faults with exactly one red test in isolation. These are your brittle permissions to merge. Write them down; they are the tests you must not skip, rename, or freeze casually.

Duplicate coverage. Faults red in many tests. Harmless, sometimes healthy, and occasionally a sign that twelve assertions test one thing while nothing tests the other eleven.

Attribution is measured per test in isolation, while detection is measured on the full suite. Those two runs do not share fixtures or ordering, so a test red alone may be green in the full run and vice versa. Treat a mismatch as a signal to investigate, not as a bug in the harness.

5. Add a fixture canary before you trust any of it

An empty fixture makes every downstream assertion vacuous, and a vacuous assertion looks identical to a passing one. Guard the input, not just the output.

# tests/conftest.py
import pytest

REQUIRED = {"id", "total_cents", "status"}


@pytest.fixture
def orders(load_fixture):
    rows = load_fixture("orders.json")
    assert rows, "orders.json is empty; downstream assertions would be vacuous"
    missing = REQUIRED - set(rows[0])
    assert not missing, f"fixture drifted, missing keys: {sorted(missing)}"
    return rows

Fixture canaries are checks too. If a fault trips only the canary, the attribution table says so, and you learn that your real assertions were never exercised.

6. A freeze is an attribution loss, so charge for it

Flaky freezes are usually granted on symptoms: a test failed twice in CI. The matrix lets you price the freeze. If the test you are about to quarantine is the sole detector of a fault, the freeze silently converts that fault into an escaped one.

The rule I use: a freeze requires a named owner, an expiry date, and a receipt showing another non-flaky check still catches the fault the frozen test used to catch. No receipt, no freeze. If no substitute exists, the flaky test stays in the suite and you fix it, because a flaky sole detector is worse than an honest red.

Matrix result	Freeze allowed?	Merge?
Escaped fault, no check catches it	n/a	Block until a check exists or the gap is accepted in writing
Fault caught only by the flaky test	No	Block: replace or repair the detector first
Fault caught by flaky test plus two stable ones	Yes, with expiry and owner	Merge
Fault caught by many stable tests	Yes	Merge; note the redundancy for later pruning
No faults written for a changed module	n/a	Hold: the catalog is incomplete, not the suite clean

That last row keeps me honest. A clean matrix over an empty catalog means I did not think hard enough about the patch.

7. Where free model access and a free server actually help

Two parts of this loop are tedious rather than hard: drafting plausible faults from a diff, and running hundreds of sequential suite invocations. MonkeyCode's free model access is useful for the first — asking for candidate invariants and fault patches from the agent's own diff gives me a starting catalog in minutes. I never accept a drafted fault or assertion without applying it and reading the resulting matrix row; the model proposes, the runner decides.

The free server option covers the second: the matrix is embarrassingly parallel across faults, so it can run off my laptop while I keep editing. Both are operator-supplied availability claims, not benchmarks — measure your own runtime before wiring anything into CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

8. Limitations and who should skip this

Attribution only covers the faults you imagined. A clean matrix means your catalog is satisfied, never that the patch is correct. The catalog is the ceiling.

Isolation runs diverge from full-suite runs whenever tests share fixtures, ordering, or module-level state. Parametrized and dynamically generated assertions do not map cleanly onto test node IDs either. Expect to hand-attribute the awkward cases.

Skip this if your suite takes longer than a few minutes per invocation and cannot be sharded, if the repo has no reliable revert path, or if your patch touches a module with no existing fixtures to canary. And do not turn the matrix into a coverage percentage — it is a per-fault argument, and averaging it destroys the only information it carries.

Start with five faults against the files the agent actually touched. If the first run surfaces one escaped fault, the harness has already paid for itself — and if you want to try it before provisioning anything, the free tier is enough to run five faults and read the table.