Run the Check That Can Fail First: An Evidence Ladder for Agent Patches

An agent patch rarely needs more tests. It needs its checks in the right order, a stop rule at every rung, and a hard freeze on any check whose failure nobody can explain yet.

Most teams verify generated patches by running everything and reading the final color. That treats a cheap invariant check and a twelve-minute integration suite as if they carried the same information. They do not, and the ordering mistake is what makes review slow and merges fragile.

This article lays out a five-rung evidence ladder, the harness that runs it, and the rules that keep the bottom rung from becoming a permanent excuse.

The failure mode: uniform evidence

A full suite answers one question: "did anything break?" It does not answer "did this patch do what it claims?" Those are different questions, and only the second one is about the patch in front of you.

When the suite is the only gate, two costs appear. First, most of the runtime goes to code the diff never touched, so feedback arrives late. Second, any flaky node inside the suite gets a silent vote on every merge — which is how a merge decision ends up being made by a test that fails 3% of the time for reasons unrelated to the change.

Ordering fixes both. Cheap, high-discrimination checks run first and can stop the ladder early.

The ladder

Each rung has an entry condition and a stop rule. A patch is not "green"; it is "green through rung N," and that number belongs in the PR description.

1. Rung 0 — quarantine gate. Any node listed in .ci/quarantine.txt is removed from evidence collection for this run. It can still run in the nightly lane. It cannot be the reason a patch merges.
2. Rung 1 — red-before. The tests the patch adds must fail against the parent commit. A new test that passes on both parent and child constrains nothing about this change.
3. Rung 2 — properties with teeth. For each touched invariant, run a property check that has demonstrably rejected at least one deliberately broken implementation. Properties that survive every mutant are discarded.
4. Rung 3 — fixture pin. Run checks whose fixtures were authored outside the patch. This rung is deliberate: the agent does not get to define its own ground truth.
5. Rung 4 — full suite. Only entered if rungs 1–3 all passed and the diff crosses a module boundary. Here the suite is a regression net, not the primary evidence.

A harness that enforces the order

The ladder is only real if a script refuses to run rung 4 after a failed rung 1. This is the harness I use, trimmed to the parts that matter.

#!/usr/bin/env python3
"""patch_evidence.py - ordered evidence rungs for an agent-authored patch.

    python patch_evidence.py --base origin/main --head HEAD \
        --quarantine .ci/quarantine.txt --ledger .evidence/ledger.jsonl
"""
from __future__ import annotations
import argparse, hashlib, json, pathlib, subprocess, time


def run(cmd, timeout, cwd=None):
    t0 = time.monotonic()
    try:
        p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout)
        return ("pass" if p.returncode == 0 else "fail",
                round(time.monotonic() - t0, 2),
                (p.stdout + p.stderr).strip().splitlines()[-1:])
    except subprocess.TimeoutExpired:
        return "timeout", round(time.monotonic() - t0, 2), [f"exceeded {timeout}s"]


def quarantine(path):
    out = {}
    for raw in pathlib.Path(path).read_text().splitlines():
        line = raw.split("#")[0].strip()
        if not line:
            continue
        node, *meta = line.split()
        out[node] = dict(m.split("=", 1) for m in meta if "=" in m)
    return out


def ladder(base, head, qfile, ledger):
    diff = subprocess.run(["git", "diff", "-U0", f"{base}...{head}"],
                          capture_output=True, text=True).stdout
    patch = hashlib.sha256(diff.encode()).hexdigest()[:16]
    frozen = quarantine(qfile)
    rungs = []

    def record(rung, check, status, secs, witness, **extra):
        row = dict(patch=patch, rung=rung, check=check, status=status,
                   seconds=secs, witness=witness[-1] if witness else None, **extra)
        rungs.append(row)
        pathlib.Path(ledger).parent.mkdir(parents=True, exist_ok=True)
        with open(ledger, "a") as fh:
            fh.write(json.dumps(row) + "\n")

    # Rung 1 - red-before: the patch's own tests must fail on the parent commit.
    tree = "/tmp/parent-tree"
    subprocess.run(["git", "worktree", "add", "--detach", tree, base],
                   capture_output=True, text=True)
    new_tests = [l[6:] for l in diff.splitlines() if l.startswith("+++ b/tests/")]
    if new_tests:
        subprocess.run(["git", "checkout", head, "--", *new_tests], cwd=tree)
        status, secs, tail = run(["pytest", "-q", "-x", *new_tests], 300, cwd=tree)
        # pytest must FAIL here; a pass means the new test has no discriminating power.
        record("red_before", ",".join(new_tests),
               "pass" if status == "fail" else "no_teeth", secs, tail)

    # Rung 2 - seeded so any property failure is replayable verbatim.
    seed = int(hashlib.sha256(diff.encode()).hexdigest()[:8], 16)
    status, secs, tail = run(["pytest", "-q", "props/",
                              "--hypothesis-seed", str(seed)], 600)
    record("property", "props/", status, secs, tail, seed=seed)

    # Rung 3 - fixtures authored outside the patch.
    status, secs, tail = run(["pytest", "-q", "tests/fixtures/"], 600)
    record("fixture_pin", "tests/fixtures/", status, secs, tail)

    subprocess.run(["git", "worktree", "remove", "--force", tree],
                   capture_output=True, text=True)

    # Rung 4 - only if nothing above already failed.
    if all(r["status"] == "pass" for r in rungs):
        status, secs, tail = run(["pytest", "-q"], 900)
        record("suite", "pytest -q", status, secs, tail)

    print(json.dumps(rungs, indent=2))
    return 0 if all(r["status"] == "pass" for r in rungs) else 1


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--head", default="HEAD")
    ap.add_argument("--quarantine", default=".ci/quarantine.txt")
    ap.add_argument("--ledger", default=".evidence/ledger.jsonl")
    a = ap.parse_args()
    raise SystemExit(ladder(a.base, a.head, a.quarantine, a.ledger))

Two details are load-bearing. Rung 1 records no_teeth, not fail, because a test that passes on the parent is a finding about the test, not about the patch. And every rung writes a witness — the last line of output — so the ledger says what failed, not merely that something did.

Properties that survive a mutant are not evidence

Property checks are the cheapest way to constrain a patch's behavior across inputs. They are also trivially easy to write in a form that can never fail.

The fix is a teeth check. Every property must reject at least one deliberately broken implementation of its target. If a property passes all mutants, it is deleted from the registry.

# mutation_registry.py
MUTANTS = {
    "normalize_events": {
        "no_sort":     lambda evs: list(evs),
        "dedupe_by_id": lambda evs: list({e.id: e for e in evs}.values()),
        "drop_tail":   lambda evs: list(evs)[:-1],
    },
}

# tools/property_teeth.py - run in CI before the property rung
from props.registry import PROPERTIES
from mutation_registry import MUTANTS


def kills(check, target, mutant):
    try:
        check(mutant, target["sample"])
        return False          # survived: this property carries no information
    except AssertionError:
        return True


survivors = [p["name"] for p in PROPERTIES
             if not any(kills(p["check"], p["target"], m)
                        for m in MUTANTS[p["target"]])]

if survivors:
    raise SystemExit("properties with no teeth: " + ", ".join(survivors))

Run it as python tools/property_teeth.py. A non-zero exit blocks rung 2, which means a model-drafted invariant never reaches the ledger until it has demonstrably caught something.

The freeze: quarantine with an expiry, not a memory hole

A flake freeze is only honest if it ends. The format below forces a decision at a stated date.

# node                                              expires=    owner=     reason=
tests/test_cache.py::test_ttl_survives_clock_skew   2026-11-30  @finley    shared-runner clock skew, issue 412
tests/test_api.py::test_pagination_order            2026-11-07  @dana      upstream ordering change, issue 418

Four rules make it work.

1. A frozen node is deselected per run with --deselect <node>; it never contributes a vote to a merge.
2. An entry that reaches its expiry must be fixed, deleted, or re-justified by a named owner. Auto-renewal is not an option the file supports.
3. A patch that passes only with quarantine applied is reported as "green minus quarantine," and the excluded nodes are listed in the PR.
4. New tests added by the patch start outside quarantine. The patch author cannot widen its own leniency.

Which rungs a patch actually needs

Not every diff earns the full ladder. The table below is the triage rule I apply before the harness runs.

Patch class	Rung 1 red-before	Rung 2 properties	Rung 3 fixture pin	Rung 4 suite
Pure value mapping	required	required (2+)	skip	only if serialization changed
Retry / state machine	required	required (idempotence, monotonicity)	required	required
Schema or migration	required	required (round-trip)	required	required
Rename, comment, log text	n/a	skip	skip	not evidence
Dependency bump	n/a	skip	required (lockfile)	required

Two rows are the useful ones. A rename patch has no behavioral claim to test, so demanding a new test for it produces theater. A retry-loop patch has the opposite problem: it needs properties, because the interesting failures are orderings and repeated delivery, which example-based tests rarely reach.

Where a free model endpoint and a free server fit

The rung that benefits from a model is rung 2's drafting step: proposing candidate invariants and a mutant list for the symbols the diff touched. Drafting is cheap text work, so an operator-supplied free model endpoint is enough for it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same is true of execution. Rungs 1 through 3 are short, isolated pytest invocations that do not need a large runner, which is what makes the operator-supplied free server option a reasonable place to put them. Keep the ledger in your own repository regardless of where the rungs execute.

What the free side does not provide is authority. A drafted property is a hypothesis until a mutant kills it, and a host without an uptime commitment cannot be the only place a green result exists.

Limitations and who should not use this

• Drafting is not verification. Some model-drafted properties will survive all mutants and must be discarded. Expect the registry to be smaller than the draft set, and treat that shrink as the quality signal.
• Seeds must be recorded. A property failure found under --hypothesis-seed is unreproducible unless that integer is in the ledger. The harness in this article derives it from the diff hash for exactly that reason.
• Some code has no testable seam. If the touched logic is inseparable from a live network or a clock, you cannot run rung 2 meaningfully. Fix the seam first; do not paper over it with a mocked property.
• Skip this approach if nobody owns the quarantine file. Without enforced expiry, the ladder degrades into "run fewer tests," which is strictly worse than the uniform suite you have today.
• Skip it in environments that require a fixed CI of record for audit. A best-effort host is fine for rungs 1–3, but the artifact of record has to live somewhere durable and contracted.

The ladder does not reduce the size of your test suite. It reduces the size of the claim you make about any single patch, which is the harder and more useful discipline. Start by writing the quarantine file — before you write a single property, give every current flake an expiry date and a name.