Opinion: Let the Test Suite Pick the AI Patch, Not the Reviewer

The most expensive part of an AI-assisted fix is no longer the model call; it is the human minutes spent deciding if the patch is correct. Free model access has inverted the economics of generation, so the review gate has to move from eyeballs to invariants. My position is blunt: treat the model as a candidate generator, let a focused test suite dispose of the candidates, and review only the minimal surviving diff. Everything else is reading diffs for entertainment.

The bottleneck moved from generation to judgment

Two years ago, a single AI patch was a scarce artifact that deserved a careful line-by-line read. Today, free model access can produce a dozen plausible variants in the time it takes to finish a coffee. Human attention does not scale with candidate count; it degrades after the third diff, and the degradation is silent. The scarce resource is no longer generation, so the gate has to be a deterministic check that never gets tired.

Why reading the diff is the wrong unit of analysis

An AI diff can look coherent and still violate an invariant that lives outside the visible hunk. The failure usually sits in the interaction between the change and a state assumption written months earlier, not in the lines the model touched. That is why the diff is the wrong unit of analysis and the invariant is the right one. Your test suite already encodes the invariants the team cares about, so let it act as the arbiter.

The workflow

1. Write the invariant test before you generate anything. The test should capture the failing behavior in the smallest possible surface, for example "the retry budget is never negative" or "a drained queue returns 503, not 500".
2. Generate K candidates from the same problem statement, keeping the prompt identical so the results are a sample rather than a conversation.
3. Apply each candidate to a clean worktree and run the focused test in isolation.
4. Rank the passing candidates by diff size and discard every failure, no matter how plausible the patch looks.
5. Review the minimal survivor for security and style only, because correctness has already been decided by the test.

The ordering matters. A diff autopsy is still valuable, but it belongs after the test filter, not before it, because the filter is what makes the autopsy cheap.

The harness

Here is the reproducible core of the workflow. The generation call is left as a stub because the client API differs by environment; the selection logic is the part worth copying.

# candidate_harness.py
# Generate K candidate patches, apply each to a clean worktree,
# and rank them by a focused invariant test instead of by reading diffs.

import json
import subprocess
import tempfile

REPO_URL = "https://github.com/your-org/target-repo.git"
FOCUSED_TEST = "pytest tests/test_invariants.py -x -q"
CANDIDATE_COUNT = 12


def generate_candidates(problem: str, count: int) -> list[dict]:
    """Return `count` distinct patches for the same problem statement.

    Keep the prompt identical across calls so the candidates form a
    sample, not a sequence of incremental edits.
    """
    return []  # model client call goes here


def apply_and_test(candidate: dict) -> dict:
    with tempfile.TemporaryDirectory() as worktree:
        subprocess.run(
            ["git", "clone", "--quiet", "--depth", "1", REPO_URL, worktree],
            check=True,
        )
        apply = subprocess.run(
            ["git", "apply"],
            input=candidate["patch"],
            cwd=worktree,
            capture_output=True,
            text=True,
        )
        if apply.returncode != 0:
            return {
                "id": candidate["id"],
                "passed": False,
                "diff_lines": len(candidate["patch"].splitlines()),
                "reason": "patch does not apply",
            }
        test = subprocess.run(
            FOCUSED_TEST,
            shell=True,
            cwd=worktree,
            capture_output=True,
            text=True,
        )
        return {
            "id": candidate["id"],
            "passed": test.returncode == 0,
            "diff_lines": len(candidate["patch"].splitlines()),
            "output_tail": (test.stdout + test.stderr)[-400:],
        }


def main(problem: str) -> None:
    candidates = generate_candidates(problem, CANDIDATE_COUNT)
    results = [apply_and_test(c) for c in candidates]
    passing = sorted(
        (r for r in results if r["passed"]),
        key=lambda r: r["diff_lines"],
    )
    report = {
        "passing": passing,
        "failed": [r["id"] for r in results if not r["passed"]],
    }
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main("service returns 500 when the retry budget is exhausted")

A typical run produces a table like this:

Candidate	Invariant test	Diff size	Verdict
A	pass	41	review and merge
B	pass	87	keep as fallback
C	fail	12	reject, even though it is tiny
D	fail	210	reject

The counter-intuitive result is candidate C: it is the smallest diff and the most readable one, yet it fails the invariant. Reading diffs by eye would have ranked C first; the harness ranks it last.

What the free tier changes

MonkeyCode's free model access makes step 2 cheap enough to run with K=12 instead of K=1, which changes the exercise from picking a patch to sampling a distribution. Its free server option gives the harness a disposable execution target, so the experiment never pollutes your local checkout or your staging state. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When generation is free, verification is the only scarce resource left, and that is exactly the resource this harness spends.

The free server is not the thing being patched here; it is the disposable bench where the harness runs. That distinction matters because the harness is allowed to fail loudly, to leave broken worktrees behind, and to burn through twelve clones without guilt.

Limitations

This approach only works when an invariant test exists, or when you are disciplined enough to write one first; without it, the harness is just a fancy linter. A passing run is evidence, not proof, because property tests can be flaky and models can overfit to a weak test. The method does not replace security review, and it should not be used for auth, payment, or data-loss paths where the blast radius is too large for a harness alone. Teams with zero test coverage should fix that gap before adopting this workflow, because the gate is only as honest as the invariant behind it.

Who should not use this

Skip this workflow for one-off scripts, throwaway migrations, or changes where the invariant is trivially satisfied by an empty patch. Skip it when the failure is a race condition that your test suite cannot reproduce deterministically, because the harness will rank noise. Skip it when the team treats the test suite as a formality and merges on green without understanding the change. In those environments, the old line-by-line review is still the lesser evil.

Conclusion

The reviewer's job changes from judging diffs to writing invariants, which is a harder skill and a more honest one. Free model access did not make review obsolete; it made the old review style obsolete. Write the invariant, build the harness, and let the test suite earn its keep. The model proposes, the test disposes, and you review the survivor.