Most conversations about free coding models ask the wrong question. People ask "can it fix my bug?" when the more expensive question is: when it fixes the bug, what else does it break?
A patch that passes the one test you pointed it at but silently breaks three others is worse than no patch at all — it converts a known failure into an unknown one. And in my experience reviewing AI-generated diffs, regression risk is exactly where cheaper and free-tier models diverge most from the frontier ones. Not in whether they can produce a plausible fix, but in whether the fix respects the rest of the codebase.
So instead of trusting vibes, I built a small harness that answers one measurable question per model: what percentage of its accepted-looking patches introduce regressions in the existing test suite? This post is that harness, the decision table I use with the results, and an honest look at where the whole approach falls apart.
The setup
The ingredients are deliberately boring:
- A repo with a real test suite. The more of the suite is unrelated to the bug you're fixing, the better — those are your tripwires.
- A handful of candidate patches for the same bug, generated by whatever model you're evaluating. I generate several candidates rather than one, because a single attempt tells you almost nothing.
- An isolated environment to run everything, because you're about to execute model-generated code against a test suite many times.
On that last point: I run this harness on a disposable cloud server rather than my laptop, partly so model-generated code never touches my working machine, and partly so I can leave multi-hour runs going without thinking about it. For this round I used MonkeyCode, which currently offers free model access (useful for generating candidate patches without rationing prompts) and a free server option (useful as the sacrificial sandbox the harness runs in). Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing in the harness depends on that provider, though — any box you can SSH into and any model endpoint will do the same job.
The harness
The core trick is git worktrees. Each candidate patch gets applied to its own worktree, the full test suite runs there, and the result is compared against the suite's behavior on the base commit. A regression is any test that passed on base and fails after the patch.
#!/usr/bin/env bash
# regress_check.sh <base_commit> <patch_file> <test_command>
# Example: ./regress_check.sh HEAD~1 candidate_03.patch "pytest -x -q"
set -u
BASE="$1"
PATCH="$2"
TEST_CMD="$3"
WT=$(mktemp -d)/wt
git worktree add --detach "$WT" "$BASE" >/dev/null 2>&1
# Baseline: which tests pass before the patch?
( cd "$WT" && eval "$TEST_CMD" --tb=no -q 2>/dev/null | tail -n 1 ) > "$WT.baseline.txt"
if git -C "$WT" apply "$PATCH" 2>/dev/null; then
( cd "$WT" && eval "$TEST_CMD" --tb=no -q 2>/dev/null | tail -n 1 ) > "$WT.patched.txt"
echo "=== $PATCH ==="
echo "baseline: $(cat "$WT.baseline.txt")"
echo "patched : $(cat "$WT.patched.txt")"
else
echo "=== $PATCH === APPLY FAILED"
fi
git worktree remove --force "$WT" >/dev/null 2>&1
For per-test granularity (which you want — summary lines alone hide which tests flipped), swap the summary capture for a machine-readable report:
# pytest example: emit JSON-ish results per test
pytest -q --tb=no -rA | grep -E '^(PASSED|FAILED)' | sort > "$WT.after.txt"
# Then diff against the baseline file:
# tests in baseline-pass but not in after-pass = regressions
comm -23 "$WT.before.txt" "$WT.after.txt" > "$WT.regressions.txt"
Run it across N candidate patches for the same bug and you get the metric that actually matters:
regression_rate = patches_with_new_failures / patches_that_apply_and_fix
Note the denominator. Patches that don't apply and patches that don't fix the bug are filtered out first — you only care about the danger hiding inside patches that look successful.
What I look for in the output
After a few runs of this against real bugs, the results tend to cluster into patterns that a single trial would never show:
- Apply-failure rate is a rough proxy for how well the model tracked the actual repo state. Models that hallucinate surrounding context produce patches that don't apply at all. That's annoying but safe — git rejects them.
- Fix-but-regress is the dangerous bucket. The model fixes the reported bug by changing a shared helper, and two tests in an unrelated module go red. These are the patches that survive a lazy review.
- Consistency across candidates matters more than any single result. A model that produces one clean patch and four regressing ones is telling you its clean output is partly luck.
The decision table
This is how I translate harness output into policy:
| Observed regression rate | My rule |
|---|---|
| 0% across 5+ candidates, small diffs | Model output can go straight to a PR, still with human review |
| Under ~20%, regressions in adjacent files only | Use it, but always run the full suite before merging; never trust the targeted test alone |
| 20–50% | Use it for exploration and diagnosis only, not for patches I intend to keep |
| Over 50%, or regressions in unrelated subsystems | Don't let this model touch shared code paths; restrict it to greenfield or throwaway scripts |
The thresholds are mine, not laws. The point is that after one afternoon of harness runs, you stop arguing about whether a model is "good" and start arguing about numbers you produced yourself.
Limitations, because there are several
- Your test suite is the ceiling. If your suite has 40% coverage, a 0% regression rate means the model got lucky in the dark. The harness measures detected regressions, not regressions.
- Flaky tests poison everything. Quarantine known flakes before running this, or they'll show up as fake regressions in every column.
- Candidate generation isn't controlled. Asking the model for five patches isn't five independent samples; they share the model's biases. This measures practical reliability, not statistical truth.
- It doesn't test judgment. A model can pass this harness and still make terrible architectural choices that no test suite will catch.
Who should skip this entirely
If your repo has no meaningful test suite, build the tests first — this harness has nothing to measure without them. If your changes are mostly UI polish, copy, or config, regression rate is a low-signal metric and a quick manual review is cheaper. And if you're evaluating a model for a one-off throwaway script, none of this matters; the harness is for deciding what gets near code you'll maintain.
Closing thought
The free-model landscape right now makes it tempting to evaluate by asking "did it fix my bug once?" — the demo question. The harness above takes an afternoon to set up and permanently changes the question to "how often does it fix the bug cleanly?" That second number is the one that determines whether a model saves you time or just moves your debugging somewhere harder to see.
If you want to try this, any model plus any spare server works — I've been running mine through MonkeyCode's free model access on its free server tier since that's what was in front of me, but the script doesn't care. Run it against your own repo's history of real bugs, and I'd genuinely like to hear what regression rates you get; my sample sizes are small and every codebase tells a different story.
Top comments (0)