DEV Community

Dakota Liu
Dakota Liu

Posted on

AI Wrote the Fix. Who Tests the Fixer? A 50-Line Regression Gate

#ai #testing #opensource #tutorial

There's a thread making the rounds on DEV this week: AI promoted every developer to reviewer, and nobody tested the reviewer. I felt that one. The assistant next to me "fixes" things all day. Some fixes are great. Some are confidently wrong. Reading the diff doesn't tell me which is which — so I stopped reading and started measuring.

Here's a from-zero-to-running harness that scores every AI fix in about 50 lines of bash. It re-runs the test suite after each AI patch, records pass/fail, and gives you a number you can use before you grant merge rights.

The setup that makes this affordable: MonkeyCode's free model access and its free server option, so the loop costs nothing and runs somewhere that isn't my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of this writing, the project advertises a 10M-token free allowance plus a free server tier. Free tiers move, so re-check the README before you plan a big run.

The setup: a repo that is broken on purpose

You don't need a real codebase to test this. You need one failing test and one function that should be easy to fix.

# calculator.py
def add(a, b):
    return a + b

def divide(a, b):
    if b == 0:
        raise ZeroDivisionError("boom")
    return a / b
Enter fullscreen mode Exit fullscreen mode 
# test_calculator.py
from calculator import add, divide

def test_add():
    assert add(2, 3) == 5

def test_divide_by_zero_returns_none():
    # Design decision: we want None back, not an exception.
    assert divide(5, 0) is None
Enter fullscreen mode Exit fullscreen mode

Verify the baseline:

git init && git add . && git commit -m "broken baseline"
pytest test_calculator.py
# 1 failed, 1 passed
Enter fullscreen mode Exit fullscreen mode

Any assistant that can't fix this one function should not be touching your production code. The gate will tell you quickly.

The gate: one trial, one verdict

The script below does four things: snapshots the broken state, asks the assistant for a patch, applies it, and re-runs the suite. The only piece you must swap is the adapter in the middle.

#!/usr/bin/env bash
# regression_gate.sh — one AI-fix trial, one verdict.
set -euo pipefail

TEST_TARGET="${1:-test_calculator.py}"
RUN_ID="$(date +%Y%m%d-%H%M%S)"

# 1. Baseline.
pytest -q "$TEST_TARGET" || true

# 2. ADAPTER: replace with your assistant's real fix command.
#    The exact command name and flags depend on the CLI version
#    you installed — the README is the source of truth.
FIX_PATCH="$(monkeycode-fix \
  --prompt "Fix the failing test in $TEST_TARGET. Output only a unified diff, nothing else.")"
echo "$FIX_PATCH" > ".fix_${RUN_ID}.patch"

# 3. Apply and re-run.
if git apply ".fix_${RUN_ID}.patch" 2>/dev/null; then
  if pytest -q "$TEST_TARGET" > ".out_${RUN_ID}.log" 2>&1; then
    RESULT="pass"
  else
    RESULT="fail"
  fi
else
  RESULT="apply_error"
fi

# 4. Record.
echo "$RUN_ID,$RESULT,$(wc -c < ".fix_${RUN_ID}.patch")" >> runs.csv
echo "$RESULT"
Enter fullscreen mode Exit fullscreen mode

Verify the first trial:

bash regression_gate.sh
cat runs.csv
# 20260829-091500,pass,241
Enter fullscreen mode Exit fullscreen mode

One pass proves nothing. Free-tier models are nondeterministic. That's why you loop.

Loop it, and let a server do the waiting

Ten trials give you a pass rate you can trust more than a single lucky run. Run them sequentially, resetting the repo each time:

for i in $(seq 1 10); do
  git restore . 2>/dev/null || true
  bash regression_gate.sh
  sleep 5   # be gentle with rate limits
done
Enter fullscreen mode Exit fullscreen mode

You don't want this loop on your laptop. This is exactly where the free server option earns its keep: provision it, grab the connection string it gives you, clone the repo there, and let it run while you sleep.

ssh user@your-free-server
# clone your repo, then:
nohup bash -c 'for i in $(seq 1 10); do git restore .; bash regression_gate.sh; sleep 5; done' > loop.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Verify the loop finished:

wc -l runs.csv
# 11  (header + 10 trials)
Enter fullscreen mode Exit fullscreen mode

Read the CSV like a human 

awk -F, 'NR>1 {t++; if ($2=="pass") p++} END {printf "%d/%d passed (%.0f%%)\n", p, t, 100*p/t}' runs.csv
Enter fullscreen mode Exit fullscreen mode

Then apply this decision table:

Pass rate over 10 trials Verdict
80%+ Safe to auto-apply, but you still review every diff
50–79% Branch-only. Review each change carefully
< 50% Suggestions only. No auto-apply
Many apply_error The prompt is off, or the model emits dirty diffs. Fix the prompt first

What this buys you

A number. Before this harness, trust was a feeling. After it, trust is a percentage you can paste into a PR description. It also catches the worst failure mode of free-tier assistants: a confident fix that doesn't fix anything. And it flips the "what do you do while the AI codes?" question around — you're not hovering over the output, you're running the gate that decides whether that output ships.

Where it breaks

  • Token budgets. Ten trials on a tiny repo are cheap. A real failing suite can burn a lot of tokens per trial, so watch the 10M allowance.
  • Nondeterminism is real. A pass rate is a distribution, not a guarantee.
  • Patch formats. Models drift in and out of producing clean diffs; when they do, you get apply_error instead of a failed test.
  • The gate only checks what your tests check. A green suite is not a correctness proof.
  • Free tiers change. The 10M-token allowance and the free server are the project's current offer, not a forever contract.

Who should skip this

Teams with strict compliance or on-prem-only requirements — a hosted free server may not be acceptable, period. Teams shipping security or regulatory patches shouldn't let a pass-rate gate grant auto-merge; those need deterministic review. And if your project is one file with two tests, this harness is more code than the app. Skip it until the suite is worth protecting.

The 30-second version

Your AI assistant is a junior developer with perfect confidence. Juniors get their work verified by tests before they merge. Same rule, same gate. Fifty lines of bash, ten trials, one CSV — and next time the assistant says "fixed it," you can say "prove it."

If you want to run this loop against MonkeyCode's free tier, the project README has the install command and the current token numbers. Bring your own failing test — that part is free.

Top comments (0)