A Repeatable Harness for Catching Regressions in AI-Generated Code

Morgan Zhou — Wed, 05 Aug 2026 10:11:51 +0000

AI coding assistants are great at producing plausible diffs. They are less great at telling you whether the diff quietly broke an edge case three files away. After a few rounds of "looks fine, ship it, roll it back," I stopped trusting eyeball reviews of generated patches and built a small, repeatable harness that any AI-generated change has to survive before I merge it.

This post walks through that harness: a golden-input test set, a diff-aware check script, and a decision table for when free hosted model access is enough versus when you need your own infrastructure. It works with any model provider, but I'll note where free tiers (including MonkeyCode's free model access and free server option) fit naturally, since cost is usually what stops people from running this loop on every change.

The problem: generated code fails in boring ways

The failures I actually see from AI-assisted changes are not dramatic. They're things like:

A refactored parser that now trims whitespace it used to preserve.
A "simplified" retry loop that dropped the jitter, so retries thunder in sync.
An off-by-one in pagination that only triggers when the result count is an exact multiple of the page size.

None of these show up in a casual read of the diff. All of them show up if you run the change against a fixed set of known-tricky inputs and compare behavior before and after.

The artifact: a three-part harness

The harness has three pieces, each boring on its own:

Golden inputs: a checked-in directory of inputs that historically broke things, plus the expected behavior for each.
A runner: a script that executes the current code against every golden input and diffs the output against expectations.
A gate: CI (or a pre-merge script) that fails if the runner reports a mismatch the author didn't explicitly bless.

Here's a minimal runner in Python. It's deliberately dependency-free so it works anywhere:

#!/usr/bin/env python3
"""golden_run.py — run golden inputs, diff against expected outputs.

Layout:
  golden/
    case_001.input.txt
    case_001.expected.txt
    ...
"""
import json
import subprocess
import sys
from pathlib import Path

GOLDEN = Path("golden")
BLESSED = Path("golden/blessed_changes.json")  # diffs a human approved

def run_case(input_path: Path) -> str:
    # Swap this for whatever invokes your code path under test.
    result = subprocess.run(
        [sys.executable, "app/transform.py"],
        input=input_path.read_text(),
        capture_output=True, text=True, timeout=30,
    )
    return result.stdout

def main() -> int:
    blessed = json.loads(BLESSED.read_text()) if BLESSED.exists() else {}
    failures = []
    for inp in sorted(GOLDEN.glob("*.input.txt")):
        case = inp.name.replace(".input.txt", "")
        expected = (GOLDEN / f"{case}.expected.txt").read_text()
        actual = run_case(inp)
        if actual != expected and case not in blessed:
            failures.append(case)
    if failures:
        print("Unblessed regressions:", ", ".join(failures))
        return 1
    print("All golden cases pass (or are explicitly blessed).")
    return 0

if __name__ == "__main__":
    sys.exit(main())

The blessed_changes.json file is the important part. When the AI's change intentionally alters behavior, you don't just update the expected file — you add the case name plus a one-line reason, which forces a human to acknowledge the behavioral delta in code review:

{
  "case_014": "Whitespace trim is intentional; parser contract updated in docs/parser.md"
}

Where the AI fits in the loop

The harness changes how I use the assistant. Instead of "generate the fix and I review it," the loop becomes:

Describe the bug and ask the model for a fix plus a new golden input that reproduces it.
Run the harness. The new case should fail on the old code and pass on the new code. If it passes on both, the repro case is wrong — very common, and catching it here saves a bad merge.
If other cases fail, feed the mismatch output back to the model and iterate, or fix by hand.

Step 2 is where iteration cost matters. Verifying that a repro case actually reproduces often takes three or four round trips with the model, and that adds up on metered APIs. This is where free-tier access is genuinely useful: I route these exploratory iterations through MonkeyCode, which offers free model access and a free server option, and keep paid capacity for nothing. The verification loop doesn't need the strongest model — it needs a cheap, fast one you don't hesitate to re-run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One practical note: because the free server is shared infrastructure, I keep golden inputs free of secrets and customer data regardless of provider. That's a habit worth having anyway.

When free hosted access is enough, and when it isn't

Situation	Free hosted model + free server	Your own infra / paid tier
Repro-case verification, small patches	Yes — cost is the whole point	Overkill
Nightly full-harness runs in CI	Fine if latency is acceptable	Better if you need SLAs
Inputs contain proprietary code or PII	No — sanitize first or don't	Yes, with controls
Load/perf regression testing	No — shared infra skews timing	Yes
You need a pinned model version for reproducibility	Check what's guaranteed; assume not	Yes

The last two rows are the real limitations. Shared or free servers are the wrong place for timing-sensitive benchmarks, and "free" usually comes with no guarantee about which exact model version you hit, so don't treat outputs as reproducible across weeks. For behavior-diff testing that's fine — your golden expectations are the source of truth, not the model. For anything where the model's output is the artifact, pin a version somewhere you control.

Also: if your golden set has fewer than a dozen cases, this harness is ceremony. The payoff starts when the set grows into the dozens and manual re-checking stops being realistic.

Limitations, honestly

The harness catches behavioral diffs against cases you thought of. It says nothing about cases you didn't. Property-based tests complement it well.
Golden expected files rot when behavior changes frequently; the blessing mechanism mitigates this but depends on reviewers actually reading the reasons.
Free tiers change. Anything built on "this costs nothing today" should degrade gracefully to a paid or local fallback.

Wrap-up

The shift that made AI-generated code reliable enough for me wasn't a better model — it was making every generated change prove itself against a fixed, growing set of nasty inputs, with a human sign-off on any intentional behavior change. If you want to try the loop without committing budget, MonkeyCode's free tier is a reasonable place to run the iterative steps; the harness itself is provider-agnostic and yours to keep either way.

Stop Trusting Vibes: A Repeatable Harness for Testing LLM-Generated Code on a Free Server

Morgan Zhou — Wed, 05 Aug 2026 06:39:39 +0000

Liquid syntax error: Unknown tag 'endraw'

DEV Community: Morgan Zhou