Green Tests Lie: How I Gate AI-Generated Patches Before They Touch Main

Every team chat I'm in has landed on the same uncomfortable question: an agent can write a plausible patch in ninety seconds, but who vouches for it? Reading the diff by eye feels like due diligence. In practice, it mostly rewards fluency — and generated code is fluent by construction.

The stance I settled on: treat agent output like a pull request from a stranger with no commit history. You wouldn't merge that because it read nicely. You'd merge it because it survived your gauntlet. Below is the gauntlet I run, built entirely on compute that costs nothing, followed by an honest list of where it falls apart.

Why eyeballing diffs keeps letting bad patches through

When I audit agent patches that got a thumbs-up and later caused trouble, three defects recur:

• Prompt-shaped correctness. The code nails the scenario in the request and crumbles on the adjacent ones — empty input, DST boundaries, a record with ten thousand rows.
• Scope creep in the edit set. You asked for a fix in one module; the patch also nudges a lint config or a lockfile you never mentioned.
• Gates quietly lowered. The suite is green because a strict assertion became a loose one, or a test file got an early return.

None of these announce themselves in a casual read. So the process below makes each one impossible to skip.

The gauntlet, stage by stage

Four stages, each producing an artifact you can revisit months later:

1. Quarantine — run the patch in an environment with no network and no secrets.
2. Scope audit — diff the patch against what was actually requested.
3. Adversarial probes — tests you write before looking at the implementation.
4. Decision record — one file capturing inputs, outputs, and verdict.

Stage 1: Quarantine

The test environment should cost less to destroy than to disinfect. Mine is a scratch clone executed inside a container with networking disabled:

#!/usr/bin/env bash
# quarantine.sh — run an untrusted patch with no way in or out
set -euo pipefail

REPO=$1
PATCHFILE=$2
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

git clone -q "$REPO" "$WORKDIR/tree"

docker run --rm \
  --network none \
  --cap-drop ALL \
  --read-only \
  --tmpfs /tmp \
  -v "$WORKDIR/tree":/repo \
  -v "$PATCHFILE":/queued.patch:ro \
  -w /repo \
  local/gate-runner:py312 \
  sh -c 'git apply /queued.patch && python -m pytest -x -q'

Two properties carry all the weight: traffic cannot escape (--network none), and nothing sensitive can wander in (no credential mounts, no capabilities, read-only root). A useful mental model: you're grading the artifact, not the agent. It doesn't matter where or how the model executed — only whether the patch survives this box.

Stage 2: Scope audit

Before reading a single implementation line, get the full inventory of what the patch touches:

git apply --stat /queued.patch
git apply --check /queued.patch   # a patch that won't apply cleanly gets rejected on the spot

Then classify every path relative to the original request:

Path in patch	Asked for?	Verdict
src/parse_date.py	yes	normal review
tests/test_parse_date.py	no	manual line-by-line
pyproject.toml	no	reject pending justification

The one heuristic I'd tattoo on the process: unrequested modifications to tests, CI configs, or dependency pins start with a presumption of guilt. Softening a gate is the cheapest way to make a wrong patch look right.

Stage 3: Adversarial probes

A passing project suite only tells you the patch preserved behaviors someone previously thought to test. Everything else is on you. I write a small set of hostile probes aimed at the change's probable weak spots. Say the patch rewrote a date-string parser:

# probes/test_date_hostile.py — written by me, never shown to the model
import pytest
from src.parse_date import parse_date

def test_rejects_feb_29_on_non_leap_year():
    with pytest.raises(ValueError):
        parse_date("2025-02-29")

def test_no_silent_rollover_on_month_13():
    with pytest.raises(ValueError):
        parse_date("2025-13-01")

def test_whitespace_and_unicode_padding_not_accepted():
    with pytest.raises(ValueError):
        parse_date("\u20032025-01-01")

def test_epoch_boundary_does_not_wrap_negative():
    assert parse_date("1969-12-31").year == 1969

One discipline makes this stage honest: write the probes before you open the diff. Once you've seen the implementation, your supposedly independent tests drift toward the cases the model already covers. Attack first, read second.

Stage 4: Decision record

Every accepted patch leaves one plain-text bundle:

ask: "make parse_date strict about malformed input"
date: 2026-08-09
compute: free model access + free server via MonkeyCode
scope: 1 requested file, 0 surprise files
project suite: 312 passed
hostile probes: 4 passed (attempt 1 failed 2, regenerated, then passed)
verdict: merged; human pass over parse_date.py:8-41

This feels like paperwork until a September merge quietly regresses in November. Then the record turns a forensic afternoon into a ten-minute lookup.

Doing this on zero budget

The gauntlet consumes two things: repeated model calls (hostile probes regularly send a patch back — budget two or three regeneration rounds per change) and sacrificial compute to run them in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My current loop runs on MonkeyCode's free model access together with its free server option, which suits exactly this pattern: frequent, low-stakes regeneration where per-call pricing would otherwise push you to ration attempts. That said, nothing in the four stages depends on any specific vendor — any model endpoint plus any disposable machine (a spare laptop, a CI free tier) runs the identical gauntlet. If you're assembling this from zero, rehearsing the workflow on a free tier is a reasonable first step before committing budget anywhere.

Where the gauntlet breaks

• It proves behavior, not intent. Hostile probes won't surface a deliberately concealed flaw. Human diff review remains required — this process feeds it evidence, it never replaces it.
• A container is hygiene, not a vault. No-network, dropped-capability defaults are prudent, not a promise against a determined exploit. Never quarantine untrusted patches on a host with privileges you'd miss.
• Weak probes, false confidence. Stage 3 carries the rigor. If you phone it in, the remaining stages merely document your complacency.
• Skip the ceremony where it doesn't pay: one-token typo fixes don't justify four stages. Conversely, irreversible domains — payments, signing, authorization — need a domain expert reading every line regardless of what any suite says.
• Free tiers shift. The model and server availability described here are operator-supplied claims at the time of writing; re-verify before baking them into a team process.

The actual takeaway

The meaningful change is one of phrasing: stop asking "does this generated patch look correct?" and start asking "what can I demonstrate about it?" An offline quarantine, a scoped file audit, probes authored before the reveal, and a saved verdict — that chain is a demonstration, and today it costs essentially nothing to run.

Comments

[Theo Valmis]

The decision record stage is the part most AI code review setups skip, and it's the one that actually pays off later. Quarantine and adversarial probes catch the patch today. The decision record is what lets someone six months from now figure out why a rule exists before they change it, which is usually the moment institutional judgment gets lost.