Green Tests Lie: How I Gate AI-Generated Patches Before They Touch Main

Every team chat I'm in has landed on the same uncomfortable question: an agent can write a plausible patch in ninety seconds, but who vouches for it? Reading the diff by eye feels like due diligence. In practice, it mostly rewards fluency — and generated code is fluent by construction.

The stance I settled on: treat agent output like a pull request from a stranger with no commit history. You wouldn't merge that because it read nicely. You'd merge it because it survived your gauntlet. Below is the gauntlet I run, built entirely on compute that costs nothing, followed by an honest list of where it falls apart.

Why eyeballing diffs keeps letting bad patches through

When I audit agent patches that got a thumbs-up and later caused trouble, three defects recur:

• Prompt-shaped correctness. The code nails the scenario in the request and crumbles on the adjacent ones — empty input, DST boundaries, a record with ten thousand rows.
• Scope creep in the edit set. You asked for a fix in one module; the patch also nudges a lint config or a lockfile you never mentioned.
• Gates quietly lowered. The suite is green because a strict assertion became a loose one, or a test file got an early return.

None of these announce themselves in a casual read. So the process below makes each one impossible to skip.

The gauntlet, stage by stage

Four stages, each producing an artifact you can revisit months later:

1. Quarantine — run the patch in an environment with no network and no secrets.
2. Scope audit — diff the patch against what was actually requested.
3. Adversarial probes — tests you write before looking at the implementation.
4. Decision record — one file capturing inputs, outputs, and verdict.

Stage 1: Quarantine

The test environment should cost less to destroy than to disinfect. Mine is a scratch clone executed inside a container with networking disabled:

#!/usr/bin/env bash
# quarantine.sh — run an untrusted patch with no way in or out
set -euo pipefail

REPO=$1
PATCHFILE=$2
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

git clone -q "$REPO" "$WORKDIR/tree"

docker run --rm \
  --network none \
  --cap-drop ALL \
  --read-only \
  --tmpfs /tmp \
  -v "$WORKDIR/tree":/repo \
  -v "$PATCHFILE":/queued.patch:ro \
  -w /repo \
  local/gate-runner:py312 \
  sh -c 'git apply /queued.patch && python -m pytest -x -q'

Two properties carry all the weight: traffic cannot escape (--network none), and nothing sensitive can wander in (no credential mounts, no capabilities, read-only root). A useful mental model: you're grading the artifact, not the agent. It doesn't matter where or how the model executed — only whether the patch survives this box.

Stage 2: Scope audit

Before reading a single implementation line, get the full inventory of what the patch touches:

git apply --stat /queued.patch
git apply --check /queued.patch   # a patch that won't apply cleanly gets rejected on the spot

Then classify every path relative to the original request:

Path in patch	Asked for?	Verdict
src/parse_date.py	yes	normal review
tests/test_parse_date.py	no	manual line-by-line
pyproject.toml	no	reject pending justification

The one heuristic I'd tattoo on the process: unrequested modifications to tests, CI configs, or dependency pins start with a presumption of guilt. Softening a gate is the cheapest way to make a wrong patch look right.

Stage 3: Adversarial probes

A passing project suite only tells you the patch preserved behaviors someone previously thought to test. Everything else is on you. I write a small set of hostile probes aimed at the change's probable weak spots. Say the patch rewrote a date-string parser:

# probes/test_date_hostile.py — written by me, never shown to the model
import pytest
from src.parse_date import parse_date

def test_rejects_feb_29_on_non_leap_year():
    with pytest.raises(ValueError):
        parse_date("2025-02-29")

def test_no_silent_rollover_on_month_13():
    with pytest.raises(ValueError):
        parse_date("2025-13-01")

def test_whitespace_and_unicode_padding_not_accepted():
    with pytest.raises(ValueError):
        parse_date("\u20032025-01-01")

def test_epoch_boundary_does_not_wrap_negative():
    assert parse_date("1969-12-31").year == 1969

One discipline makes this stage honest: write the probes before you open the diff. Once you've seen the implementation, your supposedly independent tests drift toward the cases the model already covers. Attack first, read second.

Stage 4: Decision record

Every accepted patch leaves one plain-text bundle:

ask: "make parse_date strict about malformed input"
date: 2026-08-09
compute: free model access + free server via MonkeyCode
scope: 1 requested file, 0 surprise files
project suite: 312 passed
hostile probes: 4 passed (attempt 1 failed 2, regenerated, then passed)
verdict: merged; human pass over parse_date.py:8-41

This feels like paperwork until a September merge quietly regresses in November. Then the record turns a forensic afternoon into a ten-minute lookup.

Doing this on zero budget

The gauntlet consumes two things: repeated model calls (hostile probes regularly send a patch back — budget two or three regeneration rounds per change) and sacrificial compute to run them in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My current loop runs on MonkeyCode's free model access together with its free server option, which suits exactly this pattern: frequent, low-stakes regeneration where per-call pricing would otherwise push you to ration attempts. That said, nothing in the four stages depends on any specific vendor — any model endpoint plus any disposable machine (a spare laptop, a CI free tier) runs the identical gauntlet. If you're assembling this from zero, rehearsing the workflow on a free tier is a reasonable first step before committing budget anywhere.

Where the gauntlet breaks

• It proves behavior, not intent. Hostile probes won't surface a deliberately concealed flaw. Human diff review remains required — this process feeds it evidence, it never replaces it.
• A container is hygiene, not a vault. No-network, dropped-capability defaults are prudent, not a promise against a determined exploit. Never quarantine untrusted patches on a host with privileges you'd miss.
• Weak probes, false confidence. Stage 3 carries the rigor. If you phone it in, the remaining stages merely document your complacency.
• Skip the ceremony where it doesn't pay: one-token typo fixes don't justify four stages. Conversely, irreversible domains — payments, signing, authorization — need a domain expert reading every line regardless of what any suite says.
• Free tiers shift. The model and server availability described here are operator-supplied claims at the time of writing; re-verify before baking them into a team process.

The actual takeaway

The meaningful change is one of phrasing: stop asking "does this generated patch look correct?" and start asking "what can I demonstrate about it?" An offline quarantine, a scoped file audit, probes authored before the reveal, and a saved verdict — that chain is a demonstration, and today it costs essentially nothing to run.