Three Passes Before You Trust an Agent Patch

A patch arrives from an agent at 02:14. CI turns green in ninety seconds. By 02:40 refund totals are wrong in production.

The agent did not cheat. It optimised to the only signal it could see: the example tests sitting in the repo. Example tests are a memory of past bugs, not a specification.

So the question changes. Not "did the tests pass" but "which tests was the agent allowed to read". Order matters more than the individual checks.

Pass one: properties the agent never sees

A property holds for every input. "Dedupe never grows the list" survives cases nobody wrote down. An example test defends exactly one typed case and then stops.

Hold the property file out of the agent's context. Not in the prompt, not in the working tree it edits, not in the diff it reads back. An agent that can read your invariants will satisfy the invariants and nothing else. That is the same failure mode, moved one level up.

Properties are cheap to write and expensive to fake. Three invariants catch more than thirty examples, most of the time.

Pass two: fixtures with a contract

A fixture is only a control if you can prove it did not move. Hash the fixture files and pin the digest in a manifest. If the digest drifts, the run is void, not failed.

Void and fail are different verdicts. A failure accuses the patch. Drift accuses your environment. Mixing them trains reviewers to ignore both.

Pin the seed too. A property pass with an unpinned seed is a coin you cannot re-flip.

Pass three: freeze flakes by runs, not by hours

A flake is a test whose result you cannot reproduce on demand. Freezing it by calendar time is guesswork. Count consecutive stable runs instead.

Each quarantined test carries a repeat budget. It needs twenty clean runs before it regains the right to block a patch. Until then it reports, it does not gate.

The ledger is a file in the repo, reviewed like code. Flake status is a claim someone signed.

The harness

Below is the gate I would run before any agent patch touches a shared branch. Standard library only, no framework, one JSON report out. It is illustrative rather than benchmarked; I have not measured it on your repo.

# gate.py — stdlib only. Three passes, one verdict.
import hashlib, json, random, subprocess, sys
from pathlib import Path

ROOT = Path(".")
MANIFEST = ROOT / "fixtures" / "manifest.json"
LEDGER = ROOT / "flake_ledger.json"
PROPS = "held_out_props"          # never shown to the agent
SEED = 20261026
RUNS = 200


def fixture_digest(manifest):
    h = hashlib.sha256()
    for rel in sorted(manifest["files"]):
        h.update((ROOT / rel).read_bytes())
    return h.hexdigest()[:12]


def run_properties():
    sys.path.insert(0, str(ROOT))
    props = __import__(PROPS)
    rng = random.Random(SEED)
    for i in range(RUNS):
        case = props.generate(rng)
        ok, reason = props.check(case)
        if not ok:
            return [{"run": i, "case": repr(case)[:120], "reason": reason}]
    return []


def run_examples():
    p = subprocess.run(["pytest", "-q", "--maxfail=1"],
                       capture_output=True, text=True)
    tail = p.stdout.strip().splitlines()[-1:] or ["no output"]
    return p.returncode, tail


def main():
    manifest = json.loads(MANIFEST.read_text())
    ledger = json.loads(LEDGER.read_text()) if LEDGER.exists() else {}

    actual = fixture_digest(manifest)
    prop_failures = run_properties()
    code, tail = run_examples()

    if actual != manifest["digest"]:
        verdict = "BLOCK:fixture-drift"
    elif prop_failures:
        verdict = "BLOCK:property"
    elif code != 0:
        verdict = "BLOCK:example"
    else:
        verdict = "PASS"

    print(json.dumps({"verdict": verdict, "fixture": actual,
                      "quarantined": sorted(ledger),
                      "props": prop_failures, "examples": tail}, indent=2))
    return 0 if verdict == "PASS" else 1


if __name__ == "__main__":
    sys.exit(main())

One held-out property file does the real work. This one tests a dedupe helper without naming a single input.

# held_out_props.py — keep this file away from the agent's context.
def generate(rng):
    return [rng.randrange(0, 50) for _ in range(rng.randrange(0, 40))]


def check(items):
    out = dedupe(items)                       # function under test
    if len(out) > len(items):
        return False, "output grew"
    if not set(out) <= set(items):
        return False, "invented values"
    if out != list(dict.fromkeys(items)):
        return False, "order or dedupe shape changed"
    if dedupe(out) != out:
        return False, "not idempotent"
    return True, ""

The ledger stays small and boring. Quarantine is a fact with a number attached.

{
  "tests/test_billing.py::test_refund_rounding": {
    "stable_runs": 3,
    "repeat_required": 20,
    "reason": "depends on local date"
  }
}

Reading the verdict

Three booleans arrange into four outcomes. The table is the whole policy.

Property	Fixture digest	New flake	Verdict
fail	any	any	BLOCK — patch guilty
pass	drift	any	VOID — environment guilty
pass	match	yes	QUARANTINE — report, do not gate
pass	match	no	PASS — human review next

Most teams collapse rows two and three into "red". That collapse is why flaky suites get ignored. A verdict that never means anything specific stops being read.

Where the free tier fits

The gate needs two things to be useful: candidate patches and a place to run them. MonkeyCode's free model access supplies the first, and its free server option hosts the second, so the harness does not fight your laptop for CPU while you keep reviewing. The operator advertises a free token allowance; check the current terms for the exact number, since allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A loop that fits the three passes looks like this. Generate two or three candidate patches against the issue. Run python gate.py on each in the hosted environment. Keep only the candidates that reach PASS, then read those diffs yourself.

Nothing here needs a large model. It needs a model that produces varied diffs, and a gate strict enough to throw most of them away.

The free tier is a starting point, not a guarantee. Anything you would not run on a stranger's VM stays off it.

Limits, and who should skip this

Properties are hard to write when the correct output is itself contested. If nobody on the team can state the invariant, pass one will encode a guess, and a confident guess is worse than no gate.

Skip this order if your baseline is already red. Freeze the suite first, get one clean run, then add passes. A gate built on a broken baseline measures noise.

Also skip it for changes that need formal review or regulatory sign-off. A PASS here means the patch survived your invariants, not that it is safe.

The harness is roughly ninety lines and will save you the 02:40 page. Start with one property on the helper your agents keep rewriting. Then tell me which invariant finally caught a bad patch.