The Gate Rejected Nothing. So I Injected 40 Bugs to See If It Could.

An agent-patch gate that never rejects a patch is a hypothesis, not a control. You cannot measure what it catches until you give it something to catch. So I stopped trusting real patches and started manufacturing failures.

For the last two weeks I have been running a three-stage review pipeline for agent-written code: property checks, pinned fixtures, and a freeze on flaky tests. Every real candidate passed. That was the problem. Passing real candidates only proves the sample was easy. Agent patches are easy by construction — the model sees the test suite, and it writes code that satisfies exactly what the tests say. The failures live in what the tests do not say: boundary values, error handling, fixture drift, nondeterminism.

The method that exposed those gaps is mutation testing applied to the gate itself. Take a clean patch, inject a known defect, run the full gate, and record whether the defect is caught. Repeat across bug classes. The output is a detection matrix that shows blind spots instead of vibes.

Pass rate is the wrong number

A gate score of 100% on real patches has two possible readings. Either the gate is excellent, or the patches are easy. The trend I keep seeing in review gates is the same one I see in benchmark harnesses: a number that looks like confidence and behaves like a guess.

The evaluation needs two numbers instead of one:

1. Seed recall — how many injected bugs does the gate catch?
2. Clean false-positive rate — how many known-good patches does the gate reject?

Set separate targets. I aim for 100% seed recall and under 5% false positives on a sample of clean patches. Both numbers are cheap once the harness exists. Neither is obtainable from production traffic alone.

The bug classes

Mutation testing only works if the seeds resemble real agent failure modes. From direct experience with C++ and Python agents, the signatures repeat: inverted comparisons, dropped boundary guards, swallowed exceptions, wrong fixture pins, and injected nondeterminism.

Seed class	Typical mutation	Expected gate layer
Inverted compare	if a < b: becomes if a >= b:	property check
Dropped zero guard	if value == 0: removed	property check
Swallowed error	except Exception: becomes except Exception: pass	property check
Wrong fixture pin	fixture("a") becomes fixture("b")	fixture pin
Injected sleep	time.sleep(random.random()) inserted	flake freeze
Removed invariant	assert x > 0 deleted	property check

Each row names the layer that should catch it. If a row lands in slipped_through, you have found a hole in your strategy — and found it before an agent ships it.

The harness

The harness is deliberately small. It takes a patch, applies one mutation, runs the gate, and classifies the outcome. This is the compact version; apply_patch and print_matrix are thin wrappers you likely already have in your CI tooling.

# seed_gate.py -- mutate an agent patch, then watch your gate react.
from collections import defaultdict
from pathlib import Path
import random
import subprocess
import tempfile

MUTATIONS = [
    ("invert_compare", lambda s: s.replace("if a < b:", "if a >= b:")),
    ("drop_zero_guard", lambda s: s.replace("if value == 0:", "")),
    ("swallow_error", lambda s: s.replace(
        "except Exception:", "except Exception: pass")),
    ("wrong_fixture", lambda s: s.replace(
        '@pytest.fixture(name="a")', '@pytest.fixture(name="b")')),
    ("inject_sleep", lambda s: s + "\n    time.sleep(random.random())"),
]

GATE_LAYERS = [
    ("property", ["pytest", "-q", "--maxfail=1", "tests/test_properties.py"]),
    ("fixture_pin", ["pytest", "-q", "tests/test_pins.py"]),
    ("flake_freeze", ["python", "freeze.py", "--check"]),
]

def run_gate(workdir: Path) -> str:
    for layer, cmd in GATE_LAYERS:
        result = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True)
        if result.returncode != 0:
            return f"caught_by_{layer}"
    return "slipped_through"

def main(patch_file: Path, repo: Path, seeds: int = 40) -> None:
    clean = patch_file.read_text()
    matrix = defaultdict(lambda: defaultdict(int))
    for _ in range(seeds):
        name, mutate = random.choice(MUTATIONS)
        with tempfile.TemporaryDirectory() as tmp:
            workdir = Path(tmp)
            workdir.joinpath("patch.diff").write_text(mutate(clean))
            apply_patch(repo, workdir / "patch.diff")  # your own wrapper
            matrix[name][run_gate(workdir)] += 1
    print_matrix(matrix)

Run it with python seed_gate.py agent_patch.diff /path/to/repo --seeds 40. Forty seeds, one command, one matrix.

Read the matrix, not the verdict

Here is an example of what the matrix looks like after the gate has been in place for a few days:

class              caught_by_property  caught_by_fixture_pin  caught_by_flake_freeze  slipped_through
invert_compare     8                    0                      0                       0
drop_zero_guard    5                    0                      0                       3
swallow_error      2                    0                      0                       6
wrong_fixture      0                    7                      0                       1
inject_sleep       0                    0                      9                       1

Two patterns stand out. Swallowed errors slipped through at a high rate: six of eight seeds. Wrong fixture pins escaped once. Both classes look fine in unit tests — errors are quietly ignored, and fixture drift only bites in integration.

The action is concrete. Add a property that asserts the function fails loudly on invalid input. Add an explicit fixture-name check to the review layer. Rank fixes by the detection gap, slipped_through / total per class, and close the largest gap first.

Run it on a budget

Generating 40 seed candidates by hand is drudge work. I generate candidates with a free model endpoint and hand-verify every mutation before it runs. A mutant that is not actually a defect pollutes the matrix, so verification is not optional.

MonkeyCode's free model access keeps this loop cheap, and its free server option gives the harness a disposable place to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two caveats from production scars: a free endpoint has returned HTTP 200 with an empty body in my own runs, so every generation gets a lease-based check instead of a blind retry; and a free server is an acceptable home for a calibration harness, not for production traffic. Treat both as capabilities with limits, not as guarantees.

Limitations

Seeds are adversarial samples, not a distribution. 100% seed recall proves the gate catches the bugs you thought of. It says nothing about the bug you did not encode. Mutation classes also decay: a model learns the review gate over time, and next quarter's failures will look different from this quarter's seeds. Refresh the suite on a schedule.

The false-positive side matters just as much. A gate that rejects everything scores 100% recall. That is why the clean-patch sample is not optional. Run the harness in a mode that skips mutation, count rejections on known-good patches, and investigate the gate if the rate climbs past 5%.

Who should not use this

Skip the seed harness when patch volume is tiny and blast radius is small. A weekend script that gets one human review does not need a calibration loop. Skip it when you cannot verify mutations by hand — a bad seed suite produces a confident, wrong matrix. Skip it while the gate itself is still changing daily; calibrating a moving target means chasing your own noise.

When the pipeline is stable enough to be boring, that is the moment to test it. Nothing here is exotic: 40 seeded bugs, three test commands, one matrix. Seed your own gate and see what falls through. The first run is usually embarrassing. That is the point.

Comments

[Kane Lim]

This is a strong way to validate an agent-patch gate because you’ve separated “the gate passed” from “the gate is actually effective.” That distinction is easy to miss when evaluating AI-generated code.

The detection matrix is particularly useful. Looking only at aggregate pass/fail results can hide exactly where the control is weak. Measuring seed recall by failure class makes the gaps actionable—for example, swallowed exceptions slipping through suggests the gate is validating expected behavior but not sufficiently validating failure semantics.

One point I’d consider extending is the distinction between mutation adequacy and behavioral adequacy. A mutation can be syntactically valid and still fail to represent a realistic agent defect. Over time, the seed corpus should probably be derived from historical agent failures in addition to hand-designed mutations. That gives you a feedback loop:

production failure → generalized mutation class → calibration seed → gate improvement → regression seed

I’d also make the clean-patch false-positive measurement continuous rather than treating 5% as a universal threshold. The acceptable rate depends heavily on the cost of a rejected patch and the review workflow. More important is detecting statistically meaningful drift in both false positives and mutation recall.

There’s another interesting opportunity here: have the agent propose mutations, but keep the oracle independent from the agent. Otherwise the evaluation can gradually become optimized for the same assumptions that produced the original patch. Independent mutation generation, deterministic execution, and periodically refreshed historical seeds could make the calibration much harder to game.

The core principle is excellent: a verification system should be tested against controlled failures, not validated merely by observing successful executions. This pattern should become increasingly important as autonomous coding agents become part of CI/CD.

I work with a small Canada-based remote development team focused on software engineering, AI automation, and developer tooling. We’re interested in building long-term relationships with developers working on problems like agent reliability, testing, and evaluation.