Can Your Agent Patch Gate Reject Anything? Run a Mutation Audit and Find Out.

Your review pipeline just approved every patch for six straight weeks, and the dashboard is gloriously green. That streak feels like evidence that your gate works, but it proves very little about its ability to catch defects. A gate's strength is defined by the worst thing it rejects, and a pipeline that has rejected nothing has no observed strength at all. This article walks through a mutation audit on a pool of fifty-two accepted agent patches, and the script below captures the exact procedure so you can run it on your own pool within an hour.

The idea behind the audit is simple enough to explain in one sentence. Take a patch your gate already accepted, inject exactly one semantic defect, and run the gate again. If the gate rejects the mutant, that defect class is covered. If the mutant passes, you have found a blind spot, and that blind spot is worth more than the original green checkmark.

Why a perfect acceptance record is a measurement gap

Review pipelines for agent patches typically stack three layers. Property checks assert invariants that should hold on every input, pinned fixtures reproduce the production environment, and a flaky-test freeze keeps nondeterministic failures out of the signal. Each layer contributes to the green record, so the audit measures them as one combined gate first and then disables layers to attribute every miss. The mutation audit does exactly this, and it never touches your production code.

The audit workflow, step by step

You can run the entire audit in six small steps, and each step is deliberately narrow.

1. Collect a folder of accepted patches, and use a chronological slice rather than a curated highlight reel.
2. Define mutant classes from your own bug history, because textbook defects are the ones your tests already catch.
3. Apply one mutation per run against exactly one file, and never combine two mutations in a single run.
4. Invoke your real gate against the mutated worktree, not a simplified mock version of it.
5. Exclude compile-dead mutants from the denominator, because they never reached the gate's verdict at all.
6. Print the detection rate per class, which is rejected mutants divided by live mutants.

Step six is where the honesty lives, because the rate comes from the gate's actual verdicts rather than from what you expect the tests to do.

The script

Holding the audit in one file keeps the procedure honest and reviewable. mutant_audit.py reads a pool directory, chooses a mutation class deterministically from the seed, applies one textual change to one file, compiles the result, and asks your configured gate for a verdict.

#!/usr/bin/env python3
# mutant_audit.py - one mutation per patch, then count gate rejections.

import argparse
import py_compile
import random
import subprocess
import tempfile
from pathlib import Path

MUTATIONS = {
    'off_by_one':   [('retries = 3', 'retries = 4'), ('retries = 3', 'retries = 2')],
    'sign_flip':    [('shipping_cost = -', 'shipping_cost = +')],
    'comparison':   [('status == COMMITTED', 'status != COMMITTED'),
                     ('status == REFUNDED', 'status != REFUNDED')],
    'guard_removed': [('if not order.is_paid():\n', '')],
    'logic_swap':   [('discount > 0.1', 'discount > 0.5')],
}

def apply_mutation(lines, cls):
    for i, line in enumerate(lines):
        for old, new in MUTATIONS[cls]:
            if old in line:
                lines[i] = line.replace(old, new, 1)
                return lines, '{!r} -> {!r}'.format(old, new)
    return None, None

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('pool_dir')
    parser.add_argument('--gate', default='./gate.py')
    parser.add_argument('--seed', type=int, default=20260828)
    args = parser.parse_args()

    rng = random.Random(args.seed)
    rows = []
    for path in sorted(Path(args.pool_dir).glob('*.py')):
        cls = rng.choice(list(MUTATIONS))
        lines = path.read_text().splitlines(keepends=True)
        mutant, _ = apply_mutation(lines, cls)
        if mutant is None:
            continue
        with tempfile.TemporaryDirectory() as tmp:
            worktree = Path(tmp) / 'review'
            worktree.mkdir()
            (worktree / path.name).write_text(''.join(mutant))
            try:
                py_compile.compile(str(worktree / path.name), doraise=True)
                dead = False
            except py_compile.PyCompileError:
                dead = True
            result = subprocess.run(args.gate, shell=True, cwd=worktree,
                                    capture_output=True)
            accepted = result.returncode == 0
            rows.append((cls, dead, not accepted))

    header = '{:<14}{:>6}{:>10}{:>8}'.format('class', 'live', 'rejected', 'rate')
    print(header)
    for cls in MUTATIONS:
        live = [r for r in rows if r[0] == cls and not r[1]]
        rejected = [r for r in live if r[2]]
        rate = 100 * len(rejected) / len(live) if live else 0.0
        row = '{:<14}{:>6}{:>10}{:>7.0f}%'.format(cls, len(live), len(rejected), rate)
        print(row)

if __name__ == '__main__':
    main()

You run it with a pool directory and a gate command, and the gate command must work from the scratch worktree that the script creates; the default seed keeps the synthetic run stable across machines.

python mutant_audit.py accepted_pool/ --gate "./gate.py --branch review"

Reading the verdict table

The table below is a representative run against the synthetic pool; your own numbers will shift, and the ordering of the rows is more stable than the percentages.

Mutant class	Example change	Live	Rejected	Detection
off_by_one	retries = 3 → retries = 2	11	8	73%
sign_flip	shipping_cost = - → +	9	9	100%
comparison	status == → status !=	13	5	38%
guard_removed	deleted if not order.is_paid():	8	2	25%
logic_swap	discount > 0.1 → > 0.5	9	3	33%

Overall, the gate caught twenty-seven of fifty live mutants, which lands at fifty-four percent. The average is almost useless here, and the minimum column is the actual message. Sign flips are fully covered because the property check asserts the exact shipping total, while removed guards slip through because no test touches the unpaid order path. The comparison class is weak because the pinned fixtures only ever produce one status value, and the frozen flaky-test suite cannot compensate for missing branches. Each weak row maps to a concrete investment: add an unpaid-order fixture for the guard class, a second status seed for the comparison class, and an exact discount invariant for the logic-swap class.

Where free models and a free server fit in the loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free models propose candidate mutation sites from the accepted patch pool, because a model reads the whole file instead of a diff summary. A human confirms each candidate before it enters the audit, since an unconfirmed proposal can be a non-mutation or a compile error in disguise. The weekly recalibration run moves to MonkeyCode's free server option whenever the local runner queue saturates. Both options are availability details from the operator's documentation, and this article claims no quotas, latency figures, or durability promises. The audit itself depends on neither option, and the script above runs against any gate you already run.

Turn the weakest row into a sentinel

An audit is a snapshot, and snapshots age badly. The durable version is a sentinel: a committed diff that the pipeline must reject before it reviews any real patch. Create one .diff file per mutant class in a canaries/ directory, and wire the check below into your pre-review hook.

#!/usr/bin/env bash
# sentinel.sh - prove the gate can reject before anything real is reviewed
set -euo pipefail

for diff_file in canaries/*.diff; do
    git apply "$diff_file"
    if ./gate.py --patch HEAD >/dev/null 2>&1; then
        git apply -R "$diff_file"
        echo "sentinel accepted: $diff_file - blocking the pipeline" >&2
        exit 1
    fi
    git apply -R "$diff_file"
done
echo "sentinel check passed: gate rejects every known mutant class"

A sentinel never ships, and it never gets reviewed for product behavior. Its only job is to test the reviewer, so it fails the pipeline the day someone loosens a property, unfreezes a flaky test, or removes a guard check. Without the sentinel, that regression would surface weeks later as a production incident with an invisible root cause.

Limitations

These caveats keep the audit honest:

• The detection rate describes your mutant menu, not your real defect distribution; real bugs still order from a menu you did not write.
• Per-class denominators are small at this pool size, so treat the lowest row as a strong hint rather than a precise figure.
• The script mutates at the text level, and some replacements can behave differently from a human bug in the same line.
• The gate runs against a scratch worktree, and production configuration drift can hide a miss that the audit would otherwise reveal.

Who should not use this yet

If your gate is merely compilation plus unit tests, run the audit once to confirm the obvious and then build the property layer first. If your review is entirely human, a mutation audit has nothing to attach to; you need a scripted gate before you need a scripted critic. If your accepted patch pool holds fewer than twenty items, the per-class denominators are noise, and the most honest finding will be the sample size itself.

The green streak was the unknown

A perfect record and an untested gate produce the same dashboard, and only a mutation audit can tell them apart. Run the script once, bring the table to the next review meeting, and argue about the weakest row before touching production code. The script in this article is the entire starting point, and the sentinel turns that one-time finding into a permanent regression check.