Kill Mutants in the Agent's Diff

Picture a Tuesday queue with fourteen agent patches waiting. Each one touches three files and arrives with a green suite beside it. Two tests in that suite fail at random, and the dashboard never says which two.

Reviewers read diffs slower than agents produce them. A full green run answers one weak question: did anything break? It never answers the useful one: can this patch's new logic fail at all?

I want a gate that answers the second question in minutes. It needs three mechanisms, and all three are boring on purpose. Patch-local mutation, coverage contexts, and a quarantine ledger.

Why the whole suite is the wrong instrument

Running everything is easy to build and hard to trust. The suite mixes evidence about the changed lines with evidence about code the patch never touched. A red test outside the diff blocks the merge anyway. A green run outside the diff proves nothing at all.

Mutation testing flips the problem around. Stop asking whether tests pass, and ask whether a deliberate defect survives. If you break the new comparison and every test still passes, the patch has no witness.

That is the fact I want before review, not after an incident. The catch is cost. Full-repo mutation on a large Python project is a weekend job, not a gate. So the mutation set must stay tiny and scoped to the diff.

Map the patch to the tests that actually ran

Start with changed line numbers, parsed from a zero-context diff so hunks stay small.

# diff_lines.py -- sketch, Python 3.11
import re, subprocess

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def changed_lines(base="origin/main"):
    diff = subprocess.run(
        ["git", "diff", "-U0", base, "--"],
        capture_output=True, text=True, check=True
    ).stdout
    files, current = {}, None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            current = line[6:]
            files.setdefault(current, set())
        elif line.startswith("@@"):
            m = HUNK.match(line)
            start, count = int(m.group(1)), int(m.group(2) or 1)
            files[current].update(range(start, start + count))
    return {f: sorted(n) for f, n in files.items() if n}

Now record which test executed each of those lines. coverage.py supports this with dynamic contexts, and the config change is one line.

# .coveragerc
[run]
dynamic_context = test_function

Run the suite once, then export JSON with the contexts included.

coverage run --rcfile=.coveragerc -m pytest -q
coverage json --show-contexts -o /tmp/cover.json

Intersecting the two files is short.

def tests_for_lines(cover_json, diff_map):
    data = json.load(open(cover_json))
    picks = set()
    for path, info in data["files"].items():
        targets = set(diff_map.get(path, []))
        for lineno, ctxs in info.get("contexts", {}).items():
            if int(lineno) in targets:
                picks.update(c for c in ctxs if c)
    return sorted(picks)

Two honest notes. A context appears only if that line executed during that test. Config branches, feature flags, and lazy imports can leave a changed line with no context at all. When that happens, the gate must return "no evidence" and never "pass."

Mutate only the lines the agent touched

Keep the operator set small and textual. Five replacements cover a lot of real logic: equality flips, boundary shifts, constant offsets, and boolean inversions.

MUTATIONS = [("==", "!="), ("<=", "<"), (">=", ">"), ("True", "False"), ("+ 1", "- 1")]

def propose_mutants(path, lines, limit=6):
    src = Path(path).read_text().splitlines()
    out = []
    for n in lines:
        for old, new in MUTATIONS:
            if old in src[n - 1]:
                out.append((path, n, old, new))
                break
        if len(out) == limit:
            break
    return out

Apply one mutant, run only the picked tests, then restore the file. A failing run means the mutant died, which is the result you want.

def run_mutant(root, mutant, picked):
    path, n, old, new = mutant
    original = Path(path).read_text()
    lines = original.splitlines()
    lines[n - 1] = lines[n - 1].replace(old, new, 1)
    Path(path).write_text("\n".join(lines) + "\n")
    try:
        r = subprocess.run(["pytest", "-q", "-x", *picked],
                           cwd=root, capture_output=True)
    finally:
        Path(path).write_text(original)
    return r.returncode != 0    # True == killed

Fixtures sit in the middle of this loop, and they must not drift between mutants. Hash the directory once, then record the digest next to the gate output.

def fixture_digest(root="tests/fixtures"):
    h = hashlib.sha256()
    for p in sorted(Path(root).rglob("*")):
        if p.is_file():
            h.update(p.read_bytes())
    return h.hexdigest()[:12]

If the digest changes mid-run, discard every result from that run. A mutant that dies because the input changed is not evidence of anything.

Quarantine beats deletion for flaky tests

A flaky test is not useless. It is simply untrustworthy as a witness. Deleting it hides a real race under the carpet. Leaving it in the gate makes every result random.

Keep a ledger instead, and let the gate read it. Each entry names the test, the observed failure mode, and the owner who agreed to fix it.

{
  "tests/pricing/test_round.py::test_half_up": {
    "reason": "shared clock, fails about 1 in 40 runs",
    "owner": "pricing",
    "added": "2026-10-06"
  }
}

One rule makes the ledger useful. A quarantined test can never be the witness that kills a mutant. If it is the only test covering a changed line, the gate returns "no evidence" and stays closed. The patch then waits for a test someone trusts.

The gate has three outcomes

Keeping the decision space small keeps the conversations short.

Outcome	Meaning	Action
killed_all	every diff mutant died	merge eligible for review
survivors	at least one mutant lived	request a test that fails
no_evidence	no trusted test covers the diff	gate stays closed

The gate prints the mutant list, the killed count, and the fixture digest. Reviewers read that log in twenty seconds and know exactly where the patch stands. Note that killed_all is not a badge of quality. It means the diff is pinned by something, which is the minimum bar for handing code to a human.

```bash # proposed run order, not a benchmark
python diff_lines.py > /tmp/diff.json
coverage run --rcfile=.coveragerc -m pytest -q
coverage json --show-contexts -o /tmp/cover.json
python gate.py --diff /tmp/diff.json --cover /tmp/cover.json --limit 6

## Limitations, and who should skip this

Equivalent mutants are the main noise source. Flipping a flag no test can observe produces survivors that mean nothing. Expect a nonzero equivalent rate, and review survivors by hand before blocking anyone.

Text-level mutation can also break syntax. Guard each mutation with a compile step, and skip it if the file stops parsing. Generated code with heavy metaprogramming tends to defeat this approach entirely.

Teams without per-test coverage contexts should not start here. Build the context data first, even if the gate arrives a quarter later. Repos with a suite already over thirty minutes should fix that first too.

The real cost is concrete. The gate adds one full coverage run per patch, plus a handful of short focused runs. On a quiet laptop that is fine. In a busy CI queue it is not, so scope it to risky patches only.

## Where MonkeyCode fits, briefly

Proposing candidate mutations on the diff lines is a narrow generation task, and it is the one place a model helps here. The operator states that MonkeyCode offers free model access and a free server option. The free server matters because mutation runs are bursty: idle most of the day, then six at once.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not claiming benchmarks in this piece, and any numbers from your own runs are yours to produce. Use the free access to propose candidate mutations, keep your thresholds yours, and treat a survivor as a question rather than a verdict. If you want to run this gate somewhere other than your laptop, the project is open source and the free server option is a low-risk place to start.