Survivor Mutants Block the Agent Merge

The morning review opened on a quiet agent patch. Twelve unit tests had passed in eleven seconds. Credit posting had been rewritten without a human draft.

A reviewer inverted one comparison in the production file. The suite stayed green after a full rerun. That silence was the defect hiding in the tests.

Agents search for a passing suite under time pressure. They do not search for faults they would miss. A tautology still feeds coverage tools and CI badges.

Tests should act like predators in a small habitat. Production lies are the prey those predators hunt. A predator that never kills is expensive decoration.

Mutation testing asks a blunt question during review. Did these tests notice a tiny production lie. If they did not, the badge is theater.

This workflow is a merge gate for agent patches. It plants small syntactic faults in touched files. The suite must kill those faults before anyone merges.

Full-file revert is a related but coarser check. Revert asks whether tests notice a full disappearance. Mutants ask whether tests notice a quiet falsehood.

Disappearance is loud and often trivial to detect. A flipped operator is quiet and more realistic. Agents are unusually good at shipping quiet falsehoods.

A ledger that looks tested

Consider a morning ledger example, not a memoir. The class credits an account once per key. Negative amounts must raise a clear domain error.

# ledger.py  (example)
class LedgerError(Exception):
    pass


class Ledger:
    def __init__(self):
        self._balances = {}
        self._seen = set()

    def credit(self, account, amount, key):
        if amount <= 0:
            raise LedgerError("amount must be positive")
        if key in self._seen:
            return self._balances.get(account, 0)
        self._seen.add(key)
        current = self._balances.get(account, 0)
        self._balances[account] = current + amount
        return self._balances[account]

An agent often adds tests that only poke types. The functions remain callable, so the job is green. Idempotency and bounds never come under threat.

# test_ledger_weak.py  (example)
from ledger import Ledger


def test_credit_returns_int():
    ledger = Ledger()
    value = ledger.credit("a1", 10, "k1")
    assert isinstance(value, int)


def test_credit_exists_on_instance():
    assert callable(Ledger.credit)

Those two tests survive brutal production edits. Delete the seen-key set and rerun the file. Flip the amount comparison and rerun again.

Return a constant integer from credit next. The weak suite still reports success. Coverage can rise while detection power stays near zero.

Plant mutants on the hunk

A useful gate does not need a research fuzzer. It needs a short, reviewable list of kills. Apply those kills only to files the agent touched.

The script below is a labeled local proposal. It copies the project into a temp tree. It applies one replacement, then runs pytest once.

# mutant_gate.py  (proposal)
from __future__ import annotations

import shutil
import subprocess
import sys
import tempfile
from pathlib import Path

MUTATIONS = [
    ("<=", "<"),
    (" in ", " not in "),
    (" + ", " - "),
    ("return ", "return 0  # mutant\n"),
]


def apply_one(text: str, old: str, new: str) -> str | None:
    if old not in text:
        return None
    mutated = text.replace(old, new, 1)
    if mutated == text:
        return None
    return mutated


def pytest_code(cwd: Path) -> int:
    completed = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "--maxfail=1"],
        cwd=cwd,
        capture_output=True,
        text=True,
    )
    return completed.returncode


def find_survivors(project: Path, target: str) -> list[tuple[str, str]]:
    survivors: list[tuple[str, str]] = []
    original = (project / target).read_text(encoding="utf-8")
    for old, new in MUTATIONS:
        candidate = apply_one(original, old, new)
        if candidate is None:
            continue
        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp) / "work"
            shutil.copytree(project, root)
            (root / target).write_text(candidate, encoding="utf-8")
            try:
                compile(candidate, str(root / target), "exec")
            except SyntaxError:
                continue
            if pytest_code(root) == 0:
                survivors.append((old, new))
    return survivors


if __name__ == "__main__":
    leftover = find_survivors(Path(".").resolve(), "ledger.py")
    if leftover:
        print("SURVIVORS", leftover)
        raise SystemExit(1)
    print("all planted mutants died")

Compile before pytest so invalid mutants never count. An invalid file is not a killed fault. It is a broken instrument, and you ignore it.

Run a clean baseline before you plant anything. A red or flaky baseline poisons the score. Freeze those flakes in a separate job first.

python -m pytest -q --maxfail=1
python mutant_gate.py

If the first command fails, stop the gate. You are measuring noise instead of detection power. Repair the suite, then return to the mutants.

The return-constant mutant exposes type-only assertions. An integer zero still satisfies isinstance checks. Behavior tests that name balances will fail it.

Tests that can kill

Replace the weak tests with behavior that can hunt. Name the amount guard and the duplicate key. Keep each test on a fresh ledger instance.

# test_ledger_strong.py  (example)
import pytest
from ledger import Ledger, LedgerError


def test_rejects_zero_amount():
    ledger = Ledger()
    with pytest.raises(LedgerError):
        ledger.credit("a1", 0, "k0")


def test_rejects_negative_amount():
    ledger = Ledger()
    with pytest.raises(LedgerError):
        ledger.credit("a1", -1, "k1")


def test_duplicate_key_does_not_double_credit():
    ledger = Ledger()
    first = ledger.credit("a1", 10, "k2")
    second = ledger.credit("a1", 10, "k2")
    assert first == 10
    assert second == 10

The weak suite leaves every planted mutant alive. The stronger suite kills the membership flip mutant. It also kills the plus-to-minus swap on credit.

The less-equal to less-than mutant is picky. Zero coverage alone can leave that mutant standing. The negative case is what finally hunts it.

That leftover is a map, not a trophy. It shows a boundary the agent never named. Reviewers can demand one more example and move.

Property tests with a single example are similar theater. They wear the costume of exhaustive property checking. Planted mutants still grade those properties as comments.

If you keep properties, raise the example budget. Then run the same mutant script immediately afterward. A property that never kills a mutant is unused ink.

How review should read the score

Read three outcomes without building a status dashboard. All planted mutants die, and tests stay isolated. Merge talk can return to design and naming.

Some mutants survive on operators outside the hunk. Narrow the replacement list to the changed diff. Broad replacement lists create false drama in review.

Many mutants survive on the agent's original tests. Reject those tests and park the production diff. Green coverage is not a substitute for hunting.

Shared fixtures still cheat a mutant kill score. Two tests can kill a fault by leftover state. Fresh objects keep that predator honest during review.

Clock skew and invented files belong in other gates. This gate answers only one narrow review question. Did the new tests observe a small lie.

A morning score on the example looks like this. Weak tests keep every planted mutant alive. Strong tests kill membership, arithmetic, and the return swap.

The comparison mutant dies only after the negative case. That is the review conversation worth having. The badge never mentioned that hole.

Some teams draft tests with a coding agent first. They then score those drafts against planted mutants. The loop wants a throwaway machine and a model.

MonkeyCode offers free model access and a free server option for that isolated loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate still runs on a laptop with pytest.

Do not let the model grade its own tests. Keep the model in a draft role only. Keep planted mutants in the editor role.

Discard callable-checks that never kill a fault. Keep only tests that change color under mutation. The draft engine is optional; the score is not.

Limits of the gate

Simple string replacements are not academic mutation testing. They miss precedence bugs and many short-circuit paths. They also miss semantic faults with identical syntax.

Do not run this gate on generated protobuf stubs. Do not run it on pixel goldens alone. Do not run it while the suite is flaky.

Skip the method for a one-line config change. Skip it for a thin vendor SDK wrapper. Mutation noise will drown the little signal there.

Skip it when tests still talk to live networks. A timeout can look like a killed mutant. That kill is weather, and weather is not evidence.

Stub the boundary, then plant faults inside modules. Keep the mutation list short enough to trust. Expand it only after false kills fall.

Agent patches will keep arriving with green badges. Green remains cheap inside this review workflow today. Fault detection remains the scarcer signal in review.

Make the suite hunt before you merge its story. Count dead mutants rather than passing helper functions. Silence after a lie is still a defect.