The Test That Passed on Broken Code: Mutation-Probing AI-Generated Tests

A passing test proves nothing. A test earns trust only when it fails on broken code. AI assistants generate tests in seconds. That speed turns the old habit — read the diff, trust the green — into a real risk. This article shows a five-minute mutation probe. It measures whether an AI-written test detects injected bugs. The probe is one Python file. It needs no framework changes.

The Trap: Green on Day One

The pattern repeats across codebases. The assistant writes test_adds_item_to_cart. The suite reports three passed. Everyone moves on. Nobody breaks the code to check. Later a refactor removes the real logic. The test still passes.

Typical causes: assertions on mocks instead of outcomes. Early returns that skip the real path. Sentinels that match both correct and broken code.

def test_discount():
    result = apply_discount(100, "SAVE10")
    assert result is not None  # passes for 90.0 and for 0.0

That assertion cannot fail. It counts as coverage. It protects nothing.

The Probe: Three Deliberate Bugs

Mutation probing is empirical. The probe injects one small bug per run. It executes the test suite after every injection. A test that survives a mutant is blind to that bug.

The probe ships three mutation kinds.

1. flip_eq — turns == into !=
2. drop_return — deletes the return value
3. bump_int — adds one to an integer constant

Three kinds are enough for a smoke check. Full mutation tools exist. This probe is a gate, not a research project.

#!/usr/bin/env python3
"""probe.py — check how many injected bugs an AI-written test detects."""
import ast
import subprocess
import sys
import tempfile
from pathlib import Path


class FlipEq(ast.NodeTransformer):
    def visit_Compare(self, node):
        for i, op in enumerate(node.ops):
            if isinstance(op, ast.Eq):
                node.ops[i] = ast.NotEq()
        return node


class DropReturn(ast.NodeTransformer):
    def visit_Return(self, node):
        node.value = None
        return node


class BumpInt(ast.NodeTransformer):
    def visit_Constant(self, node):
        if isinstance(node.value, int) and node.value != 0:
            node.value += 1
        return node


MUTATIONS = {
    "flip_eq": FlipEq,
    "drop_return": DropReturn,
    "bump_int": BumpInt,
}


def run_pytest(test_path: Path, cwd: Path) -> bool:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", str(test_path), "-q", "--tb=no"],
        cwd=cwd,
        capture_output=True,
        text=True,
    )
    return proc.returncode == 0


def main() -> None:
    if len(sys.argv) != 3:
        print("usage: probe.py <module.py> <test_file.py>")
        sys.exit(2)

    module_path = Path(sys.argv[1])
    test_path = Path(sys.argv[2])

    if not run_pytest(test_path, module_path.parent):
        print("baseline: FAIL — fix the test before probing")
        sys.exit(1)

    killed = []
    for name, transformer in MUTATIONS.items():
        tree = ast.parse(module_path.read_text())
        tree = transformer().visit(tree)
        ast.fix_missing_locations(tree)

        with tempfile.TemporaryDirectory() as td:
            tmp = Path(td)
            (tmp / module_path.name).write_text(ast.unparse(tree))
            (tmp / test_path.name).write_text(test_path.read_text())
            if not run_pytest(tmp / test_path.name, tmp):
                killed.append(name)

    print("baseline: PASS")
    print(f"mutants killed: {len(killed)}/{len(MUTATIONS)}")
    for name in MUTATIONS:
        print(f"  {name}: {'KILLED' if name in killed else 'SURVIVED'}")


if __name__ == "__main__":
    main()

Running the Probe

python probe.py cart.py test_cart.py

Use four steps.

1. Place probe.py next to the module under test.
2. Run it against the AI-written test file.
3. Read the kill rate.
4. Merge only when the rate is acceptable.

A concrete run looks like this.

$ python probe.py cart.py test_cart.py
baseline: PASS
mutants killed: 2/3
  flip_eq: KILLED
  drop_return: KILLED
  bump_int: SURVIVED

The test misses off-by-one errors. Add a boundary case. Regenerate. Re-run. The loop converges in minutes.

Reading the Kill Rate

Kill rate	Meaning	Action
0/3	The test never touches the mutated logic	Reject and regenerate
1/3	Happy path only	Add failure-path cases
2/3	Real coverage with gaps	Review and merge
3/3	Strong assertions	Merge, then check brittleness

The last row needs nuance. A perfect kill rate can mean over-assertion. The test may lock implementation details. If the test breaks on every harmless refactor, the cost shifts to the next developer.

Where Free Model Access Fits

Generation is no longer the bottleneck. MonkeyCode's free models make test generation cheap. Teams request five candidates instead of one. The probe ranks them by kill rate. The strongest candidate wins.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode also offers a free server option. The probe runs in a separate environment there. Mutants never touch the local working tree. A generated test cannot poison the checkout. Generation and verification share one clean space. That separation is the practical win.

The full loop is short. Generate tests with the free models. Run probe.py on the free server. Keep the candidate with the highest kill rate. Reject the rest. Merge with evidence, not vibes.

Limitations

The probe is narrow by design. It mutates one module's AST. It cannot find missing feature tests. If the AI never wrote a test for checkout(), no mutant reveals that gap. The probe measures assertion strength, not requirement coverage.

Heavy IO and network code need mocks. A fully mocked test can pass the probe while testing nothing real. Teams without CI should fix that before adopting this gate.

Who should skip this approach: teams that treat kill rate as a score. It is a smoke check, not a proof of correctness.

The Closing

Copy probe.py into the next pull request. Run it on the last AI-written test you merged. If the test survives all three mutants, it was never a test. It was a ceremony. Green does not mean safe. A killed mutant does.