Prove Your Refactor Tests Can Fail: A Mutation-Calibrated Gate for Messy Repos

The refactor that passed every test and still broke billing

The scenario below is a composite I assembled from common review patterns, not a customer story. A payments-adjacent repo carried a roughly 900-line adjuster.py: one public adjust(), three private helpers, and a currency table pasted into two of them. A cleanup extracted one helper into its own module, the test suite went green, and the diff merged the same afternoon.

Six weeks later, finance flagged that a slice of credit adjustments were one cent low on EUR invoices. The suite had never once disagreed with the code: it asserted that adjust() returned a dict with the expected keys, not what the values were. Green tests had proved only that the suite could not see the difference it was supposed to guard.

That gap is the subject of this article. A characterization snapshot proves your refactor preserved the behavior you recorded; a calibrated harness proves the recording can detect a behavioral change at all. The first is cheap and common. The second is what most cleanups skip, and it is the one that catches the extraction which silently widens a boundary condition.

Why a passing snapshot is weak evidence on its own

• Snapshots compare output against output produced by the same code, so any blind spot in the input corpus stays invisible.
• Line coverage measures execution, not detection: a line can run ten thousand times without a single assertion caring about its result.
• Agent-generated refactor diffs now land faster than a reviewer can reason about them line by line, which turns "read it carefully" into a hope.
• The failure mode after a mechanical edit is usually a quiet branch change, not an exception, so nothing pages you.

The loop below adds one missing step and one guardrail. You freeze inputs, snapshot outputs, then deliberately break the code and require the suite to notice. Only after the alarm is calibrated do you accept the smallest change that keeps it ringing.

Step 1: Freeze the input corpus behind a seed

Random input generation is fine as long as it is reproducible. Pin the seed, commit the generator, and add boundary rows by hand so the corpus does not drift between runs.

# tests/corpus.py
import random

SEED = 20261018


def cases(n: int = 400) -> list[dict]:
    rng = random.Random(SEED)
    rows = []
    for _ in range(n):
        rows.append(
            {
                "amount": round(rng.uniform(-500, 500), 4),
                "kind": rng.choice(["credit", "debit", "fee"]),
                "currency": rng.choice(["USD", "EUR"]),
            }
        )
    # Boundary rows matter more than the random bulk.
    rows.append({"amount": -0.005, "kind": "credit", "currency": "USD"})
    rows.append({"amount": 0.0, "kind": "fee", "currency": "USD"})
    return rows

Run the suite with plugins that shuffle order or reseed data disabled, because a nondeterministic corpus makes every later comparison meaningless. In pytest that means -p no:randomly, and it means no datetime.now() inside the code path under test.

Step 2: Snapshot the messy behavior as it is today

This part is a prerequisite rather than the interesting half, so keep it boring: record the current outputs to a checked-in file and diff against it.

# tests/test_adjuster.py
import json
import pathlib

from billing.adjuster import adjust
from tests.corpus import cases

GOLDEN = pathlib.Path("tests/golden/adjuster.json")


def test_snapshot_matches():
    actual = [adjust(**c) for c in cases()]
    assert actual == json.loads(GOLDEN.read_text())

A snapshot written against buggy code is still useful, because you are not approving the behavior, you are pinning it. The next step tells you whether the pin has teeth.

Step 3: Calibrate by breaking the code on purpose

Write a handful of mutants by hand: small, plausible bugs a careless edit could introduce. Then run the suite against each one and require a red result. A mutant the suite does not catch is called a survivor, and every survivor is a hole in your safety net rather than a flaw in the mutant.

[
  {"id": "rounding-mode", "old": "return round(amount, 2)", "new": "return math.floor(amount * 100) / 100"},
  {"id": "zero-boundary", "old": "if amount < 0:", "new": "if amount <= 0:"}
]

# tools/mutation_gate.py
"""Inject known bugs; the suite must fail on each. Non-zero exit means survivors exist."""
import json
import pathlib
import subprocess
import sys

TARGET = pathlib.Path("billing/adjuster.py")
MUTANTS = json.loads(pathlib.Path("tools/mutants.json").read_text())
TEST_CMD = [sys.executable, "-m", "pytest", "-q", "-p", "no:randomly", "tests/test_adjuster.py"]


def suite_passes() -> bool:
    return subprocess.run(TEST_CMD, capture_output=True).returncode == 0


def main() -> int:
    original = TARGET.read_text()
    if not suite_passes():
        print("baseline is already red; fix that before calibrating")
        return 2
    survivors = []
    try:
        for m in MUTANTS:
            if original.count(m["old"]) != 1:
                print(f'{m["id"]}: anchor is not unique, rewrite the mutant')
                return 2
            TARGET.write_text(original.replace(m["old"], m["new"]))
            if suite_passes():
                survivors.append(m["id"])
    finally:
        TARGET.write_text(original)
    total = len(MUTANTS)
    print(f"killed {total - len(survivors)}/{total} mutants")
    for name in survivors:
        print(f"  survivor: {name}")
    return 0 if not survivors else 1

The baseline check is not decoration. Without it, a mutant that breaks an import produces a red suite and gets scored as killed, which inflates your score while proving nothing.

Where the mutants come from, and why a model helps here

Hand-written mutants encode the bugs you already fear, which is a biased sample. A useful complement is asking a model to propose domain mutations: swap round for floor, invert a comparison, drop a default currency, reorder a deduction. I use MonkeyCode for that candidate step, because proposing twenty plausible mutants from one function signature is exactly the kind of cheap generation work where a free model allowance is enough; the operator describes free model access and a free server option, and the outreach material currently cites an allowance on the order of ten million tokens. Verify the current terms on the project page before you plan around them, since free-tier details change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two rules keep the model in its lane. First, every candidate is filtered by a human who can answer "would this change a value a customer sees?" Second, a proposed mutant that survives gets inspected before it gets trusted: some are equivalence mutants, meaning the change cannot alter behavior for any input in the corpus. Swap payload.get("currency", "USD") for payload.get("currency") or "USD" and you will usually get a survivor that reflects an equivalent rewrite rather than weak tests. Delete those, and record why in a comment next to the mutant file.

Practical sizing: five to ten mutants take under a minute on a small module, and each mutant forces one full suite run. That is where the free server option fits honestly, because a matrix of twenty mutants across a slow suite is the kind of job you start on a machine that is not your laptop.

Step 4: Budget the diff so the change stays small

With the alarm calibrated, the refactor itself should be embarrassingly small: one symbol moved, one branch tidied, no new abstractions. Enforce that with a script rather than a code review comment, and exclude test files from the count so the harness does not trip its own gate.

# tools/diff_budget.py
import subprocess
import sys

MAX_FILES = 1
MAX_LINES = 40

out = subprocess.run(
    ["git", "diff", "--numstat", "origin/main...HEAD"],
    capture_output=True, text=True, check=True,
).stdout

files, changed = [], 0
for line in out.splitlines():
    added, removed, path = line.split("\t")
    if path.startswith("tests/") or added == "-":
        continue
    files.append(path)
    changed += int(added) + int(removed)

if len(files) > MAX_FILES or changed > MAX_LINES:
    sys.exit(f"diff budget exceeded: {len(files)} source file(s), {changed} changed lines")
print(f"diff budget ok: {len(files)} file(s), {changed} lines")

Wire it after the test job, and let it fail the build. When the budget trips, the answer is not a bigger budget: it is a second, separately reviewed change with its own snapshot and its own calibration run.

The re-calibration trap most teams miss

After the change merges, run the mutation gate again against the new code. This is the step that catches a harness losing coverage as the implementation moves, because a mutant the old suite killed will sometimes survive the new one.

• Old code killed zero-boundary; new code survives it, which means the extraction moved the comparison behind a guard the corpus never exercises.
• Old code killed rounding-mode; new code still kills it, which is the signal you actually want.
• A mutant that starts failing for the wrong reason, such as an import error, is a broken mutant rather than a caught bug.

Track the survivor count in the pull request description the same way you would track a coverage delta. Two numbers, killed and total, tell a reviewer more about confidence than a paragraph of prose.

Decision table: when this loop is worth the hour

Situation	Run the loop	Skip it
Deterministic function handling money, dates, or permissions	Yes, every change
Module under ~100 lines with two call sites	Yes, it is minutes
Code scheduled for deletion this quarter		Yes, do not build a harness around a corpse
Behavior depends on wall clock, network, or device sensors	Only after pinning the clock and seed	Yes, until pinned
Suite already takes more than ten minutes	Yes, with a five-mutant sample	Full matrix on every commit
You are rewriting rather than refactoring		Yes, different problem entirely

Limitations and who should not use this

The approach assumes a deterministic code path, a suite that runs in isolation, and a git history clean enough to diff against a base branch. Teams whose "tests" touch a shared staging database will spend more time stabilizing the harness than refactoring, and they should fix that first. If your repo has no CI at all, the mutation gate becomes a script nobody runs, which is worse than no script because it looks like coverage.

Equivalence mutants are the other honest caveat: the gate measures detection by the corpus you wrote, so a survivor is a question, not a verdict. Budget reviewer time for judging each survivor rather than deleting the annoying ones. And on a large legacy module, ten mutants will not tell you the refactor is safe; they will tell you where your suite is blind, which is the more actionable result.

A small first move

Pick one function with a numeric boundary, write five mutants that you genuinely fear, and run the gate before touching the implementation. Report killed over total in the pull request, then make the smallest edit that keeps that ratio intact and rerun it afterward. If you would rather not tie up your laptop for the matrix, the free server option is the cheapest way to test-drive the loop on a throwaway branch, and the project page is the place to confirm what the free tier currently includes.