Before the Smallest Safe Change, Prove Your Characterization Tests Can Fail

A green characterization suite proves agreement, not coverage. Your assertions may be broad enough to catch a flipped comparison. They may also be narrow enough to miss a dropped default. You cannot tell which from a passing run.

So audit the net before you lean on it. This workflow adds one gate between "tests pass" and "smallest safe change": mutation probes. Two mutant classes, one git worktree per mutant, one decision table, then the edit.

The Gate in One Paragraph

Write tiny patches against the module you plan to refactor. Label each patch preserving or breaking. Run your suite against every patched copy. A healthy suite fails on breaking patches and passes on preserving ones. Any other result is information about the net, not about the code.

Step 0: State Your Assumptions

This gate assumes three things. You have a characterization suite that passes today. You have the repo under git at a clean commit. You can run the suite in under a few minutes.

If any assumption fails, stop here. Write the characterization tests first. A mutation audit on top of a missing suite measures nothing.

Step 1: Capture a Baseline

Record the exact command and the current result. Store it next to the mutant set. You will compare every mutant run against this line.

cd repo
git status --porcelain   # must be empty
python -m pytest -x -q   # record the command and the summary line

Commit the harness and the mutant patches. Uncommitted mutants rot fast. Nobody remembers why a patch was labeled preserving six weeks later.

Step 2: Define the Two Mutant Classes

Breaking mutants change observable behavior. Flip < to <=. Change a default from None to {}. Drop one key from a returned dict. Swap two branches of an if.

Preserving mutants change structure only. Rename a local variable. Extract a temporary. Reorder two independent statements. Inline a single-use helper.

Keep the set small and honest. Ten breaking and five preserving mutants beat a hundred auto-generated ones you never read. Each patch should fit in a dozen lines.

I use the free model access in MonkeyCode to draft candidate breaking mutants from the target function. I paste the function, list its observable behaviors, and ask for behavior-changing edits as unified diffs. Every candidate gets human review before it becomes a file. Preserving mutants stay hand-written, because a model cannot know which structural edges your suite has frozen.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 3: Run One Worktree per Mutant

Never apply mutants to your working tree. Use git worktree so each run is isolated and parallelizable. See the git worktree docs for the underlying semantics.

# audit.py — usage: python audit.py /path/to/repo python -m pytest -x -q
import json, pathlib, subprocess, sys, tempfile

repo = pathlib.Path(sys.argv[1]).resolve()
suite = sys.argv[2:]
mutants = sorted((repo / "mutants").rglob("*.patch"))

def run(cmd, cwd):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=600)

results = []
for patch in mutants:
    kind = patch.parent.name  # "preserving" or "breaking"
    with tempfile.TemporaryDirectory() as tmp:
        wt = pathlib.Path(tmp) / "wt"
        run(["git", "worktree", "add", "--detach", str(wt), "HEAD"], repo)
        try:
            applied = run(["git", "apply", str(patch)], wt)
            if applied.returncode != 0:
                results.append({"mutant": str(patch), "kind": kind, "status": "apply-failed"})
                continue
            out = run(suite, wt)
            results.append({
                "mutant": str(patch),
                "kind": kind,
                "status": "killed" if out.returncode != 0 else "survived",
            })
        finally:
            run(["git", "worktree", "remove", "--force", str(wt)], repo)

print(json.dumps(results, indent=2))

Two failure modes look like bugs but are not. apply-failed means the patch drifted from HEAD; refresh it. A crash or import error counts as killed, which flatters your suite. Filter those out by checking the tail of the pytest output for collection errors.

Step 4: Generate Breaking Mutants Mechanical

Hand-writing ten breaking patches is slow. A small AST pass produces candidates fast. Python's ast module is enough here.

# gen_mutants.py — emits candidate patches for review, never commits them
import ast

FLIP = {ast.Lt: ast.LtE, ast.LtE: ast.Lt, ast.Gt: ast.GtE, ast.GtE: ast.Gt,
        ast.Eq: ast.NotEq, ast.NotEq: ast.Eq}

class FlipComparisons(ast.NodeTransformer):
    def visit_Compare(self, node):
        self.generic_visit(node)
        flipped = []
        for op in node.ops:
            cls = FLIP.get(type(op))
            if cls is None:
                return node          # mixed comparisons: leave untouched
            flipped.append(cls())
        node.ops = flipped
        return node

Round-trip with ast.unparse and diff the result against the original file. This drops comments, so treat output as scratch material. Read each candidate, keep the ones that change behavior, and delete the rest.

Step 5: Read the Matrix

Four outcomes matter. Everything else is noise.

Breaking survived	Preserving killed	Reading	Next action
none	none	suite is sharp and stable	run the refactor
some	none	blind spots	add one characterization test per survivor
none	some	over-fitted assertions	loosen: full-repr equality → field subset
some	some	mixed	fix blind spots first, revisit brittleness later

A surviving breaking mutant is a specific untested behavior. Name it in a comment and write the missing assertion. A killed preserving mutant is a test that will fight your refactor for no reason. Loosen it before the edit, not during it.

Step 6: Budget the Run Before You Start

Mutant cost is arithmetic, not vibes. Total wall clock is roughly suite_seconds × mutant_count, divided by parallel workers. Cap mutant count so the audit fits inside your review window.

# rough planning numbers, replace with your own measured suite time
# suite = 40s, mutants = 15, workers = 4  ->  ~150s

If the product exceeds your window, cut mutants. Do not cut the preserving class. That class is what keeps the net from blocking a legitimate change.

This is also where the free server option earns its place. A matrix of N worktrees is pure CPU and no interactivity, so it runs fine on a remote box while you read the diff. The harness above is plain CLI, so it moves without changes. I keep the suite command in one variable for exactly that reason.

Step 7: Apply the Smallest Safe Change

Now edit. One behavior, one commit, suite green. Then re-run the audit against the new HEAD.

Expect the breaking mutants to still die. A mutant that survived before and survives after is unchanged debt. A mutant that dies before and survives after means your refactor removed a tested behavior. That is the exact failure this gate exists to catch.

python audit.py . python -m pytest -x -q > after.json
# diff before.json after.json; investigate every status change

Limitations and Who Should Skip This

This gate measures your suite against your mutant set. A thin mutant set gives false confidence. Bias the set toward the behaviors the refactor touches.

The approach is poor fit for four situations. Hotfixes under a five-minute deadline. Modules whose tests require live services or hardware. Repos where the suite is already flaky, since flakiness masquerades as killed. And very large suites, where the arithmetic in Step 6 simply does not close.

It also adds no value if your characterization suite is missing. Mutation probing audits a net; it cannot weave one.

What You Get

You get a number, not a feeling. After one audit you know how many behavior changes your net catches, and how many structural changes it wrongly rejects. That is a defensible basis for the smallest safe change.

Start with five breaking mutants and three preserving ones. If the matrix already shows a blind spot, you have learned something cheaper than a production incident. Pick one module this week and run the audit before you touch it.