Seed Five Mutations Before You Trust a Characterization Test

A green characterization suite proves nothing by itself. Prove the suite can fail before you trust it. Seed deliberate mutations into the messy code, then keep only the refactors your harness actually catches.

That order matters. Most teams write tests, see green, and start cutting. The harness was never tested, so nobody knows what it guards.

The Failure Mode: Green Suites That Never Bite

Most refactor plans open with "add characterization tests." The suite goes green, and everyone relaxes. Then the smallest extract lands, the suite stays green, and behavior still changed in production. The tests asserted on shapes, not on exact output. A harness that cannot fail is decoration. Fix that before you touch the extraction.

Step 1 — Choose One Observable Surface

Pick a single output you can compare byte for byte. A JSON report, a rendered receipt, a sorted row list. Two conditions matter. The surface must be stable across runs. The surface must be the one your extraction would disturb.

Write that choice in the pull request body. One surface beats five vague assertions. Freeze the environment at the same time. Pin the clock, the locale, and the sort order. Unpinned inputs turn real failures into noise, and noise hides missed seeds.

Step 2 — Record Golden Output, Not Assertions

Capture exact bytes. Do not hand-write assert result["total"] == 42. Hand-written assertions encode what you believe. Recorded output encodes what the code does.

# char_harness.py
import json
import pathlib

GOLDEN = pathlib.Path("tests/golden")

def check(name, callable_obj, *args, **kwargs):
    """Record on first run, compare on every later run."""
    result = callable_obj(*args, **kwargs)
    payload = json.dumps(result, sort_keys=True, default=str)
    path = GOLDEN / f"{name}.json"
    path.parent.mkdir(parents=True, exist_ok=True)
    if not path.exists():
        path.write_text(payload)
        return
    approved = path.read_text()
    assert approved == payload, f"characterization drift: {name}"

Wire it into a normal test file. Paths below are illustrative.

# tests/test_characterization.py
from char_harness import check
from billing.report import build_invoice

def test_invoice_bytes():
    check("invoice_1042", build_invoice, order_id=1042)

python -m pytest tests/test_characterization.py -q
git diff --stat tests/golden/

Commit the golden files. Treat a diff in them as a behavior change. Review that diff like production code.

Step 3 — Seed Mutations the Harness Must Catch

Now attack your own harness. Write small mechanical edits to the legacy code. Each edit must change behavior a caller could observe.

# seed_mutations.py
import argparse, json, pathlib, subprocess

SEEDS = [
    ("boundary_leak", "billing/tiers.py", "amount <= 100", "amount < 100"),
    ("default_swap",  "billing/tiers.py", "default=None", "default=Decimal('0')"),
    ("order_swap",    "billing/tiers.py", "sorted(rows)", "list(rows)"),
    ("rounding_drop", "billing/money.py", "quantize(CENT)", "quantize(CENT * 10)"),
    ("tz_drop",       "billing/window.py", "astimezone(utc)", "replace(tzinfo=utc)"),
]

def suite_fails() -> bool:
    done = subprocess.run(["python", "-m", "pytest", "tests/", "-q"],
                          capture_output=True)
    return done.returncode != 0

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--report", default="mutation-report.json")
    args = ap.parse_args()

    results = []
    for name, rel, old, new in SEEDS:
        path = pathlib.Path(rel)
        original = path.read_text()
        if old not in original:
            results.append((name, "anchor_missing"))
            continue
        path.write_text(original.replace(old, new, 1))
        try:
            results.append((name, "caught" if suite_fails() else "missed"))
        finally:
            path.write_text(original)

    caught = sum(1 for _, status in results if status == "caught")
    print(f"sensitivity: {caught}/{len(results)}")
    pathlib.Path(args.report).write_text(json.dumps(results, indent=2))

if __name__ == "__main__":
    main()

Rules for good seeds: one literal anchor, one replacement, revert after each run. Anchor strings must be literal text. If an anchor is missing, the seed is stale, and stale seeds lie about reach.

Step 4 — Score Sensitivity, Not Coverage

Report a ratio: caught seeds over total seeds.

python seed_mutations.py --report mutation-report.json

Output shape from a labeled illustrative run:

[
  ["boundary_leak", "caught"],
  ["default_swap", "caught"],
  ["order_swap", "missed"],
  ["rounding_drop", "caught"],
  ["tz_drop", "anchor_missing"]
]

That snippet is a format example, not a measurement of your repository. Read it as a to-do list. missed means your surface is too narrow. Add the missing field to the golden payload. anchor_missing means the seed file drifted. Fix the anchor before reading anything else.

Line coverage percentages will not help here. A file can show high coverage while every assertion passes on the broken branch. Sensitivity is the number that predicts refactor safety.

Step 5 — Make the Smallest Safe Change

Extract one thing. Change one call site. Keep output identical. Do not rename, reformat, or clean up while you are in there. Move the body, keep the signature, adjust imports.

python -m pytest tests/ -q
python seed_mutations.py --report mutation-report-after.json

The characterization suite must stay green. The mutation sweep must still catch the same seeds. If a seed stops being caught, your harness lost reach, not your code gained safety. Revert the extract and re-seed before retrying.

Two suites, one commit. That is the whole gate.

Decision Table

Harness state	Evidence	Action
5/5 caught, anchors matched	suite fails on every seeded behavior change	proceed with one extract
3/5 caught	surface too narrow	widen the golden payload, re-seed
any anchor_missing	seed file is stale	fix anchors first
5/5 caught, suite too slow per commit	feedback loop broken	run on the smallest subtree covering the extract
5/5 caught, golden diffs unreviewed	approval has no owner	assign a reviewer before extracting

Where Free Model Access and a Free Server Fit

Two parts of this loop are mechanical. Drafting candidate mutations is one. Running the sweep again and again is the other.

MonkeyCode offers free model access and a free server option, per the provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use them in that order. The model proposes anchors and replacements from the legacy file. I review every seed by hand before it enters the seed file. The server runs the sweep so the editing machine stays free. Neither step replaces golden review. The model proposes edits; the harness decides which ones matter.

The honest boundary: a generated seed is a hypothesis. It becomes evidence only after the harness catches it. A seed your suite misses tells you more about the suite than about the code.

Limitations and Who Should Skip This

Skip the sweep if you cannot pin your inputs. Nondeterministic output makes every seed look caught. Skip it if the legacy code exposes no comparable surface at all. Skip it if an audit requirement forces a signed toolchain you do not control.

Skip it if the plan is a rewrite. Characterization tests protect behavior you intend to keep. They fight you when the behavior change is the goal.

Costs are real. Every seed adds a full suite run. Five seeds mean five extra runs per sweep, and two sweeps per extract. Keep the seed set small and stable, and delete seeds whose anchors keep drifting.

Keep the Seeds in the Repo

The seed file is the durable artifact. Code moves, and the seeds move with it. Reviewers can see which behaviors the harness guards. New teammates run one command and learn the blast radius. If you want a second machine running the sweep while you edit, the free server option is one way to get it.