Make Each Characterization Test Fail Once Before You Refactor

A characterization test that cannot fail is decoration, not a safety net.

Most messy-repo refactors fail the same way. You record golden values, they all pass, and you feel safe. Then you extract a function and ship a silent behavior change. The goldens never noticed, because they were never able to notice.

Here is a workflow that fixes that. Record behavior first. Then prove each recorded case can fail. Only then make the smallest safe change.

The rule: a golden you have not falsified is a guess

Golden values freeze what the code does today, including its bugs. That is the point. But a passing test proves nothing on its own.

A test only earns trust when you can make it fail on purpose. Without that step, your suite may be asserting on an empty result, a swallowed exception, or a stub that never runs.

So the gate is simple. Every characterization case must survive one deliberate, minimal mutation of the code under test.

Step 1: Inject the seams before you capture anything

Messy functions hide their collaborators. Time, randomness, network calls, and global writers make goldens flaky.

Do not refactor yet. Patch those seams in the harness instead. Notice the patch is temporary and lives in test code.

# golden_capture.py
import json
from app.legacy import settle_order  # the messy 90-line function under test
import app.legacy as legacy

CASES = [
    {"id": "empty_cart", "args": [[], "US"]},
    {"id": "one_item", "args": [[{"sku": "A1", "cents": 500, "qty": 1}], "US"]},
    {"id": "qty_zero", "args": [[{"sku": "A1", "cents": 500, "qty": 0}], "US"]},
    {"id": "unknown_region", "args": [[{"sku": "A1", "cents": 500, "qty": 2}], "ZZ"]},
]

LEDGER = []

def spy(name):
    def wrap(*a, **kw):
        LEDGER.append((name, [repr(x) for x in a], tuple(sorted(kw.items()))))
        return 0
    return wrap

def snapshot(case):
    LEDGER.clear()
    legacy.charge_card = spy("charge_card")
    legacy.send_receipt = spy("send_receipt")
    try:
        return {"status": "returned", "value": settle_order(*case["args"])}
    except Exception as exc:
        return {"status": "raised", "type": type(exc).__name__, "msg": str(exc)}
    finally:
        pass

def full_snapshot(case):
    out = snapshot(case)
    out["calls"] = [{"fn": n, "args": a, "kwargs": dict(k)} for n, a, k in LEDGER]
    return out

The call ledger matters more than the return value here. Extracting a writer often preserves the result and reorders the side effects.

Step 2: Capture the corpus into a file you can diff

Write goldens to disk. Commit them. A reviewable diff beats a magic assertion.

if __name__ == "__main__":
    goldens = {c["id"]: full_snapshot(c) for c in CASES}
    with open("goldens.json", "w") as fh:
        json.dump(goldens, fh, indent=2, sort_keys=True)
    print(f"recorded {len(goldens)} cases")

Run it once against the untouched file. Inspect the JSON by hand. Delete any case whose recorded behavior looks like an artifact of your harness.

Step 3: Replay the goldens as a test

Keep the replay boring. One parametrized test, exact equality, no fuzzy matching.

# tests/test_goldens.py
import json
import pytest
from golden_capture import full_snapshot, CASES

GOLDENS = json.load(open("goldens.json"))

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_behavior_is_frozen(case):
    assert full_snapshot(case) == GOLDENS[case["id"]]

At this point every test passes. That is expected and meaningless. Step 4 is the one people skip.

Step 4: Mutation gate — prove the suite can fail

Mutate the target file in a scratch copy. If the suite still passes, that golden is untested for that branch.

# mutation_gate.py
import pathlib, re, shutil, subprocess, sys, tempfile

MUTANTS = [
    (r"if qty <= 0:", "if qty < 0:"),
    (r"total = 0\b", "total = 1"),
    (r"return total", "return total + 1"),
]

def run_gate(target="app/legacy.py"):
    src = pathlib.Path(target).read_text()
    survivors = []
    for pattern, repl in MUTANTS:
        mutated, hits = re.subn(pattern, repl, src)
        if hits == 0:
            continue
        with tempfile.TemporaryDirectory() as td:
            copy = pathlib.Path(td) / "repo"
            shutil.copytree(".", copy, ignore=shutil.ignore_patterns(".git", ".venv", "__pycache__"))
            (copy / target).write_text(mutated)
            r = subprocess.run(
                [sys.executable, "-m", "pytest", "tests/test_goldens.py", "-q", "-x"],
                cwd=copy, capture_output=True, text=True,
            )
        if r.returncode == 0:
            survivors.append(pattern)
    return survivors

if __name__ == "__main__":
    survivors = run_gate()
    print("survivors:", survivors)
    sys.exit(1 if survivors else 0)

A survivor means one of two things. Either no case exercises that branch, or your spy layer hides the effect. Add a case, not a comment.

Treat the counts in that output as illustrative. The real number depends on your function.

Where a free model actually helps

The expensive part is enumerating branches, not writing asserts. This is the narrow job I hand to a model: read the messy function and propose input classes I forgot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode's free model access and free server option, as described by the operator, to draft candidate cases and a first-pass mutation list. The model's output is raw material only. Every case still has to pass capture, replay, and the mutation gate before it earns a line in the corpus. The gate is the anti-hallucination step, since a model cannot verify its own tests.

Step 5: Make the smallest safe change

The gate is green when every mutant dies. Now touch production code once.

Extract one function. Change nothing else. No renames, no formatting, no logging tweaks in the same commit.

python mutation_gate.py && pytest -q && git diff --stat

If the golden diff is empty and the gate still fails every mutant, you have a real safety net. If a golden changes, stop and explain why before continuing.

Decision table: characterize, or walk away

Situation	Characterize first?	Why
Function you touch weekly	Yes, full corpus plus gate	The investment pays back fast
One-off script, deleted next sprint	No	Goldens outlive their value
Heavy nondeterminism, no injectable seam	Patch seams first, then yes	Flaky goldens teach nothing
Function with 40+ branches, no tests	Yes, but time-box it	Gate tells you when coverage is enough
Behavior you are about to delete	No, add deletion tests after	Freezing a bug is counterproductive

Limitations and who should skip this

Goldens freeze existing bugs, not correct behavior. That is intentional, and it is also a trap if you never revisit them.

Seam patching gets fragile. If collaborators are imported deeply, the harness grows faster than the refactor.

The mutation gate needs a fast test run. On a suite that takes minutes per case, this loop becomes unusable.

Skip this approach for prototypes, generated code, or any file scheduled for replacement. Also skip it if your team will not review the golden JSON in pull requests. Unreviewed goldens turn into noise nobody trusts.

Run the capture, run the gate, then make one small change. Ship the diff you can explain line by line. If you want a place to draft those first-pass cases, the free model access and free server option in MonkeyCode are a reasonable starting point.