Characterization Tests First: The Smallest Safe Refactor in a Scary Repo

TL;DR: Freeze current behavior into golden files before you touch the code. Prove the corpus catches real breakage with mutation checks. Then ship one behavior-preserving hunk and let a hash confirm nothing moved.

Why characterization comes first

Legacy code has no spec. Its spec is what it does today, including the parts that look wrong. So record today's outputs, and make every refactor answer to them.

Unit tests describe intent. Characterization tests describe reality. In a messy repo, reality is the only reference you can trust.

Step 0 — Freeze the environment

Goldens are stable only when the environment is stable. Pin the interpreter, the lockfile, and the source hash before recording anything.

python -V > env.txt
pip freeze > requirements.lock.txt
sha256sum legacy/pricer.py >> env.txt
git rev-parse HEAD >> env.txt

Set PYTHONHASHSEED=0 when you record. Dict ordering and set iteration can otherwise move your output between runs.

Step 1 — Build a corpus from real shapes

A useful corpus needs shapes, not volume. Twenty to fifty cases is usually enough to expose a refactor mistake.

Cover these categories explicitly:

1. The common path, recorded from one real call.
2. Empty collections and zero quantities.
3. Missing or null keys.
4. Boundary values: negative, huge, and exactly at a threshold.
5. One input that currently raises.

Store cases as JSONL, one object per line, each with a stable id.

{"id":"c001","region":"eu","vip":false,"items":[{"qty":2,"unit":10.0}]}
{"id":"c002","region":"us","vip":true,"items":[]}
{"id":"c003","region":"eu","items":[{"qty":1,"unit":0.0}]}
{"id":"c004","region":"apac","vip":false,"items":[{"qty":3,"unit":7.5}]}

Write cases by hand, or generate them from logged request shapes. Strip names, emails, account numbers and free-text fields before the file is committed.

Step 2 — Record goldens with a stdlib harness

The harness must capture return values and exception types. A crash is behavior too, and a refactor can easily change which error escapes.

#!/usr/bin/env python3
# tools/characterize.py
import hashlib, json, pathlib, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpus" / "orders.jsonl"
GOLDEN = ROOT / "golden" / "quote.jsonl"

def cases():
    for line in CORPUS.read_text().splitlines():
        if line.strip():
            yield json.loads(line)

def observe(case):
    from legacy.pricer import quote
    try:
        return {"id": case["id"], "status": "ok", "value": quote(case)}
    except Exception as exc:
        return {"id": case["id"], "status": "error", "error": type(exc).__name__}

def render(records):
    return "".join(json.dumps(r, sort_keys=True) + "\n" for r in records)

def digest(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def main():
    flag = sys.argv[1] if len(sys.argv) > 1 else "--check"
    text = render([observe(c) for c in cases()])
    if flag == "--record":
        GOLDEN.parent.mkdir(parents=True, exist_ok=True)
        GOLDEN.write_text(text)
        print("recorded", len(text.splitlines()), digest(text))
    elif flag == "--digest":
        print(digest(text))
    else:
        expected = GOLDEN.read_text()
        if text == expected:
            print("OK", digest(text))
            return 0
        for n, (got, want) in enumerate(zip(text.splitlines(), expected.splitlines()), 1):
            if got != want:
                print("DRIFT at line", n)
                print("  want", want[:160])
                print("  got ", got[:160])
                break
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it with the repo root on the import path.

PYTHONHASHSEED=0 PYTHONPATH=. python tools/characterize.py --record
PYTHONPATH=. python tools/characterize.py --check

Commit golden/quote.jsonl in its own commit, before any code change. That file is now the working spec.

Step 3 — Prove determinism before trusting goldens

A flaky golden is worse than no golden. Verify that three recordings in the same environment produce one hash.

for i in 1 2 3; do PYTHONPATH=. python tools/characterize.py --digest; done | sort -u | wc -l

One unique line means deterministic. More than one means you have time, randomness, network, or hash-order dependence. Freeze the clock, seed the RNG, stub the network call, then re-record.

Step 4 — Measure corpus strength with mutation checks

Line coverage counts code that ran. Mutation checks count behavior you would notice. Only the second one protects a refactor.

The sketch below copies the package into a temp directory, applies one textual mutation, and reruns --check. A mutant that leaves the harness green marks a hole in your corpus.

# tools/mutate.py (sketch — tune MUTANTS for your module)
import pathlib, shutil, subprocess, sys, tempfile

TARGET = pathlib.Path("legacy/pricer.py")
MUTANTS = [(" + ", " - "), ("round(", "int("), ("0.9", "0.8"), (">=", ">")]

def killed(source):
    with tempfile.TemporaryDirectory() as tmp:
        shutil.copytree("legacy", pathlib.Path(tmp) / "legacy")
        (pathlib.Path(tmp) / "legacy" / "pricer.py").write_text(source)
        env = {"PYTHONPATH": tmp, "PATH": "/usr/bin:/bin", "PYTHONHASHSEED": "0"}
        run = subprocess.run(
            [sys.executable, "tools/characterize.py", "--check"],
            capture_output=True, env=env,
        )
        return run.returncode != 0

original = TARGET.read_text()
trials = [m for m in MUTANTS if original.replace(m[0], m[1], 1) != original]
score = sum(killed(original.replace(a, b, 1)) for a, b in trials)
print(f"corpus strength: {score}/{len(trials)} mutants killed")

Treat anything below 80 percent as a warning sign. Add corpus cases until the score is boring. Note that this sketch replaces only the first textual match, so review the diff of each mutant before you trust the number.

Step 5 — Make the smallest safe change

Behavior-preserving extraction is the safest first move. Pull the loop out and keep the arithmetic order identical.

-def quote(order):
-    total = 0.0
-    for line in order["items"]:
-        total += line["qty"] * line["unit"]
-    if order.get("vip"):
-        total = total * 0.9
-    return round(total + TAX[order["region"]], 2)
+def base_total(items):
+    total = 0.0
+    for line in items:
+        total += line["qty"] * line["unit"]
+    return total
+
+def quote(order):
+    total = base_total(order["items"])
+    if order.get("vip"):
+        total = total * 0.9
+    return round(total + TAX[order["region"]], 2)

Floating-point addition order matters. Do not also convert += into sum() in the same commit, even if it reads better.

Verify, then commit only that hunk.

PYTHONPATH=. python tools/characterize.py --check
git add -p legacy/pricer.py tools/characterize.py
git commit -m "refactor: extract base_total, behavior unchanged"

The printed hash is your receipt. If it matches the pre-change value, no observable behavior moved.

Step 6 — Change behavior in a separate commit

Want to fix a rounding quirk or rename a field? Do it after the extraction, alone. Update goldens in that same commit and read the diff as a specification change.

PYTHONPATH=. python tools/characterize.py --record
git diff -- golden/quote.jsonl

Every changed line must be one you can explain out loud. Unexplained golden churn usually means your corpus is recording side effects, not logic.

Where MonkeyCode fits in this loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two operator-supplied capabilities are relevant here. MonkeyCode offers free model access and a free server option. I use them for the noisy parts of this loop, never for the authoritative part.

1. The free server option runs the record/check/mutate cycle when my laptop is busy or I want a clean process. Long mutant runs stop competing with local builds.
2. Free model access drafts extra corpus cases from anonymized schema shapes and reviews the single-hunk diff for missed edge cases.

The goldens still decide. A model suggestion that fails --check is a suggestion, not a fix. Build synthetic cases from schema rather than shipping records to any remote runner.

Limitations

• Characterization tests freeze bugs as well as features. Keep a short allowlist of known-wrong goldens, each with a comment, and fix them deliberately.
• Non-deterministic code cannot be characterized until you pin time, randomness, network and hash order. Fix that first.
• A module with no consumers and under a few hundred lines is often cheaper to rewrite than to characterize.
• Free-tier availability, limits and model selections can change. Keep the harness stdlib-only so the workflow survives a provider change.
• A remote runner cannot reproduce machine-specific side effects: local paths, environment variables, licenses, fonts.

When not to use this

Skip it for throwaway scripts, greenfield code with a real written spec, or any module nobody depends on. Use it when the code is scary, load-bearing and unclear about its own behavior.

Who this is for

Backend and platform engineers who inherited a repository they did not write. If you can run a script and read a hash, you can run this loop today.

Start with one function, one corpus file, one golden file, one hunk. If you want the mutant run to happen somewhere other than your laptop, the free server option is a low-friction place to try it.