Stop Hand-Writing a Hundred Characterization Tests: Shrink a Recorded Corpus Instead

Characterization suites usually die of bloat, not of missing cases. Hand-written tests guess at the input space and pin behavior nobody calls. Record real calls at the seam, shrink them with coverage, then extract. That order is the whole article.

This is a corpus budget, not a coverage target. You want the fewest fixtures that still reach every line you are about to move. Everything else is documentary dead weight.

Why the hand-written pass fails

You sit in front of an unfamiliar module and invent inputs. Each invented input encodes your assumptions about the domain. Those assumptions are exactly what a refactor must not change, and exactly why the suite can feel safe while missing production shapes.

The result is a suite with ninety cases and three real ones. Maintenance cost lands on whoever inherits the branch. The suite decays, and the refactor happens anyway.

Recording inverts the direction. Production tells you which inputs exist today. Coverage tells you which subset of them is worth freezing.

Step 1 - Record at the seam

Wrap the function you plan to extract. Log arguments and results in a canonical form.

# recorder.py - illustrative harness; adapt names, then run it locally
import json, functools, pathlib, hashlib

CORPUS = pathlib.Path("corpus/calls.jsonl")

def canonical(obj):
    return json.dumps(obj, sort_keys=True, ensure_ascii=False, separators=(",", ":"))

def snapshot(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        call = {"args": [canonical(a) for a in args],
                "kwargs": {k: canonical(v) for k, v in sorted(kwargs.items())}}
        result = fn(*args, **kwargs)
        payload = canonical(call) + canonical(result)
        call["sha256"] = hashlib.sha256(payload.encode()).hexdigest()
        with CORPUS.open("a") as fh:
            fh.write(json.dumps(call) + "\n")
        return result
    return wrapper

Three rules keep the corpus usable. Sort keys, strip timestamps, never log object ids or memory addresses. An unstable corpus turns into a flaky suite within a week.

Redact personal fields at write time. Recording is the last moment where you still control the data.

Step 2 - Score each call by lines reached

You do not need a coverage service to rank cases. The standard library does this.

# score.py - line reach per recorded call
import trace, sys

def lines_touched(target_file, fn, args):
    tracer = trace.Trace(count=1, trace=0, ignoredirs=[sys.prefix, sys.exec_prefix])
    tracer.runfunc(fn, *args)
    return {lineno for (filename, lineno) in tracer.results().counts
            if filename == target_file}

Replay the corpus once and write a small index. Case id maps to a line set and a result hash.

python -m recorder replay corpus/calls.jsonl > corpus/index.json

Now you can answer the only question that matters. Which recorded calls reach lines that nothing else reaches?

Step 3 - Shrink with greedy set cover

Coverage selection is set cover. It is NP-hard in theory and trivial in practice. Greedy lands within a small constant factor in seconds.

# shrink.py - greedy cover over line sets
def shrink(index, universe):
    remaining = list(index.items())
    chosen, covered = [], set()
    while covered != universe and remaining:
        cid, lines = max(remaining, key=lambda kv: len(kv[1] - covered))
        gain = lines - covered
        if not gain:
            break
        chosen.append(cid)
        covered |= lines
        remaining = [(k, v) for k, v in remaining if k != cid]
    return chosen, covered

Two guards matter more than the algorithm. Keep every case that raises an exception. Keep a hand-written edge list as well: empty string, None, non-ASCII text, and the largest value your logs ever showed.

Greedy minimizes line reach only. It is blind to correlated values. Rounding and money math need extra care, which I cover below.

Step 4 - Freeze the survivors as bytes

Fixtures are golden files. One canonical JSON in, one canonical JSON out, plus a hash next to it.

def fixture(case_id, fn, args):
    out = canonical(fn(*args))
    path = pathlib.Path("tests/fixtures") / f"{case_id}.json"
    path.write_text(out, encoding="utf-8")
    path.with_suffix(".sha256").write_text(
        hashlib.sha256(out.encode()).hexdigest())

Byte comparison is the point. Rounding, ordering, and whitespace all surface as diffs instead of silent drift. Run the frozen suite before you touch anything.

pytest tests/characterization -q

Step 5 - Make the smallest safe change

Extract one function. Keep the original public name as an alias at the old import path. Do not rename parameters, reorder arguments, or fix the naming you dislike. Those are separate commits with separate evidence.

# before: billing/invoice.py holds build_totals()
# after
from billing.totals import build_totals  # noqa: F401  (alias keeps call sites stable)

Re-run the frozen suite. Any byte diff is a behavior change, intended or not. Fix the extraction, not the fixture, unless you can write down why the new bytes are correct.

Step 6 - Watch for the line-number trap

Here is the failure that bites people who like this method. Coverage-based selection is valid only for the pre-refactor module. After the move, line numbers shift, so re-running the selection picks a different corpus and a different story.

Never re-select coverage from the refactored module to validate the refactor. Validate with bytes. Coverage chooses the corpus once, at the start.

Add one more alarm. Log when production reaches a line your extracted module never saw during replay. New reachable code after a supposedly pure move means the extraction changed control flow.

Decision table

Situation	Record and shrink	Hand-write
Large input space, real traffic	Yes	Too slow to guess
Under about 15 branches	Overkill	Yes
Money, tax, rounding	Yes, plus hand-written edge values	Those values still needed
Sensitive payloads	Redact at record time	Prefer synthetic data
Path you cannot replay offline	No	Yes
One-line rename	Not worth it	Yes

Limitations and who should skip this

Recording cannot capture inputs production never sends. It inherits every blind spot of current traffic.

Coverage minimization is blind to value correlation. Two branches may need values that must vary together, and the shrinker will happily keep only one of them.

Exception paths are usually under-recorded. Offline batch jobs and rare error branches still need hand-written cases.

Corpora carry personal data. If you cannot redact reliably, do not record at all.

Skip the whole approach when the target function is small, when traffic is thin, or when a clean rewrite with fresh tests is cheaper than preserving behavior.

Where free tooling fits

The mechanical parts here are boring, and boredom is where free tiers help. I used MonkeyCode's free model access to draft the recorder and shrinker harness, then to triage fixture diffs into "intended" and "suspicious" piles. Its free server option ran long replay sweeps over the recorded corpus while my laptop stayed on code review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability and limits change, so check the current terms before you plan around them. Treat any free tier as a convenience, never as the evidence behind a refactor.

Closing checklist

1. Record at one seam for a day.
2. Score every call by lines reached.
3. Shrink with greedy set cover plus a hand-written edge list.
4. Freeze survivors as canonical bytes and hashes.
5. Move one function. Keep call sites stable.
6. Verify by byte diff, never by fresh coverage.
7. Alarm on any line production reaches that replay never did.

The suite you keep should be small enough to read and strong enough to argue with. Spend that budget on the code you are actually moving, not on the size of your test folder.