Characterization Tests Are Not Enough: Add a Diff-Budget Guard

Two failure modes kill a legacy refactor. Behavior drifts, or the change spreads across the repo. Characterization tests catch the first one only.

So I add a second gate before touching a messy module: a diff budget. You freeze the recorded call trace and the public symbol surface. CI then rejects any commit that spans too many Python files or drops a public name.

Order matters here. Record first, replay second, freeze the surface third, set the budget fourth. Only then move one function.

Step 1 — Record a real trace, not invented cases

Invented cases encode your assumptions about legacy code. A recorded trace encodes what the entry point actually received. Run the real path once with a fixed seed and a fixed clock, and log every boundary call.

# trace_recorder.py
import functools, json, pathlib
from wire import encode

LOG = pathlib.Path("traces/calls.jsonl")

def record(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        result = fn(*args, **kwargs)
        LOG.parent.mkdir(parents=True, exist_ok=True)
        row = {
            "fn": f"{fn.__module__}.{fn.__qualname__}",
            "args": [encode(a) for a in args],
            "kwargs": {k: encode(v) for k, v in sorted(kwargs.items())},
            "result": encode(result),
        }
        with LOG.open("a") as fh:
            fh.write(json.dumps(row, sort_keys=True) + "\n")
        return result
    return wrapper

Decorate functions at the module boundary, not every private helper. Fifty traced calls pin a parsing or pricing path well enough to refactor. If two runs produce different JSONL, stop and fix determinism first.

Step 2 — Replay with a strict wire codec

Replay is where most harnesses cheat. They compare repr() and hope. Use an explicit codec instead, and make unknown types raise loudly.

# wire.py
from decimal import Decimal

def encode(value):
    if isinstance(value, (str, int, float, bool, type(None))):
        return {"t": type(value).__name__, "v": value}
    if isinstance(value, (list, tuple)):
        return {"t": type(value).__name__, "v": [encode(v) for v in value]}
    if isinstance(value, dict):
        return {"t": "dict", "v": [[encode(k), encode(v)] for k, v in sorted(value.items())]}
    if isinstance(value, Decimal):
        return {"t": "Decimal", "v": str(value)}
    raise TypeError(f"add a wire codec for {type(value)!r}")

# tests/test_trace_replay.py
import importlib, json, pathlib, pytest
from wire import decode, encode

TRACE = pathlib.Path("traces/calls.jsonl")

def load_cases():
    return [json.loads(l) for l in TRACE.read_text().splitlines() if l.strip()]

def resolve(dotted):
    module_name, _, qualname = dotted.rpartition(".")
    obj = importlib.import_module(module_name)
    for part in qualname.split("."):
        obj = getattr(obj, part)
    return obj

@pytest.mark.parametrize("case", load_cases(), ids=lambda c: c["fn"])
def test_recorded_behavior(case):
    fn = resolve(case["fn"])
    args = [decode(a) for a in case["args"]]
    kwargs = {k: decode(v) for k, v in case["kwargs"].items()}
    assert encode(fn(*args, **kwargs)) == case["result"]

The codec is the safety valve. When a refactor introduces a new return type, TypeError fires loudly. Silent repr comparisons do not.

Step 3 — Snapshot the public surface with ast

Trace replay proves behavior. It does not notice that you moved parse_row out of reporting.parse. Capture the module's public names as a sorted list.

# surface.py
import ast, pathlib, sys

def public_symbols(src):
    return sorted(
        node.name
        for node in ast.parse(src).body
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
        and not node.name.startswith("_")
    )

if __name__ == "__main__":
    print("\n".join(public_symbols(pathlib.Path(sys.argv[1]).read_text())))

One decision hides in those lines: underscores mean private. Pick that rule before you start, not after the guard fails.

Step 4 — Enforce a diff budget in CI

Now the guard. It compares the working tree against a base commit and fails on two conditions: file count over budget, or a public symbol that existed at base and no longer exists.

#!/usr/bin/env python3
# guard_diff.py
import ast, pathlib, subprocess, sys

BUDGET_FILES = 3

def symbols(src):
    return sorted(
        n.name for n in ast.parse(src).body
        if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
        and not n.name.startswith("_")
    )

def symbols_at(base, path):
    proc = subprocess.run(["git", "show", f"{base}:{path}"],
                          capture_output=True, text=True)
    return symbols(proc.stdout) if proc.returncode == 0 else []

def symbols_now(path):
    p = pathlib.Path(path)
    return symbols(p.read_text()) if p.is_file() else []

def main(base):
    proc = subprocess.run(
        ["git", "diff", "--name-only", "--diff-filter=ACMR", base, "--", "*.py"],
        capture_output=True, text=True, check=True)
    changed = [line for line in proc.stdout.splitlines() if line]
    problems = []
    if len(changed) > BUDGET_FILES:
        problems.append(f"touched {len(changed)} Python files; budget is {BUDGET_FILES}")
    for path in changed:
        before, after = symbols_at(base, path), symbols_now(path)
        lost = [name for name in before if name not in after]
        if lost:
            problems.append(f"{path}: public symbols removed: {lost}")
    if problems:
        print("diff budget exceeded:")
        for item in problems:
            print(f"  - {item}")
        return 1
    print(f"diff budget ok: {len(changed)} Python files changed")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it in CI or a pre-commit hook:

python guard_diff.py origin/main

Measure your own baseline first. git diff --stat on your last supposedly small change tells you the real number. Mine started at three Python files. Pick yours from data, not ambition.

Step 5 — Make the smallest safe change

Extract one function into a new module. Keep the old module as the public door with a re-export.

# reporting/parse.py — public surface unchanged
from .parse_extracted import parse_row  # noqa: F401

The guard passes because no public symbol disappeared and only two files changed. Replay the trace and confirm identical results. If tests only pass after you edit an expected value, you changed behavior. Revert and split the commit.

Decision table

Signal after the change	What it means	Action
Trace replay green, surface identical, 2 files touched	Extraction is behavior-preserving	Commit with the shim
Replay green, one public symbol lost	You moved the door, not the code	Add the re-export, rerun guard
Replay fails on TypeError from the codec	A new type crossed the boundary	Add the codec, then re-examine the change
Replay fails with changed values	Behavior moved, not structure	Revert and split the diff
Guard fails on file count	The change is too wide for one commit	Cut at the extraction boundary

Where a free model and a free server help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, per the operator's description of current availability. I used that free model access to draft two things: the trace recorder and the first version of the AST surface extractor. Both drafts needed edits. The recorder used repr() for arguments, which breaks on sets and datetimes. The extractor compared whole files instead of public names. The corrections are the interesting part, and they came from running the replay, not from reading generated code.

The free server option matters for one narrow step: replaying a trace on a clean checkout. Your laptop carries environment drift your CI machine does not. A second short-lived machine that runs pytest tests/test_trace_replay.py on the same commit is a cheap second opinion. Free-tier terms change, so check the provider's current page before you build process on top of it.

Limitations and who should not use this

• Traces only cover the inputs you recorded. New input shapes are unguarded by design.
• The codec needs manual work for every non-primitive type. That cost is real on rich domain objects.
• A three-file budget is wrong for wide, mechanical changes such as import renames.
• If the module has no deterministic entry point, fix determinism first. This workflow assumes repeatable runs.
• If you cannot run the legacy path at all, characterization tests are unavailable. Read and diagram instead.

The number that matters

The budget is the whole article. Behavior tests tell you whether the refactor is correct. The budget tells you whether it is reviewable. On a messy repo, reviewable is the constraint that decides whether the change ships at all. If you try this on a legacy module, the budget number you land on will tell you more about your repo than the coverage report did.