Run Characterization Tests Before Any Messy-Repo Diff

Do not start a messy-repo refactor with a rewrite. Capture live outputs as characterization tests before edits. Then ship the smallest diff that still passes.

This workflow treats current behavior as the contract. It does not assume the existing code is correct. It assumes production currently depends on that behavior.

The failure mode

Unscoped refactors mix cleanup with silent behavior drift. Reviewers cannot tell which lines were load-bearing. The merge looks neat and still breaks a caller.

Coding assistants often amplify that mix of changes. They rename, reformat, and drop edge branches together. Goldens separate style edits from true contract edits.

What you lock

Lock one path: inputs, stdout, stderr, and return data. Leave private helpers untouched until that path is pinned. One pinned path is enough for the first diff.

Do not freeze timestamps, process ids, or absolute paths. Normalize those fields before you hash the golden. Otherwise every run looks like a false regression.

Step 1: Inventory the blast radius

List every file that imports the messy module. Count call sites with a plain repository search. Stop at the first module with under six callers.

rg -n "from report import|import report" --type py

Pick the command that already exists in production. Do not invent a new CLI only for tests. Characterization must exercise the real entry path.

Step 2: Record a fixture and a live dump

Build a tiny input file that hits the weird branch. Keep the fixture in-repo and fully deterministic. Avoid network calls and wall-clock in that fixture.

mkdir -p messy_report/fixtures messy_report/goldens messy_report/tests
cat > messy_report/fixtures/orders.jsonl <<'EOF'
{"id":"A-1","qty":2,"flag":"rush"}
# skip me
{"id":"B-9","qty":0,"flag":""}
not-json
{"id":"C-3","qty":4,"flag":"rush"}
EOF

The next module is a labeled synthetic helper. It is not taken from a private codebase. It encodes cache, stderr, and separator quirks on purpose.

# messy_report/report.py
from __future__ import annotations

import json
import os
import sys
from pathlib import Path

CACHE = {}
SEP = os.environ.get("REPORT_SEP", "|")


def run(path, verbose=False, limit=None):
    key = str(path)
    if key in CACHE and not verbose:
        return CACHE[key]
    lines = Path(path).read_text(encoding="utf-8").splitlines()
    rows = []
    errors = 0
    for i, line in enumerate(lines):
        if limit is not None and i >= int(limit):
            break
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        try:
            item = json.loads(line)
        except json.JSONDecodeError:
            errors += 1
            if verbose:
                print(f"bad:{i}", file=sys.stderr)
            continue
        qty = int(item.get("qty") or 0)
        flag = item.get("flag") or "std"
        rid = item.get("id", "?")
        rows.append(f"{rid}{SEP}{qty}{SEP}{flag.upper()}")
    text = "\n".join(rows)
    if verbose:
        print(text)
        print(f"errors={errors} count={len(rows)}", file=sys.stderr)
    result = {"text": text, "errors": errors, "count": len(rows)}
    CACHE[key] = result
    return result


if __name__ == "__main__":
    path = sys.argv[1]
    verbose = "--verbose" in sys.argv
    limit = None
    for arg in sys.argv[2:]:
        if arg.startswith("--limit="):
            limit = arg.split("=", 1)[1]
    out = run(path, verbose=verbose, limit=limit)
    if not verbose:
        sys.stdout.write(out["text"] + ("\n" if out["text"] else ""))
    raise SystemExit(1 if out["errors"] else 0)

Dump the current CLI once, including streams and exit status. That dump is golden number one. Commit it before anyone touches report.py.

export REPORT_SEP='|'
python messy_report/report.py messy_report/fixtures/orders.jsonl --verbose \
  > messy_report/goldens/orders.stdout.txt \
  2> messy_report/goldens/orders.stderr.txt
echo $? > messy_report/goldens/orders.exit.txt

Step 3: Turn the dump into a fast test

Wrap the dump in a subprocess test, not a mock soup. Mocks hide the separator, cache, and stderr format. The point is the real process contract.

# messy_report/tests/test_orders_golden.py
from __future__ import annotations

import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
FIX = ROOT / "fixtures" / "orders.jsonl"
GOLD = ROOT / "goldens"
SCRIPT = ROOT / "report.py"


def _run():
    env = os.environ.copy()
    env["REPORT_SEP"] = "|"
    env.pop("PYTHONHASHSEED", None)
    proc = subprocess.run(
        [sys.executable, str(SCRIPT), str(FIX), "--verbose"],
        cwd=str(ROOT),
        env=env,
        capture_output=True,
        text=True,
        check=False,
    )
    return proc.stdout, proc.stderr, str(proc.returncode)


def test_verbose_orders_match_goldens():
    stdout, stderr, code = _run()
    assert stdout == (GOLD / "orders.stdout.txt").read_text()
    assert stderr == (GOLD / "orders.stderr.txt").read_text()
    assert code == (GOLD / "orders.exit.txt").read_text().strip()

Run the test against the untouched tree first. A red test here means the dump is wrong. Do not refactor while the golden harness is red.

pytest -q messy_report/tests/test_orders_golden.py

Step 4: Name one seam, then stop naming

Read the golden and list behaviors that callers can observe. Write them as a short contract list. Anything absent from that list stays eligible for later cleanup.

1. Blank lines and # comments are skipped.
2. Invalid JSON increments errors and prints bad:{i}.
3. Missing flag becomes STD after uppercasing.
4. REPORT_SEP joins id, qty, and flag.
5. Verbose mode prints rows on stdout and totals on stderr.
6. Nonzero errors yield exit status 1.

Choose one seam that cannot change those six facts. Extracting a line parser is a usual first seam. Renaming CACHE is not a first seam.

Step 5: Apply the smallest safe change

Change only the parse loop body. Keep run(), CACHE, and the CLI flags intact. The test must stay green without golden edits.

def _parse_row(line: str, index: int, sep: str, verbose: bool):
    try:
        item = json.loads(line)
    except json.JSONDecodeError:
        if verbose:
            print(f"bad:{index}", file=sys.stderr)
        return None, 1
    qty = int(item.get("qty") or 0)
    flag = item.get("flag") or "std"
    rid = item.get("id", "?")
    return f"{rid}{sep}{qty}{sep}{flag.upper()}", 0

Call _parse_row from the existing loop. Do not add type checks or new flags here. Extra safety work belongs in a later diff.

Re-run the same pytest command after the extract. If stdout, stderr, or exit status moves, revert. The extract is wrong even if the code looks cleaner.

Step 6: Re-run, then freeze the PR scope

A green golden means the seam is observationally identical. That is the only merge criterion for this PR. Formatting the rest of the package waits.

Add a second fixture only after the first PR merges. New fixtures expand coverage. They should not travel with the extract.

git add messy_report/report.py messy_report/tests messy_report/goldens messy_report/fixtures
git diff --cached --stat

If the cached stat shows extra files, unstage them. One helper extract plus goldens is the whole change. Broader trees hide the contract in noise.

Decision table for the next edit

Use this table when a teammate wants a bigger sweep. Each row is a stop rule, not a preference. Skip the row if goldens are missing.

Observed situation	Allowed change	Stop rule
No goldens on main	Record dumps only	Tests fail on current main
Goldens green, names ugly	Extract one helper	No golden file changes
Need a real behavior change	Update fixture, golden, and code	Diff shows all three together
Cache causes cross-test bleed	Reset CACHE in the test	Production cache logic unchanged
Separator comes from the environment	Pin REPORT_SEP in the harness	Do not hardcode it in run()
Assistant rewrites the module	Reject the patch	Re-apply the one-seam extract

The table is the original artifact for review. Paste it into the PR body. Reviewers then check rows, not taste.

Where a free model and free server fit

Local loops stall when the messy tree needs a clean runtime. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option.

Use the model to draft the subprocess harness from the contract list. Use the free server when your laptop cannot run the dump cleanly. Do not ask either one to rewrite report.py in one pass.

Feed the model the fixture, the CLI flags, and the six locked facts. Ask for a test file that compares streams and exit status. Discard any patch that edits production code and goldens together.

Common golden misses

Trailing newlines are the first false failure. Verbose stdout in this helper has no final extra blank line. Your dump command must not add one.

Working directory is the second false failure. The script should receive an absolute fixture path. cwd still needs pinning because imports may read relative files.

Environment is the third false failure. REPORT_SEP must be set in the test env. Unset hash seeds if later code starts hashing row keys.

Cache is the fourth false failure. CACHE keys on str(path) and skips work. Tests that call run() twice in-process must clear CACHE first.

import report

report.CACHE.clear()

Limitations

This method does not prove the messy module is correct. It proves the next diff did not change observed outputs. Wrong goldens will protect wrong behavior.

It also fails on un-normalized nondeterminism. Random ids, current time, and unordered sets will thrash. Normalize or reject those paths before recording dumps.

Binary outputs and huge snapshots are a poor fit. Prefer a sliced fixture that still hits the branch. Gigabyte goldens will not get reviewed.

Security-sensitive parsers need a different rule. If current behavior is an exploit, do not golden it. Break that path with an explicit regression test instead.

Who should not use this approach

Skip this if you are on a true greenfield module. There is no production contract to freeze. Write intent tests and implement forward.

Skip this if the change is an advertised API break. Update docs, versions, and caller patches in the same PR. Goldens would block the break you meant to ship.

Skip this if no one can run the real entry path. A reimplemented “test double CLI” is not characterization. It is a second system with new bugs.

Skip this if the team cannot keep goldens in review. Unreviewed snapshots rot and then get rewritten. At that point the harness is theater.

Close

Start with one module, one fixture, and one golden triple. Extract one seam only after that triple stays green. Stop the PR when the cached stat matches that scope.