Do not edit a tangled module on first contact. Record inputs, outputs, and side effects before any rewrite. Then change the smallest surface that still compiles.
Messy repositories punish wide, unfenced refactors every time. Hidden callers depend on accidental return shapes today.
A coding model will happily rewrite the whole file. Characterization tests freeze behavior before that rewrite starts.
Why characterization beats a clean-room rewrite
A characterization test does not encode product intent. It records what the current code actually does.
That record becomes a regression fence for one edit. Golden values look ugly and still beat silent drift.
Drift is the usual cost of cleanup in legacy Python. AI-assisted edits raise that cost without a fence.
Models optimize for local elegance over hidden coupling. They miss import-time globals and file side effects. Pin those effects before inviting any model in.
Scope one function, not the package
Pick one entry point with a stable public name. Ignore private helpers on the first recording pass.
Helpers move only after the golden fence exists. A good target mixes computation with local I/O.
It reads env, opens a path, or mutates a cache. Pure arithmetic is the wrong first candidate here.
Stop if the function launches live network calls. Stub those seams in a later, separate pass. This workflow assumes local and deterministic fixtures only.
Artifact: a golden I/O table
The artifact is a JSON table of recorded calls. Each row stores args, kwargs, return, and exception type.
Optional columns store selected filesystem and env snapshots. The table is generated rather than hand-written.
A thin wrapper records live calls during a fixture run. Replay then compares new behavior against that table.
Label the following code as a proposed harness. Adapt names to the real module under test.
# characterize.py — proposed harness, not production code
from __future__ import annotations
import hashlib
import json
import os
import traceback
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Callable
GOLDEN = Path("golden_io.json")
@dataclass
class Row:
name: str
args_repr: str
kwargs_repr: str
result_repr: str
exc_type: str | None
env_pin: dict[str, str]
path_sha: dict[str, str]
def _repr(value: Any) -> str:
try:
return json.dumps(value, sort_keys=True, default=str)
except TypeError:
return repr(value)
def _sha256(path: Path) -> str:
if not path.exists():
return "missing"
digest = hashlib.sha256()
digest.update(path.read_bytes())
return digest.hexdigest()
def record(
fn: Callable[..., Any],
name: str,
args: tuple[Any, ...],
kwargs: dict[str, Any],
env_keys: list[str],
watch_paths: list[Path],
) -> Row:
env_pin = {k: os.environ.get(k, "") for k in env_keys}
before = {str(p): _sha256(p) for p in watch_paths}
exc_type = None
result: Any = None
try:
result = fn(*args, **kwargs)
except Exception as exc: # characterization, not a swallow
exc_type = type(exc).__name__
result = traceback.format_exc(limit=2)
after = {str(p): _sha256(p) for p in watch_paths}
path_sha = {k: f"{before[k]}->{after[k]}" for k in after}
return Row(
name=name,
args_repr=_repr(args),
kwargs_repr=_repr(kwargs),
result_repr=_repr(result),
exc_type=exc_type,
env_pin=env_pin,
path_sha=path_sha,
)
def write_golden(rows: list[Row]) -> None:
payload = [asdict(r) for r in rows]
GOLDEN.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
def assert_golden(rows: list[Row]) -> None:
current = [asdict(r) for r in rows]
if not GOLDEN.exists():
raise SystemExit("missing golden_io.json; run with --record first")
expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
if current != expected:
GOLDEN.with_suffix(".got.json").write_text(
json.dumps(current, indent=2, sort_keys=True), encoding="utf-8"
)
raise AssertionError("characterization mismatch; see golden_io.got.json")
Keep the golden file inside version control always. Treat a mismatch as a failed refactor, not noise.
Do not regenerate goldens to silence a red test. Regeneration is allowed only after an intentional behavior change.
Numbered workflow
1. Inventory callers without changing code
Search for the function name with a bounded grep. Record each call site in a short table.
Note argument shapes, not opinions about design. Nested helpers must wait for a later extract.
rg -n "def process_batch|process_batch\(" -g "*.py"
Stop after the public name is fully inventoried. Do not refactor during this inventory pass at all.
2. Freeze env keys and watched paths
List the environment keys the function actually reads. List files that it may create or rewrite.
Empty lists mean the target is too pure. Choose a messier entry point in that case.
ENV_KEYS = ["APP_MODE", "BATCH_ROOT"]
WATCH = [Path("var/last_batch.json")]
Commit that list before recording any golden rows. Changing the list later invalidates the golden table.
3. Record one fixture corpus
Drive the function through small, real-ish fixtures. Include the empty case and one error case.
Three to seven rows beat a giant unreadable dump. Name each row after the fixture, not the outcome.
# record_corpus.py — proposed fixture driver
import sys
from pathlib import Path
from characterize import assert_golden, record, write_golden
from messy.batch import process_batch # stand-in import
ENV_KEYS = ["APP_MODE", "BATCH_ROOT"]
WATCH = [Path("var/last_batch.json")]
CASES = [
("empty", (Path("fixtures/empty"),), {}),
("one_row", (Path("fixtures/one_row"),), {"strict": False}),
("bad_ext", (Path("fixtures/bad_ext"),), {"strict": True}),
]
if __name__ == "__main__":
mode = sys.argv[1] if len(sys.argv) > 1 else "--record"
rows = [
record(process_batch, name, args, kwargs, ENV_KEYS, WATCH)
for name, args, kwargs in CASES
]
if mode == "--record":
write_golden(rows)
elif mode == "--replay":
assert_golden(rows)
else:
raise SystemExit("use --record or --replay")
Run the driver once on a clean working tree. Commit golden_io.json with the driver in one change.
Do not edit the JSON file by hand. Hand edits hide the real behavior under test.
4. Replay before any refactor
Replay must pass on the current HEAD commit. A red replay means fixtures or env pins are dirty.
Fix the fixtures before touching any production code. A green replay is the only start signal.
python record_corpus.py --replay
Wire replay mode to assert_golden in the driver. Keep record mode behind an explicit CLI flag.
5. Permit one structural change
Allowed production changes stay small and strictly mechanical. Rename a local, extract a helper, or drop dead code.
Forbidden changes include new I/O and new defaults. If replay stays green, stop the commit there.
Do not chain a second extract in the same commit. One green fence should guard one structural change. Split further cleanup into later, fenced commits.
6. Use a model only after the fence exists
A free coding model can propose extra fixture rows. It should not rewrite the tangled function first.
Feed it the golden table and the caller inventory. Reject any patch that spans more than one helper.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.
Use the free server when local env pins leak. Ask the free model for extra fixture rows only.
Review every proposed row against the caller inventory. Do not accept a full-file rewrite from the model.
Paste one helper extract at most after review. Replay the golden table immediately after that paste.
Decision table
The table below is the review gate for diffs. Use it on every refactor pull request.
| Signal on HEAD | Action | Stop condition |
|---|---|---|
| No golden file | Record three to seven rows | Replay is green |
| Golden mismatch before edit | Fix fixtures or env pins | Replay is green |
| Replay green, code messy | Extract one helper | Replay still green |
| Model suggests a rewrite | Reject; request one helper | Diff stays under one function |
| New I/O appears in the diff | Revert the change | Golden paths unchanged |
A diff that violates a row is out of scope. Send that diff back for a smaller change.
How to read a mismatch
When replay fails, open golden_io.got.json first thing. Diff it against golden_io.json with a normal tool.
Classify the delta before touching any production code. Env pin changes mean the machine, not the function.
Path sha changes mean a fixture leaked into var. Result repr changes mean the edit was too large.
Revert the edit, shrink the diff, and replay again. Do not refresh the golden to hide the delta.
Commit shape
Use two commits after the first green replay. Commit one is the harness plus the golden file.
Commit two is the single structural production change. Never mix recorder edits with production code edits.
Reviewers should see a tiny, fenced second commit. That split is the whole point of the fence.
Fixture layout
Keep fixtures next to the driver, not in tmp. A tiny tree is enough for this characterization.
Avoid copying the entire customer dataset into git. Three directories cover empty, happy, and error paths.
fixtures/
empty/.keep
one_row/batch.csv
bad_ext/notes.txt
Each fixture directory should be documented in one line. That line belongs in the driver, not a wiki.
What the golden table does not prove
Characterization does not prove the function is correct. It proves the next edit did not change recorded behavior.
Wrong behavior stays wrong if the corpus missed it. Nondeterministic clocks will flake without an injected clock.
Freeze time at the wrapper if timestamps appear. Random IDs belong in the watched-path hash.
Binary files need hashes, not JSON string dumps. The proposed harness already hashes each watched path. Do not put raw bytes into result_repr.
Who should not use this approach
Do not use this method on greenfield modules. Write intent tests for new code instead.
Characterization is for code nobody fully trusts. Do not use this on cryptographic primitives either.
Golden ciphertexts can hide accidental cryptographic weakness. Those modules need known-answer tests from specs.
Do not use this when live network I/O is required. The harness does not stub sockets or DNS.
Add seams first, then characterize the local core. Skip this if the team cannot review JSON goldens.
Unreviewed goldens become a license to freeze bugs. Review is a required part of the method.
Limitations of the proposed harness
The harness uses repr and JSON fallbacks for values. Object identity will not round-trip through the table.
Pin field-level dicts when object graphs matter. Serialized traces can include machine-specific file paths.
Prefer exc_type over full traces in CI. The workflow assumes Python and a writable workspace.
Other languages still need an equivalent recording wrapper. The same decision table still applies across languages.
Free model access does not replace human review. Free server runs do not replace local CI. Both are optional accelerators after the fence exists.
Close
Characterize one tangled function before the first edit. Record a small golden I/O table and replay it. Change one surface only after that replay stays green.
If a free model and free server are available, generate extra fixture rows there. Keep the refactor itself small and local.
Top comments (0)