Do not refactor a messy module until tests pin current behavior. Characterization tests freeze observable outputs before any structural change.
Naive cleanup often changes results that nobody ever documented. A tangled file encodes edge cases inside accidental control flow.
This workflow records those edges before the first edit. Then you apply one mechanical change and re-run the harness.
What the tests must pin
Pin inputs, outputs, and error types, not internal names. Call the public entry points that production already uses.
Record returned structures, stdout bytes, and raised exception classes. Skip private helpers until a later extract has dedicated pins.
Treat current bugs as frozen contracts without an explicit ticket. A refactor that silently fixes rounding is still a behavior change.
Why one change beats a rewrite
One change means one intent, not a weekend rewrite. Rename one symbol, extract one function, or delete one dead branch.
Do not mix formatting, typing, and logic in the same commit. Each commit must show a characterization delta of zero.
Large diffs hide the line that altered a total. Reviewers cannot verify behavior when structure and values move together.
Numbered workflow
1. Isolate one messy entry point
Pick a function that existing callers already import. Avoid starting with a package-wide rename or folder move.
Write the call signature beside the new test path. Keep fixture names stable so later diffs stay small.
2. Capture a golden corpus
Drive the function with fixtures from logs, bugs, and support tickets. Serialize results to JSON with sorted keys and integer money.
Store one JSON file per case under tests/goldens. Commit those files before any production code changes.
3. Assert replay equality
Load each golden file and call the same entry point. Compare with an exact matcher, not a fuzzy snapshot helper.
Fail the run on the first mismatch and print both payloads. That print is the characterization debugger you will use.
4. Make one mechanical edit
Extract a local helper or rename a confusing identifier. Do not change branches, defaults, or numeric literals yet.
Re-run the harness until every golden file still matches. If a file drifts, revert the edit and shrink the change.
5. Only then open a behavior ticket
New behavior needs a new test, not a silent golden rewrite. Update goldens only when a ticket names the old output as wrong.
Normalize before you pin
Raw dumps often include clocks, ids, and unordered containers. Those fields create false diffs that block safe extracts.
Strip or freeze time in the recorder wrapper only. Sort lists that the public contract treats as sets.
Do not sort lists when order is part of the contract. Document each normalization rule next to the recorder.
Artifact: replay harness and recorder
The Python below is a labeled, unexecuted proposal. Adapt the module path and entry name to your tree.
# characterization_harness.py
# Proposal: freeze public compute_totals() output before any extract.
from __future__ import annotations
import json
import importlib
from pathlib import Path
GOLDEN_DIR = Path("tests/goldens/order_totals")
MODULE_UNDER_TEST = "billing.order_totals"
ENTRY = "compute_totals"
def load_cases():
cases = []
for path in sorted(GOLDEN_DIR.glob("*.json")):
payload = json.loads(path.read_text(encoding="utf-8"))
cases.append((path.name, payload))
return cases
def call_entry(inputs: dict):
mod = importlib.import_module(MODULE_UNDER_TEST)
fn = getattr(mod, ENTRY)
try:
result = fn(**inputs)
return {
"ok": True,
"result": result,
"error_type": None,
"error_msg": None,
}
except Exception as exc:
return {
"ok": False,
"result": None,
"error_type": type(exc).__name__,
"error_msg": str(exc),
}
def replay():
failures = []
for name, payload in load_cases():
observed = call_entry(payload["inputs"])
expected = payload["output"]
if observed != expected:
failures.append((name, expected, observed))
return failures
if __name__ == "__main__":
failed = replay()
if not failed:
print("characterization: 0 diffs")
raise SystemExit(0)
for name, expected, observed in failed:
print(f"DIFF {name}")
print("expected", json.dumps(expected, sort_keys=True))
print("observed", json.dumps(observed, sort_keys=True))
raise SystemExit(1)
Record goldens with a sibling script that only writes files. Never overwrite golden files during a replay run.
# record_goldens.py
# Proposal: one-time capture. Review each file before commit.
import json
from characterization_harness import GOLDEN_DIR, call_entry
CASES = [
("empty_cart", {"items": [], "region": "US", "coupon": None}),
(
"single_taxed",
{
"items": [{"sku": "A", "qty": 1, "cents": 1999}],
"region": "US",
"coupon": None,
},
),
(
"coupon_stack",
{
"items": [{"sku": "A", "qty": 2, "cents": 500}],
"region": "EU",
"coupon": "SAVE10",
},
),
(
"unknown_region",
{
"items": [{"sku": "B", "qty": 1, "cents": 100}],
"region": "XX",
"coupon": None,
},
),
]
def main():
GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
for name, inputs in CASES:
output = call_entry(inputs)
path = GOLDEN_DIR / f"{name}.json"
body = json.dumps(
{"inputs": inputs, "output": output},
indent=2,
sort_keys=True,
)
path.write_text(body, encoding="utf-8")
print("wrote", path)
if __name__ == "__main__":
main()
This example shows shape only, not a measured tax table. Your live module fills result during the record run.
{
"inputs": {
"coupon": null,
"items": [{"cents": 1999, "qty": 1, "sku": "A"}],
"region": "US"
},
"output": {
"error_msg": null,
"error_type": null,
"ok": true,
"result": {
"subtotal_cents": 1999,
"tax_cents": 0,
"total_cents": 1999
}
}
}
Inspect every JSON file after the record script finishes. Commit goldens and the harness in the same change.
python record_goldens.py
python characterization_harness.py
git add tests/goldens/order_totals characterization_harness.py record_goldens.py
git commit -m "Pin order_totals outputs before any extract"
Optional pytest wrapper keeps the same exact compare.
# tests/test_order_totals_characterization.py
# Proposal: pytest entry that fails on any golden drift.
from characterization_harness import replay
def test_order_totals_goldens_match():
failed = replay()
assert failed == [], [name for name, _, _ in failed]
python -m pytest tests/test_order_totals_characterization.py -q
Decision table for the first cut
Use exactly one row from the table below. Leave every other row for a later, separate commit.
| Signal in the messy file | First allowed change | Forbidden in that commit |
|---|---|---|
| Mixed names for one concept | Rename the local aliases | New branches or default values |
| Long function with one caller | Extract a private helper | Public API shape change |
| Commented-out block | Delete the dead block | Nearby live logic edits |
| Magic number used twice | Extract a named constant | Recalculate the number |
| Duplicate dump calls | Extract one serializer | Encoding or key-order change |
The harness must stay green after the chosen row. A red run means the edit was not mechanical.
Using a free model only for fixture drafts
A model can propose extra fixtures from logs and stack traces. It must not become the oracle for correct totals.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option. Use that pair to draft candidate cases on a throwaway clone.
Paste a redacted log slice and the public function signature. Ask for input dictionaries that hit missing branches, not rewrites.
Review every generated case against real production traces. Drop any fixture you cannot explain in one plain sentence.
Humans commit the golden files after manual review. The harness must reject any later silent drift.
A free server helps when the messy repo will not boot locally. Clone, install, record goldens, then discard the instance.
Do not treat generated tests as coverage proof. They remain drafts until replay matches committed files.
Limitations
This method does not prove the module is correct. It only proves the next edit did not change recorded outputs.
Golden files rot when upstream data formats change. Re-record through a reviewed, ticketed process, never by habit.
Exact JSON compare fails on unordered sets and timestamps. Normalize those fields in the recorder, not in production code.
The harness will not catch unused private helpers. Those need a later extract with their own pins.
Floating-point money will flake if you serialize raw floats. Store integer cents, as the example fixtures do.
Seed-dependent randomness will also break an exact replay. Inject clocks and rng objects before you record anything.
Who should skip this approach
Do not use this on a greenfield module with no callers. Write intent tests instead of freezing accidental behavior.
Do not use this as a substitute for a security review. Characterization will happily freeze a leak or an auth hole.
Do not freeze personal data inside golden files. Redact, hash, or synthesize those fields before recording.
Skip this if you cannot run the module twice deterministically. Non-deterministic I/O needs seams before any golden capture.
Skip this if product owners already rejected the current outputs. Fix the behavior under a ticket, then pin the new contract.
Close
Pin the public outputs before the first structural edit. Change one mechanical thing and re-run the harness.
The run must return zero diffs before you continue. That sequence is the entire method for messy modules.
Top comments (0)