Characterize First: One Safe Edit in a Messy Repo

A messy refactor fails without locked current outputs. Characterization tests freeze those current outputs in place. Only then is one small change actually safe.

Skip the big rewrite on the first pass. Skip the clever rename storm as well. Capture what the module returns today instead.

The failure mode

Legacy modules hide rounding, aliases, and implicit defaults. Callers often depend on those hidden accidents. A clean extract can still change bytes.

Unit tests written after the edit miss that drift. They encode hoped behavior rather than current behavior. The regression then ships disguised as a cleanup.

Hoped tests also skip odd types at the boundary. Strings sneak in where integers were assumed. Gold files make those accidents visible early.

The core rule

Treat today's outputs as the working specification. Do not improve behavior in the first edit. Preserve output bytes, then change only structure.

This is not praise for bad logic. It is only a sequencing rule for edits. Correctness work comes after the gold lock.

Artifact: a gold-file characterization harness

The artifact is a small Python snapshot harness. It calls one public function across fixed cases. It writes JSON gold files you can diff.

Label the listing below as sample code. Adapt names to your real messy module. Keep secrets and live customer payloads out.

1. Isolate one messy entry point

Pick one function that already has real callers. Record its name, arity, and observed argument types. Ignore private helpers until the public lock holds.

# probe_quote.py — sample, not production telemetry
from legacy_quote import quote_total

CASES = [
    {"qty": 1, "unit_cents": 199, "coupon": None, "state": "CA"},
    {"qty": 3, "unit_cents": 199, "coupon": "SAVE10", "state": "CA"},
    {"qty": 0, "unit_cents": 500, "coupon": None, "state": "NY"},
    {"qty": 2, "unit_cents": 50, "coupon": "SAVE10", "state": "TX"},
    {"qty": 9, "unit_cents": 333, "coupon": "", "state": "ca"},
    {"qty": "4", "unit_cents": "100", "coupon": "save10", "state": "Ny"},
]

Six cases are a start, not coverage. Add shapes mined from logs after this lock. Prefer observed argument shapes over invented clean ones.

2. Snapshot outputs to gold files

Run the snapshot against current committed HEAD only. Do not snapshot a dirty worktree mix. The gold file must match what callers already run.

# snapshot.py — sample harness
import json
from pathlib import Path
from probe_quote import CASES
from legacy_quote import quote_total

GOLD = Path("gold")
GOLD.mkdir(exist_ok=True)


def run_case(case):
    value = quote_total(**case)
    return {
        "input": case,
        "output": value,
        "output_type": type(value).__name__,
    }


def main():
    rows = [run_case(c) for c in CASES]
    path = GOLD / "quote_total.json"
    text = json.dumps(rows, indent=2, sort_keys=True) + "\n"
    path.write_text(text)
    print(f"wrote {path} n={len(rows)}")


if __name__ == "__main__":
    main()

Commit the gold file in its own change. That commit is the behavior lock. Later diffs then show output movement clearly.

python snapshot.py
git add gold/quote_total.json snapshot.py probe_quote.py
git commit -m "lock quote_total characterization gold"

3. Replay gold on every later edit

The replay test is intentionally boring. It reloads JSON and calls the same function. Equality plus type name are the whole oracle.

# test_quote_characterization.py — sample
import json
from pathlib import Path
from legacy_quote import quote_total


def test_quote_total_matches_gold():
    rows = json.loads(Path("gold/quote_total.json").read_text())
    assert rows, "gold file must not be empty"
    for row in rows:
        actual = quote_total(**row["input"])
        assert actual == row["output"]
        assert type(actual).__name__ == row["output_type"]

python -m pytest test_quote_characterization.py -q

A failing assertion means behavior moved. Structure changes must keep this test green. Type changes also count as behavior here.

Sample messy module under test

The module below is intentionally awkward. Mixed types and silent defaults live here. Use it only as a teaching stand-in.

# legacy_quote.py — messy on purpose
def quote_total(qty, unit_cents, coupon=None, state="CA"):
    q = int(qty)
    cents = int(unit_cents)
    raw = q * cents
    code = (coupon or "").strip().upper()
    if code == "SAVE10":
        raw = int(raw * 0.9)
    st = str(state).upper()
    if st == "CA":
        raw = int(raw * 1.0725)
    elif st == "NY":
        raw = int(raw * 1.08)
    if raw < 0:
        raw = 0
    return raw

Note the float multiply and truncating int() calls. Callers may depend on that truncation today. A Decimal rewrite would change some stored totals.

Empty coupon strings already differ from missing coupons. Case folding on state is also load-bearing. Gold files catch those details without a narrative spec.

Grow the case list from traces

Static cases miss the ugly production shapes. Add a log miner after the first lock. Keep the miner read-only and sampled.

# mine_cases.py — sample, synthetic logs only
import ast
import json
from pathlib import Path


def parse_line(line):
    # expected: quote_total kwargs as a Python dict literal
    return ast.literal_eval(line.strip())


def main():
    seen = []
    for line in Path("synthetic_quote.log").read_text().splitlines():
        if not line.strip():
            continue
        case = parse_line(line)
        if case not in seen:
            seen.append(case)
    Path("mined_cases.json").write_text(json.dumps(seen, indent=2) + "\n")


if __name__ == "__main__":
    main()

Re-snapshot after merging mined rows into CASES. Do not edit gold by hand after that merge. Hand-edited gold files hide accidental churn.

Cap the miner on unique argument tuples. Huge gold files slow reviews and hide signal. A few dozen diverse rows beat thousands of clones.

Numbered workflow

Follow this order on every messy entry point. Do not start at the extract. The lock is the first deliverable.

1. List every caller of the chosen messy function.
2. Collect live argument shapes from logs and tests.
3. Snapshot gold JSON from current committed code.
4. Commit those gold files before any refactor starts.
5. Wire a replay test into the default CI job.
6. Perform one structural edit with no behavior pitch.
7. Re-run the replay test on the same cases.
8. Halt the change if gold output drifts unexpectedly.

Keep each change reviewable in one diff. Do not mix a bugfix with an extract. Bugfixes update gold in a separate commit.

Smallest safe change

Safe means gold stays byte-for-byte identical. Extract a local helper and stop. Rename an internal variable and stop.

Here is one safe extract against the sample module. Public results must still match the gold file. Coupon parsing moves; tax branches stay put.

def _apply_coupon(raw, coupon):
    code = (coupon or "").strip().upper()
    if code == "SAVE10":
        return int(raw * 0.9)
    return raw


def quote_total(qty, unit_cents, coupon=None, state="CA"):
    q = int(qty)
    cents = int(unit_cents)
    raw = _apply_coupon(q * cents, coupon)
    st = str(state).upper()
    if st == "CA":
        raw = int(raw * 1.0725)
    elif st == "NY":
        raw = int(raw * 1.08)
    if raw < 0:
        raw = 0
    return raw

Re-run pytest after this extract lands. If gold matches, the extract is finished. Do not fix California rounding in that same patch.

If gold fails, revert the extract first. Debug the helper in isolation after revert. Never “while we are here” on failing gold.

Decision table

Use the table as a merge gate. Structure work and behavior work stay split. Gold updates always need an explicit reason.

Proposed edit	Gold first	Same commit as extract	Verdict
Extract _apply_coupon	Required	No extra edits	Safe if gold holds
Rename raw to subtotal	Required	Local only	Safe if gold holds
Replace int() with round()	Required	Never with extract	Behavior change
Switch totals to Decimal	Required	Never with extract	Behavior change
Add a new coupon code	Required	Own commit	Intentional delta
Delete the qty==0 path	Required	Never silent	Likely break

Read each row before opening the editor. If the verdict is behavior change, stop. Write a new gold commit with the intended delta.

Intentional deltas still need a before snapshot. The before snapshot proves the old contract. The after snapshot documents the new contract.

Optional model help, with a hard gate

A free coding model can draft extra cases from traces. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.

Feed the model the function signature and gold JSON. Ask only for additional input rows, not new logic. Run every proposed row through quote_total on current code.

Keep rows that match a fresh snapshot. Drop rows the live function cannot accept. Drop rows that describe desired future math.

The free server can execute snapshot.py and pytest. Do not upload customer quotes or keys. Gold files should contain synthetic inputs only.

The model is a case generator, not a referee. The gold file remains the only referee. Human review remains the last merge gate.

What this workflow does not claim

It does not prove functional correctness. It proves stability against a recorded sample. Coverage still depends on the case list quality.

It does not replace partner contract tests. It does not replace property tests for parsers. It is a lock on one messy entry point.

Float tax math remains a later problem. Characterization makes that later change visible. Visibility is the point of the gold file.

Who should not use this approach

Do not use this as a substitute for design. Greenfield modules need real specs, not gold accidents. Do not freeze known security bugs as gold.

Do not snapshot functions that hit production networks. Do not store PII inside JSON fixtures. Do not run untrusted model output without pytest.

Skip this when the function already has tight tests. Skip this when you cannot execute the module. Skip this when the public result is nondeterministic.

Nondeterministic code needs clocks and seeds first. Freeze time before you freeze outputs. Randomness without seeds makes gold files lie.

Limits of the sample harness

The harness compares Python equality only. It will miss log side effects. It will miss file writes and extra queries.

Extend the snapshot if those channels matter. Capture stdout, SQL, and filesystem diffs next. Still change one concern per commit after that.

JSON also struggles with NaN and raw bytes. Prefer explicit output_type as shown above. Add a hex encoding step when bytes appear.

CI should run the replay test on every pull request. A sample job is a single pytest node. Do not hide gold failures behind allowlists.

# sample CI snippet, unexecuted
steps:
  - run: python -m pytest test_quote_characterization.py -q

If the job is red, the extract is not done. Comments about “equivalent refactor” do not override gold. Bytes are the review language here.

Closing sequence

Lock outputs. Extract one helper. Re-run gold. That loop is the whole method.

Keep gold files in version control. Review every model-drafted extra case. Ship one behavior-preserving edit at a time.