Record Golden Outputs Before One Messy-Repo Change

Do not open a messy repository with a rewrite. Freeze current outputs with characterization tests first. Then change one seam and re-run those pins.

Characterization tests do not prove the module is correct. They prove the next edit did not drift behavior. That gap matters more than a clean file tree.

Why naive extracts fail

Messy functions mix I/O, mutation, and string layout. Callers often depend on spacing and exception text. Those contracts almost never appear in nearby comments.

A reviewer can bless a smaller helper that still breaks. Stdout order, dict mutation, and rounding all leak. The diff looks tidy while production reads different bytes.

Locale, newline mode, and float formatting also hide in the stream. A helper extract can change any of those silently. Tests that only check a returned total will stay green.

What to freeze

Record four observation classes before the first edit.

1. Return values and raised exception types.
2. Stdout, stderr, and exact log line text.
3. Mutated inputs plus any module-level tables.
4. Written files, including encoding and trailing newlines.

Internal local names are not a customer contract. Do not freeze wall-clock values unless the API returns them. Do not freeze set iteration order unless callers sort later.

Prefer byte hashes over pretty JSON pretty-printers. Pretty printers reorder keys and inject spaces. Hashes fail when one newline or rounding digit changes.

Workflow

Follow this sequence. Do not skip a recording step.

1. Isolate one entry point

Pick one public function, CLI, or HTTP handler. Do not characterize an entire package on day one. Write that call surface on a short checklist.

Name the process cwd, env keys, and argument vector. Those three inputs change path and format output. Leave them pinned even if the function looks pure.

2. Capture a real fixture set

Collect three to seven inputs that resemble production. Include one empty case, one typical case, one ugly case. Store fixtures as files, not as chat transcripts.

Anonymize account ids before the files land in git. Keep currency strings and quantity edges intact. Synthetic invoices encode fiction if production never sent them.

3. Record goldens from the current tree

Run the entry point under a harness. Hash stdout, return payloads, and side-effect files. Commit those goldens before touching production code.

Use a frozen working directory for every recording run. Relative paths inside the renderer will otherwise drift. Capture sys.stdout.buffer bytes, not decoded text guesses.

4. Draft tests, then delete guesses

A coding model can propose loaders and assertion names. Delete every assertion the recorded run does not support. Keep checks that fail when one output byte flips.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Use those to draft the harness and execute it, not to invent goldens.

Generated asserts often lock private locals and comment phrasing. Those are not the contract. The recorded byte stream is the contract.

5. Change one seam

Extract one helper or rename one binding. Do not fix bugs and extract in one commit. Re-run the golden suite after that single change.

If two seams look equally safe, take the smaller diff. Smaller diffs make a red golden easier to attribute. Attribution is the whole point of this net.

6. Stop if goldens move

A moved golden is a product decision. Revert the seam or document the byte-level delta. Do not refresh snapshots because the new diff looks cleaner.

Artifact: messy invoice renderer

The module below is compact and intentionally unsafe. It mutates caller dicts and writes aligned lines. Tax policy sits inside the renderer, not a policy module.

# invoice_legacy.py
from __future__ import annotations

import sys
from typing import Any, TextIO

TAX = 0.0875


def render_invoice(items: list[dict[str, Any]], out: TextIO | None = None) -> dict[str, float]:
    stream = out or sys.stdout
    total = 0.0
    lines: list[str] = []
    for item in items:
        name = str(item.get("name", "")).strip() or "item"
        qty = int(item.get("qty") or 1)
        price = float(item.get("price") or 0)
        if qty < 0:
            raise ValueError("qty")
        line = price * qty
        total += line
        item["_line"] = round(line, 2)
        lines.append(f"{name:12s} {qty:3d} x {price:7.2f} = {line:8.2f}")
    tax = round(total * TAX, 2)
    due = round(total + tax, 2)
    stream.write("INVOICE\n")
    for line in lines:
        stream.write(line + "\n")
    stream.write(f"TAX {tax:.2f}\n")
    stream.write(f"DUE {due:.2f}\n")
    return {"total": round(total, 2), "tax": tax, "due": due}

That function is one seam with four hidden contracts. Mutation of _line is a contract. Trailing newlines and column padding are contracts too.

Record goldens as files

Keep goldens next to fixtures. Do not paste expected strings into chat. The recorder below writes hashes and raw stdout.

# record_goldens.py
from __future__ import annotations

import hashlib
import io
import json
from pathlib import Path

from invoice_legacy import render_invoice

ROOT = Path(__file__).parent
FIX = ROOT / "fixtures"
GOLD = ROOT / "goldens"


def _hash(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def record(name: str) -> None:
    payload = json.loads((FIX / f"{name}.json").read_text(encoding="utf-8"))
    items = payload["items"]
    buf = io.StringIO()
    result = render_invoice(items, out=buf)
    stdout = buf.getvalue().encode("utf-8")
    body = {
        "stdout_sha256": _hash(stdout),
        "stdout": stdout.decode("utf-8"),
        "result": result,
        "mutated_lines": [item.get("_line") for item in items],
    }
    GOLD.mkdir(exist_ok=True)
    (GOLD / f"{name}.json").write_text(
        json.dumps(body, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )
    (GOLD / f"{name}.out").write_bytes(stdout)


if __name__ == "__main__":
    for fixture in sorted(FIX.glob("*.json")):
        record(fixture.stem)
        print(fixture.stem)

Sample fixture fixtures/typical.json stays small and ugly on purpose.

{
  "items": [
    {"name": "bolt", "qty": 10, "price": 0.4},
    {"name": "  washer", "qty": 2, "price": 1.25},
    {"name": "", "qty": null, "price": 3}
  ]
}

Add fixtures/empty.json with "items": []. Add fixtures/ugly.json with a negative qty expected to raise. Keep the raise path in a separate test, not a stdout hash.

Characterization tests

The tests reload goldens and compare three pins. Stdout bytes come first. Return dicts and mutation lists follow.

# test_invoice_characterization.py
from __future__ import annotations

import io
import json
from pathlib import Path

import pytest

from invoice_legacy import render_invoice

ROOT = Path(__file__).parent
FIX = ROOT / "fixtures"
GOLD = ROOT / "goldens"


def _cases() -> list[str]:
    return sorted(p.stem for p in FIX.glob("*.json") if p.stem != "ugly")


@pytest.mark.parametrize("name", _cases())
def test_stdout_and_result_match_golden(name: str) -> None:
    items = json.loads((FIX / f"{name}.json").read_text(encoding="utf-8"))["items"]
    golden = json.loads((GOLD / f"{name}.json").read_text(encoding="utf-8"))
    buf = io.StringIO()
    result = render_invoice(items, out=buf)
    assert buf.getvalue() == golden["stdout"]
    assert result == golden["result"]
    assert [item.get("_line") for item in items] == golden["mutated_lines"]


def test_negative_qty_still_raises_value_error() -> None:
    items = json.loads((FIX / "ugly.json").read_text(encoding="utf-8"))["items"]
    with pytest.raises(ValueError, match="qty"):
        render_invoice(items, out=io.StringIO())

Commands stay short and repeatable. Record once on the unclean tree. Then run pytest after every seam.

python -m pip install pytest
python record_goldens.py
python -m pytest test_invoice_characterization.py -q

Commit goldens/*.json and goldens/*.out in the same change. Reviewers then see the pinned bytes, not a promise. Later extracts cannot hide a column shift.

Mutation probe

A green suite with no probe is theater. Flip one format byte on a throwaway branch. The typical case must fail before you trust the pin.

# probe_expect_red.py  — label: unexecuted check on a dirty branch
# Change one format string, then run pytest.
# lines.append(f"{name:12s} {qty:3d} x {price:7.2f} = {line:8.2f}")
# to
# lines.append(f"{name:12s} {qty:3d}*{price:7.2f} = {line:8.2f}")

If pytest stays green after that probe, the harness is hashing the wrong object. Fix the harness before any extract. Do not extract on an unprobed net.

Decision table

Use this table when a teammate wants a bigger rewrite.

Observed symptom	Freeze this first	Do not change yet
Column drift in stdout	Full stdout bytes	Helper extract
Totals match, files differ	File bytes plus encoding	Rounding rewrite
Caller dict gains keys	Mutated key set and values	Pure-function conversion
Exception type changed	Exact exception class	Message rewording
Tests fail only on Windows	Newline mode and cwd	Cross-platform cleanup
Model-written asserts fail locally	Delete unsupported asserts	More generated tests

Read the table left to right. The middle column is the pin. The right column is the delayed cleanup.

Smallest safe change

Extract line formatting only. Leave tax math and mutation in place. Goldens must stay byte-identical after the move.

def _format_line(name: str, qty: int, price: float, line: float) -> str:
    return f"{name:12s} {qty:3d} x {price:7.2f} = {line:8.2f}"

Replace the inline f"{name:12s}..." with _format_line(...). Run pytest. If goldens move, the extract is not a rename of a format string. Stop and inspect padding, not tax policy.

Do not move item["_line"] = round(line, 2) in that commit. Mutation order is a separate seam. Mixing seams makes a red golden unreadable.

Limitations

Characterization freezes bugs alongside features. It will not mark which output is wrong. Large fixture sets also encode one machine locale.

Generated tests overfit comments and private locals. They underfit encodings and trailing spaces. Humans still delete unsupported assertions after each draft.

A remote run does not replace production traces. If fixtures are invented, goldens encode fiction. Prefer anonymized production payloads over handmade invoices.

Hashes also hide intent. A failed sha256 does not explain the column that moved. Keep the raw .out files beside the hashes for diffing.

Who should not use this

Do not use this flow for greenfield modules. Do not use it when a written spec already pins bytes. Do not use it as a substitute for load tests.

Skip it when the change must alter bytes on purpose. Skip it when secrets live inside captured fixtures. Skip it when the entry point is not deterministic.

Skip it for true random, wall-clock, and network-timed handlers. Those need injected clocks before goldens mean anything. Characterization without injection records noise.

Close

Messy refactors fail from unpinned bytes, not ugly names. Record goldens from one entry point. Change one seam. Keep the suite red when a byte moves.