Record Golden I/O Before the First Messy Edit

Never edit a messy module before behavior is pinned. Characterization tests record current outputs, including defects. A later one-line change then has a fail-fast check.

The failure this method targets

Messy modules encode contracts in call order and globals. A cleanup often changes those contracts without errors. Tests written after cleanup cannot recover the lost baseline.

Silent output drift is the usual production symptom. Status codes stay green while downstream payloads shift. Parsers then fail in a later integration job.

What to pin, and what to ignore

Pin inputs, outputs, and cache side effects only. Do not pin traceback text or wall-clock timestamps. Do not rewrite the module during the capture run.

Skip network, filesystem, and random sources during RECORD. Stub those edges with fixed return values instead. Otherwise the golden file will thrash on every run.

Artifact: a RECORD/CHECK harness

The listing below is an illustrative fixture, not production. It mutates a process-wide cache and stringifies money. Those two facts are the hidden contract under test.

# messy_report.py
# Illustrative fixture. Not production code.

CACHE = {}


def build_report(rows, *, currency="USD"):
    if not rows:
        return {
            "ok": False,
            "items": [],
            "total": "0.00",
            "cached": len(CACHE),
        }

    items = []
    total = 0.0
    for row in rows:
        qty = row.get("qty") or 1
        price = row["price"]
        line = round(qty * price, 2)
        total += line
        key = row.get("sku") or row.get("id")
        CACHE[key] = line
        items.append(
            {
                "key": key,
                "line": f"{line:.2f}",
                "currency": currency,
            }
        )

    return {
        "ok": True,
        "items": items,
        "total": f"{total:.2f}",
        "cached": len(CACHE),
    }

The harness writes or checks a golden JSON file. RECORD mode captures the live return value. CHECK mode compares a new call against that file.

Round-trip JSON will coerce tuples into lists. Keep the function returning lists and dicts only. Otherwise CHECK fails from serialization, not behavior.

A third MUTATE mode tampers with one field on purpose. That mode must fail, or the harness is theater.

# test_characterize_report.py
# Proposal: local RECORD/CHECK/MUTATE loop. Run on your checkout.

from __future__ import annotations

import json
import os
from copy import deepcopy
from pathlib import Path

import messy_report
from messy_report import build_report

GOLDEN = Path(__file__).parent / "goldens" / "build_report.json"
MODE = os.environ.get("CHARACTERIZE_MODE", "check")

FIXTURES = [
    [],
    [
        {"sku": "A-1", "qty": 2, "price": 1.255},
        {"id": "B-9", "price": 10},
    ],
    [
        {"sku": "A-1", "qty": 1, "price": 3},
    ],
]


def _run_cases():
    messy_report.CACHE.clear()
    observed = []
    for rows in FIXTURES:
        observed.append(
            {
                "input": rows,
                "output": build_report(deepcopy(rows)),
                "cache_after": dict(messy_report.CACHE),
            }
        )
    return observed


def test_build_report_characterization():
    GOLDEN.parent.mkdir(parents=True, exist_ok=True)
    live = _run_cases()

    if MODE == "record":
        GOLDEN.write_text(json.dumps(live, indent=2, sort_keys=True) + "\n")
        assert GOLDEN.exists()
        return

    assert GOLDEN.exists(), "missing golden; run CHARACTERIZE_MODE=record once"
    expected = json.loads(GOLDEN.read_text())

    if MODE == "mutate":
        expected[1]["output"]["total"] = "999.99"

    assert live == expected

RECORD must run on a quiet local checkout only. Do not set CHARACTERIZE_MODE=record inside any CI job. CI should run CHECK against files already in git.

The commands below assume a POSIX shell. Copy the env prefix onto each pytest invocation.

python -m pip install pytest
mkdir -p goldens
CHARACTERIZE_MODE=record python -m pytest test_characterize_report.py -q
CHARACTERIZE_MODE=check python -m pytest test_characterize_report.py -q
CHARACTERIZE_MODE=mutate python -m pytest test_characterize_report.py -q

The mutate command must print a pytest assertion failure. A green mutate run means the comparison is a no-op. Stop there and fix the harness before any extract.

Decision table for the first edit

Use this table after goldens exist and CHECK passes. Rows list common edits after a messy-module extract.

Proposed edit	CHECK should	Next action
Extract _line_total with the same round	pass	keep the extract only
Stop writing CACHE	fail	revert, or RECORD with stated intent
Treat missing qty as 0	fail	treat as a behavior change
Store total as Decimal	fail	RECORD plus caller updates
Rename output key to sku	fail	do not mix with the extract

Extract a helper that preserves call results and cache writes. That is the only change this article treats as safe.

Any intentional behavior change needs a new RECORD pass. Name that RECORD pass in the pull request body.

Workflow

1. Isolate one entry point

Pick one function that callers already treat as a boundary. Do not start inside a 400-line nested helper. Write the name and its callers in the PR body.

2. Freeze fixtures on disk

Store input rows in the test as literal data. Keep the fixture small enough to read in review. Empty input, default qty, and a cache hit suffice.

3. Record goldens

Run RECORD once and open the JSON file. Confirm keys, string totals, and cached counts by eye. Commit goldens only after that manual read.

4. Prove the harness can fail

Run MUTATE and confirm pytest reports a mismatch. If MUTATE stays green, the harness is not testing. Fix the assertions before any refactor attempt starts.

5. Make one behavior-preserving edit

Extract a local helper for line-total rounding only. Leave CACHE writes in the original function body. Do not rename public keys in the same patch.

# smallest safe change; proposal, apply only after CHECK is green

def _line_total(qty, price):
    return round(qty * price, 2)


def build_report(rows, *, currency="USD"):
    if not rows:
        return {
            "ok": False,
            "items": [],
            "total": "0.00",
            "cached": len(CACHE),
        }

    items = []
    total = 0.0
    for row in rows:
        qty = row.get("qty") or 1
        price = row["price"]
        line = _line_total(qty, price)
        total += line
        key = row.get("sku") or row.get("id")
        CACHE[key] = line
        items.append(
            {
                "key": key,
                "line": f"{line:.2f}",
                "currency": currency,
            }
        )

    return {
        "ok": True,
        "items": items,
        "total": f"{total:.2f}",
        "cached": len(CACHE),
    }

6. Re-run CHECK

Run CHECK and require a clean pytest result. If CHECK fails, revert the edit without debate. The golden file wins until you plan a behavior change.

Drafting tests without trusting generated code

A free coding model can draft the harness from fixtures. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.

Use that option only to draft tests from frozen fixtures. Copy the draft into your repository by hand. Run RECORD and MUTATE on your machine before trusting it.

Generated assertions often miss global cache side effects. They also skip empty-input branches and exception types. Treat every generated test as a proposal until MUTATE fails.

Limitations

Characterization locks current bugs into the golden file. That is useful for refactors and harmful for repairs. Split bug-fix work from extract work in separate PRs.

Golden files do not replace type checks or fuzzing. They also go stale when callers change input shapes. Re-RECORD only with an explicit behavior-change note.

This method is weak on concurrency and time. Shared CACHE already makes order-dependent results likely. Pin a single-threaded fixture or you will chase flakes.

Who should skip this method

Skip this method on greenfield modules with no callers. Skip it when the public contract is being redesigned. Skip it for cryptography, authz, or privacy-critical paths.

Security behavior needs intent tests, not output snapshots. Snapshots can bless a leak if the leak already exists. Write explicit forbidden-output checks for those domains instead.

What reviewers should demand

The smallest safe change is the one CHECK still accepts. Attach the golden JSON file to the pull request. Reviewers should read that file before the diff.