Write Characterization Tests Before the Smallest Messy-Repo Change

Messy repositories fail at the first untested extract. Characterization tests freeze current inputs and outputs first. Then you apply one smallest safe change only.

Skip that order and silent behavior drift wins. Reviewers cannot see the drift without a pin.

The failure this workflow targets

Untested modules hide side effects and implicit call order. A clean extract often changes return values silently. Exception types and stdout can move without a test.

You need a golden record of today's behavior. That record is a characterization suite, not a redesign. It asserts stability of outputs, not semantic correctness.

Do not start with architecture diagrams or helper names. Start with one public function and one fixture. Everything else waits until that pin is green.

What to pin before you touch code

Pin four surfaces for each public entry point. Capture inputs, outputs, exceptions, and filesystem side effects. Skip private helpers until a public path needs them.

Use this decision table before you write any test. One signal maps to one test, never two.

Signal	Pin it?	Why
Returns a dict or list	Yes	Shape is the caller contract
Writes stdout or logs	Yes	Operators parse those lines
Raises on bad input	Yes	Type and message often matter
Writes a file	Yes	Bytes and path can drift
Reads env vars	Yes	Missing keys change branches
Comment-only edit	No	No runtime surface
Unreachable private helper	No	No caller depends on it

If two signals fire, write two separate tests. Combined assertions hide which contract actually moved.

Inventory call sites with commands

Do not guess the public surface from file names. Grep imports, CLI parsers, and test files first. Record the exact callable and its argument shape.

rg -n "from reports import|import reports" -g "*.py"
rg -n "reports\." -g "*.py"
rg -n "add_parser|ArgumentParser" -g "*.py"

Write a short inventory file, not a wiki page. Each row is one callable, one fixture name, one risk note.

# call_inventory.txt
build_summary(rows)        golden_summary_basic.json    stdout+return
build_summary([])          golden_summary_empty.json    empty-list branch
write_report(path, data)   golden_report_sha256.txt     file bytes

That inventory is the scope for week one. It is not a backlog of refactors.

Numbered workflow

Follow these eight steps in strict sequential order. Do not skip the freeze before the first edit.

1. Inventory public call sites from tests, scripts, and CLIs.
2. Record one representative input fixture per public call site.
3. Capture return values, stdout, stderr, and exception types.
4. Hash any files the function creates or rewrites.
5. Commit the golden fixtures as checked-in JSON files.
6. Run the suite twice and confirm byte-for-byte determinism.
7. Apply one smallest change, then rerun the full suite.
8. Stop immediately if any golden hash or key moves.

A smallest change touches one function and one behavior. Valid examples include a pure helper extract or rename. Two extracts in one commit is already too large.

Artifact: a characterization harness

The following Python harness is a labeled local proposal. It is not a benchmark and has no published timings. Run it against one messy module you already own.

# characterize.py
"""Proposal: freeze return, streams, exceptions, and output hashes."""
from __future__ import annotations

import hashlib
import io
import json
from contextlib import redirect_stderr, redirect_stdout
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable

GOLDEN_DIR = Path("tests/goldens")


@dataclass(frozen=True)
class Trace:
    return_value: Any
    stdout: str
    stderr: str
    exc_type: str | None
    exc_msg: str | None
    file_sha256: dict[str, str]


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def capture(
    fn: Callable[..., Any],
    args: tuple[Any, ...],
    kwargs: dict[str, Any],
    output_files: list[Path],
) -> Trace:
    stdout_buf = io.StringIO()
    stderr_buf = io.StringIO()
    exc_type = None
    exc_msg = None
    return_value = None
    try:
        with redirect_stdout(stdout_buf), redirect_stderr(stderr_buf):
            return_value = fn(*args, **kwargs)
    except Exception as exc:  # proposal: broad catch for characterization only
        exc_type = type(exc).__name__
        exc_msg = str(exc)
    hashes = {}
    for path in output_files:
        if path.exists():
            hashes[str(path)] = sha256_file(path)
    return Trace(
        return_value=return_value,
        stdout=stdout_buf.getvalue(),
        stderr=stderr_buf.getvalue(),
        exc_type=exc_type,
        exc_msg=exc_msg,
        file_sha256=hashes,
    )


def dump_trace(trace: Trace) -> dict[str, Any]:
    return {
        "return_value": trace.return_value,
        "stdout": trace.stdout,
        "stderr": trace.stderr,
        "exc_type": trace.exc_type,
        "exc_msg": trace.exc_msg,
        "file_sha256": trace.file_sha256,
    }


def record(name: str, trace: Trace) -> Path:
    GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
    path = GOLDEN_DIR / f"{name}.json"
    payload = json.dumps(dump_trace(trace), indent=2, sort_keys=True) + "\n"
    path.write_text(payload)
    return path


def check(name: str, trace: Trace) -> None:
    path = GOLDEN_DIR / f"{name}.json"
    expected = json.loads(path.read_text())
    actual = dump_trace(trace)
    if actual != expected:
        raise AssertionError(
            f"{name} drifted\n"
            f"expected={json.dumps(expected, sort_keys=True)}\n"
            f"actual={json.dumps(actual, sort_keys=True)}"
        )

Return values must be JSON serializable for this harness. Convert decimals, sets, and datetime objects before dump. If conversion changes meaning, pick a different pin format.

Store goldens under tests/goldens named after the entry point. Commit both the harness and the JSON in one change. The second run must match the first run exactly.

Wire pytest as a read-mostly checker

Wire a tiny pytest module so CI fails on drift. Keep the test body boring on purpose here.

Gate recording behind RECORD_GOLDENS=1 so CI stays read-only. Local first runs may record; later runs must only check.

# tests/test_characterize_reports.py
"""Proposal: one test per inventory row, nothing more."""
import os
from pathlib import Path

from characterize import capture, check, record
from reports import build_summary, write_report

RECORD = os.environ.get("RECORD_GOLDENS") == "1"
ROWS = [
    {"sku": "A-1", "qty": 2, "price": "10.00"},
    {"sku": "B-9", "qty": 0, "price": "3.50"},
]


def test_build_summary_basic(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    trace = capture(build_summary, (ROWS,), {}, output_files=[])
    if RECORD:
        record("build_summary_basic", trace)
    check("build_summary_basic", trace)


def test_write_report_bytes(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    out = tmp_path / "report.json"
    data = {"total": 20.0, "skus": ["A-1"], "skipped": 1}
    trace = capture(write_report, (out, data), {}, output_files=[out])
    if RECORD:
        record("write_report_bytes", trace)
    check("write_report_bytes", trace)

RECORD_GOLDENS=1 python -m pytest tests/test_characterize_reports.py -q
python -m pytest tests/test_characterize_reports.py -q
git add characterize.py tests/goldens tests/test_characterize_reports.py
git commit -m "Pin build_summary and write_report goldens"

The second command must not set RECORD_GOLDENS. A green second run is the freeze. Only then is an extract allowed.

When a golden moves

Treat a mismatch as a product question, not a test bug. Either revert the edit or accept a new contract. Do not "fix" the assertion to stay green.

Use this triage table during the first mismatch.

Observation	Action
Only a helper name changed	Revert and extract with same bytecode path
stdout gained a debug line	Revert the print or split a later commit
SHA-256 moved, JSON keys same	Inspect file bytes before updating
Exception type changed	Revert; that is a public contract break
Empty input now returns None	Stop; callers may unpack a dict

Update a golden only with a written reason in the commit body. "Tests failed" is not an acceptable commit reason.

Non-determinism will lie to you

Time, random, and network calls will break golden files. Stub clocks and RNG before you freeze the first fixture. Leave network out of characterization until you fake it.

# proposal: freeze time and randomness before record()
import random
from datetime import datetime, timezone

def freeze_clocks(monkeypatch) -> None:
    monkeypatch.setattr(random, "shuffle", lambda seq: None)
    monkeypatch.setattr(
        "reports.utcnow",
        lambda: datetime(2026, 9, 14, tzinfo=timezone.utc),
    )

The date in that stub is a fixture, not a metric. Replace it with your module's actual clock seam. If you cannot find a seam, do not record yet.

Draft stubs after goldens, not before

A coding model can draft harness stubs from captured traces. It must not invent expected values you never recorded.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use those only to draft harness stubs after you pin goldens.

Paste the inventory file and one captured trace into the prompt. Ask for a pytest function that calls capture and check only. Reject any stub that hardcodes return values the trace does not contain.

You still review every fixture before the first refactor commit. The model does not own the golden files.

Smallest safe change catalog

Prefer these four edits after the suite is green. Extract a pure function with no new branches. Inline a one-use helper that confuses readers.

Rename a local that tests never mention. Delete a branch that goldens prove is unreachable. Stop after one of those four edits.

Reject these edits until goldens cover the path. Do not change exception types during the first pass. Do not reorder file writes that hashes already pin.

Do not "improve" stdout formatting under a golden pin. Formatting is behavior if operators grep logs. Save style passes for a later, explicit commit.

Worked proposal: one CSV summary module

Suppose reports.py builds a summary dict from CSV rows. You do not rewrite the parser on day one. You pin build_summary(rows) return JSON and log lines.

# reports.py — messy starting point, labeled example
import json
from pathlib import Path

def build_summary(rows):
    total = 0
    skipped = 0
    skus = []
    for row in rows:
        qty = int(row.get("qty") or 0)
        if qty <= 0:
            skipped += 1
            print(f"skip {row.get('sku')}")
            continue
        total += qty * float(row["price"])
        skus.append(row["sku"])
    print(f"skus={len(skus)} skipped={skipped}")
    return {"total": total, "skus": skus, "skipped": skipped}

def write_report(path, data):
    Path(path).write_text(json.dumps(data, indent=2, sort_keys=True) + "\n")

After goldens pass twice, extract _qty as a pure helper. Keep print statements and dict keys fully identical. Rerun the suite before any second extract.

def _qty(row: dict) -> int:
    return int(row.get("qty") or 0)

def build_summary(rows):
    total = 0
    skipped = 0
    skus = []
    for row in rows:
        qty = _qty(row)
        if qty <= 0:
            skipped += 1
            print(f"skip {row.get('sku')}")
            continue
        total += qty * float(row["price"])
        skus.append(row["sku"])
    print(f"skus={len(skus)} skipped={skipped}")
    return {"total": total, "skus": skus, "skipped": skipped}

That extract is the entire first refactor commit. Leave logging, JSON keys, and validation fully untouched. Add no extra validation branch in this commit.

Limitations

Characterization tests freeze bugs alongside the intended behavior. They will not tell you the output is correct. Flaky I/O and clocks make the suite a liability.

Large goldens rot when input corpora keep growing. Prefer one small fixture per behavior, not megabyte dumps. Binary fixtures without a hexdump will waste review time.

This workflow also slows pure greenfield work down. Spec tests are cheaper when no users exist yet. Do not characterize a function you can still delete.

Who should not use this approach

Do not use this on greenfield code with no users. Write real spec tests there, not characterization pins. Do not use this for security-sensitive parsers without review.

Golden files can embed secrets from production-like fixtures. Redact paths, tokens, and personal data before the pin commit. Teams without diff discipline should not automate extracts.

A model-drafted stub is not a reviewed contract yet. If nobody reads the golden JSON, skip the model. The pin only works when a human owns the mismatch.

Close

Pin behavior, then change one function, then stop. That order is the whole method, not a slogan. The next extract waits for another green run.