Pin One Seam With Golden Files, Then Change One File

A messy repo is not a rewrite candidate. Characterization tests freeze the behavior you already ship. Then you change exactly one observed seam today.

A full-repo rewrite hides regressions inside review noise. Unreviewed AI diffs often compound that same noise. The safer work unit remains one observed boundary.

The failure mode

Messy modules share globals, files, and clock time. Unit tests that mock all three prove almost nothing. Those tests lock in the mock, not production.

A characterization test records the real output today. Tomorrow's change must match that recorded output exactly. The byte mismatches count as regressions, not style debates.

What this workflow is

This write-up is a proposed local method. It is not a measured production case study.

You inventory the seams, then capture golden output. You freeze every file except one target. You re-run the goldens after that tiny change.

Step 1: Inventory seams, not files

A seam is an entry you can invoke. Valid seams include CLI, HTTP, worker, or batch script. Files without an entry are not first-class targets.

Run a cheap inventory from the repo root. Treat the following script as a proposal only.

#!/usr/bin/env python3
"""Proposed seam inventory. Not a production scanner."""
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

ENTRY_HINTS = {"main", "handler", "run", "cli", "worker", "app"}


def file_facts(path: Path) -> dict:
    text = path.read_text(encoding="utf-8", errors="replace")
    tree = ast.parse(text)
    names = [n.name for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)]
    import_nodes = [
        n for n in ast.walk(tree) if isinstance(n, (ast.Import, ast.ImportFrom))
    ]
    return {
        "path": str(path),
        "functions": names,
        "entry_like": [
            n
            for n in names
            if n.lower() in ENTRY_HINTS or n.lower().startswith("handle")
        ],
        "import_count": len(import_nodes),
        "reads_env": ("os.environ" in text) or ("getenv" in text),
        "lines": text.count("\n") + 1,
    }


def main() -> None:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    skip = {".venv", "venv", "node_modules", "tests", "__pycache__"}
    rows = []
    for path in root.rglob("*.py"):
        if any(part in skip for part in path.parts):
            continue
        try:
            rows.append(file_facts(path))
        except SyntaxError:
            continue
    rows.sort(
        key=lambda r: (len(r["entry_like"]), r["import_count"]),
        reverse=True,
    )
    print(json.dumps(rows[:25], indent=2))


if __name__ == "__main__":
    main()

Requires Python 3.9+ for ast.unparse in the inventory script. Older runtimes can print path and line counts only.

python tools/inventory_seams.py src > seam_inventory.json

Read the JSON and pick one entry-like path. Ignore every other file during this pass.

Prefer a leaf with an obvious command. High import fan-in is a later problem. The first pin should be cheap to rerun.

Step 2: Freeze clock, cwd, and env

Golden files will flake without a frozen environment. Pin timezone, working directory, and any seed. Skip network pins unless the seam is offline.

Store the fixture input beside the golden files. Future runs must not depend on a laptop cwd. Copy input.json into tmp_path inside the test.

# tests/conftest.py — proposed fixture, not a measured harness
import pytest


@pytest.fixture
def frozen_env(tmp_path, monkeypatch):
    monkeypatch.setenv("TZ", "UTC")
    monkeypatch.setenv("APP_ENV", "characterize")
    monkeypatch.chdir(tmp_path)
    (tmp_path / "input.json").write_text(
        '{"order_id": "o-100", "qty": 2}\n',
        encoding="utf-8",
    )
    return tmp_path

Step 3: Capture golden output once

Invoke the real entry with a known input. Write stdout, stderr, and the exit code. Commit those captured files as the behavior oracle.

# tools/capture_golden.py
"""Proposed capture helper. Run by hand. Review the files."""
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path


def capture(cmd: list[str], gold_dir: Path) -> None:
    gold_dir.mkdir(parents=True, exist_ok=True)
    proc = subprocess.run(cmd, capture_output=True, text=True, check=False)
    (gold_dir / "stdout.txt").write_text(proc.stdout)
    (gold_dir / "stderr.txt").write_text(proc.stderr)
    (gold_dir / "meta.json").write_text(
        json.dumps({"cmd": cmd, "returncode": proc.returncode}, indent=2)
        + "\n"
    )
    print(f"wrote {gold_dir} rc={proc.returncode}")


if __name__ == "__main__":
    capture(sys.argv[1:], Path("goldens") / "seam_a")

python tools/capture_golden.py python -m app.cli invoice --in input.json

Inspect stdout by eye before you commit. Strip timestamps if they change every run. Unstable goldens will produce false failures later.

Record the exact command list inside meta.json. The test must not invent a second command. Drift between capture and test is a false pin.

Step 4: Characterization test against the files

The test does not encode domain rules. It only diffs captured bytes against goldens.

# tests/test_characterize_seam_a.py
import json
import subprocess
from pathlib import Path

GOLD = Path("goldens/seam_a")


def test_seam_a_matches_golden(frozen_env):
    meta = json.loads((GOLD / "meta.json").read_text(encoding="utf-8"))
    proc = subprocess.run(meta["cmd"], capture_output=True, text=True, check=False)
    assert proc.returncode == meta["returncode"]
    assert proc.stdout == (GOLD / "stdout.txt").read_text(encoding="utf-8")
    assert proc.stderr == (GOLD / "stderr.txt").read_text(encoding="utf-8")

Run that test twice before any source edit. A failing pin is not a pin.

pytest tests/test_characterize_seam_a.py -q
pytest tests/test_characterize_seam_a.py -q

Keep the test off the domain language. Words like "correct invoice" do not belong here. The oracle is the committed byte snapshot.

A tiny Makefile keeps the three commands honest. Humans should still inspect every captured golden file.

# Proposed commands. Not a full build.
.PHONY: inventory capture characterize

inventory:
    python tools/inventory_seams.py src > seam_inventory.json

capture:
    python tools/capture_golden.py python -m app.cli invoice --in input.json

characterize:
    pytest tests/test_characterize_seam_a.py -q

Direct call versus subprocess

Choose subprocess when the seam is a CLI entry. Choose a direct call when import side effects stay tame.

Direct calls share the pytest process environment with fixtures. That sharing helps monkeypatch and can hurt isolation.

Subprocess isolation matches a real worker more closely. It also makes debugging a failed pin slower.

Step 5: Decision table — freeze or change

Fill this table before you open a PR. Write the freeze-or-change choice inside the PR description.

Signal	Freeze this pass	Change this pass
Shared env read	Yes if the seam does not own it	Only if this seam owns the key
Import fan-in above three modules	Freeze callees	Change the entry module only
Hidden file I/O	Freeze paths through tmp_path	Change a pure formatter only
Clock or random	Freeze both inside the fixture	Do not "fix" time now
Comment-only drift	Ignore	Ignore
Public stdout shape	Freeze through the golden files	Change only with a new golden

Fan-in here is a local import count. It is not a historical quality metric. Count the files that import the candidate.

Optional ripgrep count if you already have rg. Otherwise grep the candidate module name by hand.

rg -l "from app.cli import|import app.cli" src tests | wc -l

If that count is high, pick a leaf seam. High fan-in turns a small edit into fallout.

Step 6: Normalize only the unstable surface

Some seams print JSON objects with shuffled keys. Add a normalizer that sorts keys, not values. Leave numeric error and field order intact when they matter.

# tools/normalize.py — proposed helper, unexecuted in this article
import json


def normalize_stdout(text: str) -> str:
    raw = text.strip()
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return text if text.endswith("\n") else text + "\n"
    dumped = json.dumps(data, sort_keys=True, separators=(",", ":"))
    return dumped + "\n"

Apply the same normalizer during the capture step. Apply it again inside the pytest assertion. Split normalizers make the golden files lie.

Do not pretty-print unless production already pretty-prints. Added whitespace will later become a fake regression. Match the real wire format as closely as possible.

Step 7: The smallest safe change

Small means one concern in one file. Rename one local, or extract one helper. Do not combine cleanup with behavior changes.

Follow this numbered procedure without skipping any rows. Print the list inside the PR body too.

1. Confirm the characterization test stays green on two runs.
2. Name the single target file in the PR title.
3. Apply one mechanical edit inside that file only.
4. Re-run the golden test after the edit.
5. Revert at once if the golden test fails.
6. Update goldens only when the output change is intended.

The next snippet is a labeled example, not production. It keeps stdout identical while extracting a formatter.

# before: mixed I/O and formatting
def handle(path: str) -> int:
    raw = Path(path).read_text(encoding="utf-8")
    data = json.loads(raw)
    line = f"{data['order_id']}:{data['qty']}\n"
    sys.stdout.write(line)
    return 0


# after: same bytes, one pure function
def format_invoice(data: dict) -> str:
    return f"{data['order_id']}:{data['qty']}\n"


def handle(path: str) -> int:
    raw = Path(path).read_text(encoding="utf-8")
    data = json.loads(raw)
    sys.stdout.write(format_invoice(data))
    return 0

The golden files must still match byte for byte. That match is the only pass bar this pass.

If you need a second helper, stop. Land the first extract as its own commit. Open a new PR after the pin is green.

Step 8: Where a free model belongs

Do not paste the whole repo into a chat. The model did not observe your production traffic. It cannot invent a trustworthy behavior oracle.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. That disclosure applies to the product mention below.

MonkeyCode offers free model access and a free server option. Those two options help only after the pin exists.

The free server can run the same pytest command. The free model can suggest extra capture inputs. Neither option replaces the committed golden files.

The following prompt is a proposal, not an executed run. Do not treat its output as captured truth.

Here is seam_inventory.json and goldens/seam_a/stdout.txt.
List five additional input.json cases this seam likely already accepts.
Do not rewrite source. Do not guess return codes.

Review each suggested input against the real seam. Capture a new golden only for inputs it already handles. Discard fields the current code cannot parse.

Limitations

Golden files do not prove functional correctness at all. They only prove that output stayed stable.

Wrong output stays wrong if you pin it. Review the first capture with a human.

Byte diffs fail on unordered JSON object keys. Normalize dumps with sort_keys when keys shuffle. Do not normalize away the bug you care about.

Subprocess tests will miss in-process monkeypatches entirely. Call a library seam directly if it has no CLI. Keep all file I/O on the tmp_path fixture.

This workflow is a poor fit for GUIs. Streaming sockets also resist stable golden files today.

Parallel tests can stomp a shared golden directory. Give each seam its own goldens folder. Do not write captures from pytest itself.

Who should skip this

Do not use this on cryptographic code paths. Pins will hide timing and constant-time defects.

Do not use this as a license to skip design. A frozen mess is still a messy design.

Skip it when the seam cannot run offline. Network goldens rot as soon as a vendor changes.

Do not let a model rewrite the golden files. That edit deletes the only oracle you had.

Skip it if nobody can name the entry command. Inventory first, or you will pin the wrong process.

What to run on Monday

1. Generate seam_inventory.json for a single package only.
2. Pick one entry-like function with low import fan-in.
3. Capture stdout, stderr, and return code as goldens.
4. Commit those goldens before any refactor commit.
5. Change one file, re-run pytest, then stop.

Start with the worker that nobody wants to touch. The core result should stay boring and local. Most of the mess remains frozen on purpose.

Do not open a second file in the same PR. Wait for the golden test on the default branch. One seam becomes slightly less mixed than yesterday.