CLI Goldens Are the Spec Your Messy Repo Already Has

Never refactor a messy repository from unverified internal guesses. Pin observable output using golden-master characterization tests first. Then apply the smallest safe change at one seam.

The failure mode

Cheap generation does not make a refactor cheap to reverse. AI diffs look coherent while they rewrite hidden contracts. Messy repositories hide those contracts in files and stdout.

A green unit suite often covers the wrong layer. Helpers receive tests, while byte-level output often does not. The job still drifts at the operator-visible boundary.

What this article specifies

This procedure is a proposed workflow, not a field report. Example code is an unexecuted template, not production evidence. No model names, quotas, or hardware claims appear below.

The original artifact is a golden-master boundary harness. It records CLI and file output, then gates the first edit. Internals stay out of scope until that gate is red on purpose.

When the method applies

Use it on brownfield scripts with weak or missing tests. Use it when several modules share one observable job. Skip it for greenfield code with a stable public API.

Step 1: Name the observable job

Pick one user-visible job, not a class name. Examples include invoice export, log redaction, or report merge. Write the job as one command with frozen inputs.

# proposed command contract, not a live capture
python -m messy_app.cli export \
  --in fixtures/in/happy/sample.json \
  --out /tmp/messy-out \
  --tz UTC

Record the exact argv and the working directory. Record environment variables that change output. Leave module names out of the contract document.

Step 2: Freeze inputs as fixtures

Copy a real input set into fixtures/in/. Strip secrets. Keep shape, encoding, and ugly edge cases.

A fixture set needs three things only. First, input files. Second, argv. Third, captured output. Anything else belongs in a later characterization pass.

Proposed layout:

fixtures/in/happy/argv.json
fixtures/in/happy/env.json
fixtures/in/happy/sample.json
fixtures/in/empty/argv.json
fixtures/in/ugly-encoding/argv.json
fixtures/goldens/happy.json
fixtures/goldens/empty.json
fixtures/goldens/ugly-encoding.json

Keep goldens in git beside the inputs. Do not regenerate them during a refactor commit.

Step 3: Record the golden masters

Run the current code once. Capture stdout, stderr, and output files. Hash binary artifacts. Keep text output as UTF-8.

Do not pretty-print captured text. Pretty-print hides significant whitespace. Do not sort keys unless the job itself sorts them.

Proposed recorder (unexecuted template):

"""Golden-master recorder. Template only. Not a live run."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
FIX = ROOT / "fixtures"
GOLD = FIX / "goldens"


def run_job(case: str) -> dict:
    case_dir = FIX / "in" / case
    out_dir = Path("/tmp") / f"messy-{case}"
    if out_dir.exists():
        for p in out_dir.rglob("*"):
            if p.is_file():
                p.unlink()
    argv = json.loads((case_dir / "argv.json").read_text())
    env = os.environ.copy()
    extra = case_dir / "env.json"
    if extra.exists():
        env.update(json.loads(extra.read_text()))
    proc = subprocess.run(
        argv,
        cwd=str(ROOT),
        env=env,
        capture_output=True,
        text=True,
        check=False,
    )
    files = {}
    if out_dir.exists():
        for p in sorted(out_dir.rglob("*")):
            if not p.is_file():
                continue
            rel = str(p.relative_to(out_dir))
            data = p.read_bytes()
            files[rel] = {
                "sha256": hashlib.sha256(data).hexdigest(),
                "n": len(data),
            }
    return {
        "returncode": proc.returncode,
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "files": files,
    }


def record(case: str) -> Path:
    GOLD.mkdir(parents=True, exist_ok=True)
    payload = run_job(case)
    path = GOLD / f"{case}.json"
    path.write_text(json.dumps(payload, indent=2, sort_keys=True))
    return path

Record at least three cases before any edit. Include one happy path. Include one empty input. Include one ugly encoding or date-boundary case.

# proposed recording loop
python -c "from recorder import record; [print(record(c)) for c in ('happy','empty','ugly-encoding')]"

Step 4: Turn recordings into tests

A characterization test must fail when output drifts. It must not assert internal call graphs. It must not assert private helper names.

Proposed test (unexecuted template):

"""Boundary characterization tests. Template only."""
import json
from recorder import GOLD, run_job

CASES = ["happy", "empty", "ugly-encoding"]


def test_golden_masters_match():
    mismatches = []
    for case in CASES:
        gold_path = GOLD / f"{case}.json"
        gold = json.loads(gold_path.read_text())
        now = run_job(case)
        if now != gold:
            mismatches.append(case)
    assert mismatches == [], mismatches

Run the tests once against the recordings. They should pass on the untouched tree. If they fail, the recorder is wrong, so fix the recorder.

Step 5: Prove the tests can fail

Break one observable on purpose. Change a date format, flip a sort, or drop a newline. Re-run the suite. It must fail on that case only.

If the suite still passes, the harness is theater. Do not refactor yet. Widen capture until the break is visible.

# proposed mutation of one observable, then restore
python -c "from pathlib import Path; p=Path('messy_app/export.py'); t=p.read_text(); p.write_text(t.replace('%Y-%m-%d','%m/%d/%Y'))"
pytest -q tests/test_goldens.py
git checkout -- messy_app/export.py

That checkout is mandatory. Leave no stray mutation in the tree. A gate that cannot go red cannot protect a later extract.

Step 6: Score seams, then pick one

Do not start inside the densest file. List files that write the job's output. Choose the seam that touches the fewest of those writes.

Proposed seam scan (unexecuted template):

"""Proposed seam score. Unexecuted. Not a benchmark."""
from collections import defaultdict
from pathlib import Path
import ast

MARKERS = {"stdout", "stderr", "write", "Path", "open"}


def score_writers(root: Path) -> dict[str, int]:
    hits: dict[str, int] = {}
    counts: dict[str, int] = defaultdict(int)
    for path in root.rglob("*.py"):
        if "tests" in path.parts or "fixtures" in path.parts:
            continue
        try:
            ast.parse(path.read_text(encoding="utf-8"))
        except (SyntaxError, UnicodeDecodeError):
            continue
        text = path.read_text(encoding="utf-8")
        score = sum(text.count(m) for m in MARKERS)
        if score:
            counts[str(path.relative_to(root))] = score
    hits = dict(sorted(counts.items(), key=lambda kv: kv[1]))
    return hits

Low score is not automatically safe. It is only a candidate list. Read the top and bottom of that list before editing.

Decision table for the first edit:

Signal	Prefer this seam	Avoid this seam
Output ownership	One module writes the job bytes	Many modules print fragments
Fixture sensitivity	One flag changes captured files	Hidden cwd changes captured files
Diff blast radius	Local helper extract	Shared date formatter move
Rollback	One file, one commit	Repo-wide format-and-fix

Only one Prefer cell needs to be true. If every Avoid cell is true, stop. Split the job before any extract.

Step 7: The smallest safe change

State the change in one sentence before editing. Valid example: extract _format_row without changing bytes. Invalid example: clean up the exporter and related utils.

Rules for the commit:

1. Touch one module, or one function, not both layers.
2. Keep argv, env, and output paths identical.
3. Do not reformat unrelated files.
4. Re-run golden masters after every hunk.
5. Abort if a second case fails without a written story.

Proposed extract (illustrative only):

# before: inline formatting inside the export loop
# after: local helper, same bytes

def _format_row(row: dict) -> str:
    date = row["ts"].strftime("%Y-%m-%d")
    amount = f"{row['amount']:.2f}"
    return f"{date},{row['id']},{amount}\n"


def export_rows(rows: list[dict], sink) -> None:
    for row in rows:
        sink.write(_format_row(row))

The helper is allowed only if goldens stay identical. If stdout gains a newline, revert immediately. Do not fix forward during this commit.

pytest -q tests/test_goldens.py
git diff --stat

git diff --stat should show one primary path. A second path needs a new change sentence.

Using a free model without trusting it

A free coding model can draft stubs from recorded fixtures. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two options can host and run this harness.

Prompt the model with fixtures, not with the whole tree. Ask for tests that compare recorded JSON, nothing else. Reject any patch that edits production modules in the same step.

The server option helps when local setup is already messy. Run recorder and pytest there. Keep fixtures in git. Treat every generated assertion as untrusted until it fails on purpose.

What the model must not do

Do not let it invent missing cases as truth. Do not let it normalize JSON if the job emits text. Do not let it rewrite timestamps to make tests stable.

Stability comes from a frozen clock in argv or env. It does not come from filters after capture.

# proposed clock freeze via env, template only
env["MESSY_NOW"] = "2026-09-05T00:00:00Z"

That timestamp is a fixture value for this draft day. It is not a performance metric. It is not a product claim.

Limitations

Golden masters do not prove correctness. They prove that observable bytes did not drift. Wrong output stays wrong if you record it.

Large binary goldens rot in review. Prefer hashes plus byte size. Nondeterministic clocks, uuids, and iteration order will flake. Pin them at the boundary, or exclude that job.

This workflow is slow on GUI apps and networked services. It fits CLI tools, batch jobs, and file transformers. It does not replace typed public APIs or consumer contract tests.

Parallel jobs that share /tmp will collide. Namespace output directories per case. Cleanup must be part of run_job, not a manual habit.

Who should not use this approach

Do not use it as a license to skip design. Do not use it on security-sensitive parsers without review. Do not use it when output must change, such as a format migration. Do not use it to rubber-stamp a multi-module rewrite.

Teams without fixture hygiene will record secrets. Teams chasing coverage numbers will snapshot noise. Neither group should run this harness.

Checklist before the first production-shaped commit

1. Three golden cases exist on the current tree.
2. A deliberate mutation failed the matching case.
3. The change sentence names one seam.
4. pytest matches goldens after the hunk.
5. git diff --stat shows one primary path.

If any box is unchecked, there is no refactor yet. There is only a guess with extra files. Put the extract back and restore the goldens.

Closing

Messy repositories do not document themselves. Their real spec is the bytes they already emit. Characterize those bytes. Then change one seam.

If free model access is already in your loop, generate boundary tests only, and keep the first production edit human-sized.