Record Entry-Point Goldens Before the Smallest Repo Cut

Record current outputs before you touch a tangled tree. Characterization tests turn today's quirks into a contract. The first edit should be the smallest seam that still compiles.

A messy repo rarely lacks tests by accident. Package imports still open files during module import. One cleanup pass can change bytes callers already store.

This article is a method, not a war story. Examples below are labeled proposals, not production patches. Run them on a copy of your tree.

1. Map the blast radius first

List every public entry before you rename a file. Scripts, CLI parsers, and package exports all count. Private helpers are not the first extraction target.

Do the mapping in this exact order.

1. Search the tree for script entry points and console wrappers.
2. Note modules imported by more than one package path.
3. Record environment keys read during import time.
4. Freeze that inventory in a committed text file.

rg -n "if __name__" -g "*.py"
rg -n "os.environ|Path.cwd|open\(" -g "*.py"
rg -n "entry_points|console_scripts" -g "*.{py,toml,cfg,in}"

Do not refactor during this grep pass. The map is the first artifact you keep. Missing an entry point is the usual silent failure.

Count call sites in the inventory file, not in chat. A number you can re-run beats a remembered list. Re-run the same three commands after every later cut.

2. Give each entry a case directory

Each case is a directory, not a hidden pytest soup. Directories keep stdin, argv, and workdirs obvious. Humans can diff them without a fixture DSL.

Use one layout and do not freelance it.

cases/invoice_csv/
  argv.json
  env.json
  stdin.bin
  target.txt
  work/
    input.csv

Keep target.txt as a relative module path only. Keep work/ as the cwd for that run. Never point the harness at live credentials or customer files.

argv.json should look like a real process vector.

["invoice.py", "--format", "csv", "input.csv"]

env.json should be a full map, not a delta. Partial env hides locale, timezone, and home-path bugs. Dump the process env once, then delete secrets by hand.

Add stdin.bin only when the entry reads standard input. Empty stdin still needs an explicit empty file. Implicit emptiness makes later diffs harder to trust.

3. Freeze bytes with a golden harness

The harness below is a proposal, not a measured benchmark. It writes goldens on the first run. Later runs fail on any byte-level drift.

Label: unexecuted example. Adapt paths before you run it.

# char_harness.py
from __future__ import annotations

import hashlib
import json
import os
import runpy
import sys
from pathlib import Path
from typing import Any

ROOT = Path(__file__).resolve().parent
GOLDEN = ROOT / "goldens"
CASES = ROOT / "cases"


def _digest(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()


def _run_case(case_dir: Path) -> dict[str, Any]:
    stdin_path = case_dir / "stdin.bin"
    argv = json.loads((case_dir / "argv.json").read_text(encoding="utf-8"))
    extra_env = json.loads((case_dir / "env.json").read_text(encoding="utf-8"))
    target = (case_dir / "target.txt").read_text(encoding="utf-8").strip()
    stdin_data = stdin_path.read_bytes() if stdin_path.exists() else b""

    old_argv = sys.argv[:]
    old_env = os.environ.copy()
    old_cwd = Path.cwd()
    try:
        sys.argv = argv
        os.environ.clear()
        os.environ.update(extra_env)
        os.chdir(case_dir / "work")
        ns = runpy.run_path(str(ROOT / target), run_name="__main__")
        return {
            "ok": True,
            "result": ns.get("RESULT"),
            "stdin_sha": _digest(stdin_data),
        }
    except Exception as exc:  # pin the type name, not a traceback
        return {
            "ok": False,
            "error_type": type(exc).__name__,
            "error": str(exc),
            "stdin_sha": _digest(stdin_data),
        }
    finally:
        sys.argv = old_argv
        os.environ.clear()
        os.environ.update(old_env)
        os.chdir(old_cwd)


def main() -> int:
    update = "--update" in sys.argv
    GOLDEN.mkdir(exist_ok=True)
    failed = 0
    for case_dir in sorted(p for p in CASES.iterdir() if p.is_dir()):
        observed = _run_case(case_dir)
        blob = json.dumps(observed, sort_keys=True, default=str).encode()
        golden_path = GOLDEN / f"{case_dir.name}.json"
        if update or not golden_path.exists():
            golden_path.write_bytes(blob)
            print(f"WROTE {golden_path}")
            continue
        expected = golden_path.read_bytes()
        if blob != expected:
            failed += 1
            print(f"DRIFT {case_dir.name}")
            print(f"  expected_sha={_digest(expected)}")
            print(f"  observed_sha={_digest(blob)}")
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the harness in two steps, never one mixed step.

python char_harness.py --update
python char_harness.py
echo $?

The first command writes goldens from a cold tree. The second command must exit zero on that same tree. Commit cases and goldens in the same review.

Pin four fields for every entry you keep.

1. Canonical input bytes or parsed argument objects stay fixed.
2. Return value, output bytes, and raised type stay fixed.
3. Working directory, env keys, and argv stay fixed.
4. Files created, updated, or deleted stay fixed.

Skip private helpers on this first pass. Private names move when you extract later. Public results are the only contract that callers already have.

4. Treat drift as a stop sign

A failed hash is not a style debate. Open the two JSON blobs side by side. Confirm whether the change was intended at all.

If the drift is accidental, revert the edit immediately. If the drift is the point, update that one golden. Never batch-update goldens after a wide rewrite.

Print both hashes in the review notes. Reviewers can reject a hash they cannot explain. Unexplained goldens are how bugs become fixtures.

5. Cut one seam, then stop

Only one behavior-preserving edit follows a green harness. Extract a function the goldens already cover. Keep the old import path as a re-export.

Follow these change rules in order.

1. Extract one function the goldens already exercise end to end.
2. Keep the old import path as a one-line re-export.
3. Do not rename keyword arguments on this same pass.
4. Re-run the harness after that single edit only.

# proposed re-export, unexecuted example
from app.billing.csv_out import render_invoice as render_invoice

Stop if any golden moves after the extract. The seam is wrong or a case is missing. Restore the file, add a case, and only then retry.

Do not count deleted lines as progress here. Progress is a green harness plus one moved symbol. Architecture cleanup waits for a later, intentional golden change.

Where a free model belongs

A model is useful after traces exist, not before. It can suggest missing argv combinations from case files. It must not replace the harness or the revert rule.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. That pair fits throwaway harness runs on a dirty tree. Paste traces in, take candidate cases out, and keep the revert decision local.

Do not ask the model to rewrite the messy package. Ask it to list inputs the goldens never cover. You still add those case directories by hand.

Reject invented APIs from chat by default. The messy module already has an API, however ugly. Characterization work records that API; it does not replace it.

Decision table for the next hour

Observation	Next step	Do not
Goldens missing for an entry	Add one case directory	Extract that entry
One function, goldens green	Extract and re-export	Rename callers
Drift on error_type only	Pin the exception class	Swallow the error
Drift on timestamps	Freeze time inside the case	Update goldens blindly
Model proposes a new API	Reject the names	Accept names from chat
Import opens a file	Pin that file in work/	Mock after the extract

Read the table left to right before each commit. The middle column is the only allowed action. The right column is how this method usually dies.

Limitations you should budget for

Characterization tests will lock bugs in place. That lock is the point of the first cut. Plan a later pass that changes goldens on purpose.

Golden JSON will rot when serializers change whitespace. sort_keys=True and default=str are part of the contract. Do not pretty-print in one run and compact in the next.

runpy.run_path is not a full process boundary. Shared process state can leak across case directories. Split to subprocesses if order-dependent failures appear.

# proposed subprocess pin, unexecuted example
import subprocess
import sys

proc = subprocess.run(
    [sys.executable, str(ROOT / target), *argv[1:]],
    cwd=case_dir / "work",
    env=extra_env,
    input=stdin_data,
    capture_output=True,
    check=False,
)
payload = {
    "returncode": proc.returncode,
    "stdout_sha": _digest(proc.stdout),
    "stderr_sha": _digest(proc.stderr),
}

Count stream bytes when you switch to subprocesses. stdout, stderr, and returncode belong in the golden. In-process RESULT shortcuts miss stream data that callers already parse.

Network, clocks, and RNG do not belong in raw goldens. Fake those seams before the first --update. Otherwise the harness trains you to ignore red runs.

Who should not use this method

Do not use this method on unread crypto or auth code. Pinning output can freeze a vulnerability as expected. Get a review before any golden update in that code.

Do not use it when you cannot reproduce a run. Flaky time, network, and RNG belong in fakes first. A drifting harness is worse than no harness.

Do not outsource the revert decision to a model. The harness is the authority for this pass. Chat text is not a test, and it is not a commit.

Skip the method if you already have contract tests. Those tests already name the behavior you would pin. Adding goldens on top only duplicates failure noise.

What done means on this pass

Done is a green harness plus one moved function. It is not a cleaned architecture. It is not a new framework around the mess.

Commit in this order only, with no extras.

1. Commit case directories and the harness with empty goldens omitted.
2. Commit goldens produced from one cold --update run.
3. Commit the single extract and the old-path re-export.
4. Commit harness output that proves zero drift on that extract.

If step four fails, revert step three before anything else. Add a case that names the missed branch. Repeat from the last green tree, not from memory.

This sequence is slow on purpose. Speed is how messy repos lose production behavior. Pin first, cut once, then stop for the day.