Freeze Output Paths and Content Digests Before One main() Split

Do not split a messy CLI until outputs freeze. Capture output paths, content digests, and stdout bytes first. Then extract one helper with no behavior change.

A green unit suite is not enough here. main() often writes files tests never open. A later extract reorders writes and still passes.

The failure this harness catches

Messy CLIs hide behavior at the process boundary. They print banners, create reports, and exit zero. Helpers pulled from main() change those artifacts first.

File counts can stay stable while contents drift. Stdout text can look identical after newline folds. Digests and raw bytes catch both classes.

This article records a black-box freeze method. It does not claim production metrics. Treat every listing as a proposed, unexecuted example.

What to freeze before any extract

Freeze four observables on one fixture tree. Keep the list short and named.

1. Process exit code as an integer.
2. Raw stdout and stderr as bytes.
3. Relative output paths as a sorted tuple.
4. SHA-256 digests of those files.

Do not freeze wall-clock timestamps in this pass. Do not freeze absolute prefixes from tmp. Those values move without a logic change.

Decision table

Observable	Freeze?	Why
Exit code	Yes	Callers branch on it
stdout bytes	Yes	Banners and tables drift
stderr bytes	Yes	Warnings are behavior
Relative output paths	Yes	Extracts reorder writes
SHA-256 of each file	Yes	Silent content edits hide
Absolute tmp prefix	No	Hosts assign new roots
mtime / ctime	No	Every run writes new times
Interpreter version banner	No	Unless the CLI prints it

Skip a row only with a written reason. Put that reason in the PR. Reviewers should reject silent golden edits.

Artifact: a black-box characterization harness

The script below is a proposed harness. It is not labeled as executed. Copy it, then point CMD at your CLI.

#!/usr/bin/env python3
"""Characterize a messy CLI before splitting main()."""

from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
FIXTURE = ROOT / "fixtures" / "sample_tree"
OUT_DIR = ROOT / "tmp_characterize"
GOLDEN = ROOT / "goldens" / "cli_sample.json"

CMD = [sys.executable, str(ROOT / "messy_cli.py"), "--out", str(OUT_DIR)]


def digest_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def collect_outputs(base: Path) -> dict[str, str]:
    mapping: dict[str, str] = {}
    if not base.exists():
        return mapping
    for path in sorted(base.rglob("*")):
        if path.is_file() and not path.is_symlink():
            rel = path.relative_to(base).as_posix()
            mapping[rel] = digest_file(path)
    return mapping


def reset_out_dir() -> None:
    if not OUT_DIR.exists():
        OUT_DIR.mkdir(parents=True)
        return
    for child in sorted(OUT_DIR.rglob("*"), reverse=True):
        if child.is_file() or child.is_symlink():
            child.unlink()
        elif child.is_dir():
            child.rmdir()
    OUT_DIR.rmdir()
    OUT_DIR.mkdir(parents=True)


def run_cli() -> dict:
    reset_out_dir()
    env = os.environ.copy()
    env["PYTHONHASHSEED"] = "0"
    env["TZ"] = "UTC"
    env["LANG"] = "C"
    proc = subprocess.run(
        CMD,
        cwd=str(FIXTURE),
        env=env,
        capture_output=True,
        check=False,
    )
    files = collect_outputs(OUT_DIR)
    return {
        "returncode": proc.returncode,
        "stdout_sha256": hashlib.sha256(proc.stdout).hexdigest(),
        "stderr_sha256": hashlib.sha256(proc.stderr).hexdigest(),
        "stdout_len": len(proc.stdout),
        "stderr_len": len(proc.stderr),
        "paths": sorted(files.keys()),
        "digests": files,
    }


def main(argv: list[str]) -> int:
    record = run_cli()
    GOLDEN.parent.mkdir(parents=True, exist_ok=True)
    if argv[1:] == ["--update"]:
        payload = json.dumps(record, indent=2, sort_keys=True) + "\n"
        GOLDEN.write_text(payload, encoding="utf-8")
        print(f"updated {GOLDEN}")
        return 0
    if not GOLDEN.exists():
        print("missing golden; run with --update once", file=sys.stderr)
        return 2
    expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
    if record != expected:
        print("characterization mismatch", file=sys.stderr)
        print("expected", json.dumps(expected, indent=2, sort_keys=True))
        print("actual", json.dumps(record, indent=2, sort_keys=True))
        return 1
    print("characterization ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Store stdout as a digest, not decoded text. Decoding hides \r\n versus \n. Length fields help when dumps look similar. SHA-256 remains the pass or fail key.

Pin PYTHONHASHSEED, TZ, and LANG in the child. Hash randomization reorders some debug prints. Timezone shifts date columns in reports. Locale changes decimal commas and month names.

Encoding traps that fake a clean extract

UTF-8 and UTF-8-SIG are not the same bytes. A helper that sets encoding="utf-8" can drop a BOM. The digest then changes with no visible diff.

Windows print() calls may emit \r\n today. Path.write_text may emit \n after an extract. Freeze bytes, not decoded strings, for that reason.

JSON dumps change when separators move. They also change when sort_keys flips. Copy the current dump flags into the extract. Upgrade the format in a later PR.

Fixture hygiene

Use a committed tree under fixtures/sample_tree. Do not read from a home directory. Home paths leak usernames into reports and goldens.

Reset OUT_DIR at the start of each run. Leftover files from a crash poison path sets. The harness deletes files before it recreates the directory.

Keep CMD in one list. Flags hidden in shell aliases are not a freeze. Reviewers must see the full vector.

Numbered workflow

Follow these steps in order. Do not skip the freeze.

1. Pick one fixture tree under fixtures/.
2. Document the exact command vector in CMD.
3. Run the harness with --update once.
4. Commit goldens/cli_sample.json in that PR.
5. Re-run without --update on a clean tree.
6. Extract one helper from main(), nothing else.
7. Re-run the harness. Digests must match.
8. Only then open a second extract PR.

Step 6 is the smallest safe change. If the extract needs extra edits, stop. Restore main() and shrink the extract.

How to read a mismatch

A path set change means write order or filters moved. A digest change with the same paths means content moved. A stdout digest change with stable files means banners moved.

Exit code changes are not cosmetic. Treat them as product changes. Update the golden only with a human note.

Do not auto-update goldens in CI. That deletes the freeze. CI should run the harness without --update.

The smallest safe change

Keep main() as the process boundary. Move one write helper out. Do not rename flags in the same PR.

Proposed extract shape, unexecuted:

def write_report(out_dir: Path, rows: list[dict]) -> Path:
    out_dir.mkdir(parents=True, exist_ok=True)
    target = out_dir / "report.json"
    payload = json.dumps(rows, indent=2, sort_keys=True)
    target.write_text(payload + "\n", encoding="utf-8")
    return target

Call it from main() with the same rows. Do not add logging in this PR. Do not change sort_keys later without a freeze update.

If messy_cli.py currently omits sort_keys, copy that bug. Characterization tests protect current behavior. They do not upgrade on-disk format.

Commands to keep in the PR

python3 characterize_cli.py --update
python3 characterize_cli.py
git add goldens/cli_sample.json characterize_cli.py
git diff --stat

--stat should list the extract, the harness, and the golden. Extra files mean the change grew. Split those files into another PR.

Using a scratch model without trusting the extract

A free coding model can draft this harness from a file list. It cannot see uncommitted fixture drift. Run the script on the local tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Sketch the harness there if a spare box is missing. Keep the golden file and the CLI run on hardware you control.

Reject any model patch that edits goldens and main() together. That hides the mismatch. The freeze is the review surface.

Limitations

This harness ignores symlink targets on purpose. Follow-up work can add os.lstat records. It also ignores directory-only paths. Empty directories will not appear in digests.

Large binary outputs make JSON goldens heavy. Switch those files to sidecar .sha256 lists if needed. Do not hash files outside --out.

Nondeterministic CLIs need more pins first. Random IDs, wall clocks, and network calls break digests. Stub those seams before you freeze. If those seams cannot be stubbed, skip this method.

Who should not use this approach

Skip this if the CLI has no stable output directory. Skip it if every run talks to a live network. Skip it if the patch must upgrade formats now.

Do not use characterization as a substitute for new-feature tests. The freeze locks the past. New flags need new fixtures and new goldens.

Close

Split main() only after paths and digests match. The extract is done when the golden is quiet. If the golden moves, the change was not small.