Pin the Output Tree Before You Touch Messy Code

Characterize the whole command before you edit any file. The real contract lives in files, streams, and exit codes. Hash that output tree, then change exactly one helper.

Messy repositories hide contracts inside mixed side effects. Internal functions blend path logic, I/O, and formatting. Early unit tests often freeze the mess instead of behavior.

A process-boundary tape beats a guessed structural rewrite. Freeze one fixture and run the current entry point once. Commit the golden manifest before any structural edit.

Why function tests stall on brownfield code

Brownfield scripts rarely expose a clean unit surface. Callers pass paths, flags, and implicit working directories. A function test would mock away behavior you must keep.

Public API diffs also miss generator tools that write files. Many internal modules never form a stable import contract. The command line and output directory are the true surface.

This article uses an example generator, not production metrics. Treat every snippet as a labeled, unexecuted template. Adapt names and paths to your own repository layout.

What the oracle must capture

Capture four observable channels from one frozen invocation. Record stdout text, stderr text, and the process exit code. Then hash every file under the designated output directory.

Skip nothing that operators actually inspect after a run. If users open out/summary.json, that file belongs in the tape. If logs are incidental, exclude them with an explicit pattern.

Absolute paths and timestamps will poison the golden file. Normalize those fields before hashing, or the tape flakes. Document each normalization rule next to the fixture.

Artifact: a golden manifest harness

The harness below is example code, not a measured benchmark. It runs one command, then writes or checks golden.json. Store the fixture and the golden file in the same commit.

#!/usr/bin/env python3
"""Process-boundary characterization for a messy CLI.

Example template. Not executed against a production repo.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import shutil
import subprocess
import sys
from pathlib import Path

IGNORE_NAMES = {".DS_Store", "Thumbs.db"}
IGNORE_SUFFIXES = {".log"}


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def sha256_file(path: Path) -> str:
    return sha256_bytes(path.read_bytes())


def normalize_text(raw: str, output_root: Path) -> str:
    # Example only: strip the output root so paths stay portable.
    return raw.replace(str(output_root), "<OUT>")


def list_output_files(output_root: Path) -> list[Path]:
    files: list[Path] = []
    if not output_root.exists():
        return files
    for path in sorted(output_root.rglob("*")):
        if not path.is_file():
            continue
        if path.name in IGNORE_NAMES:
            continue
        if path.suffix in IGNORE_SUFFIXES:
            continue
        files.append(path)
    return files


def build_manifest(
    result: subprocess.CompletedProcess[str],
    output_root: Path,
) -> dict:
    files = []
    for path in list_output_files(output_root):
        rel = path.relative_to(output_root).as_posix()
        files.append({"path": rel, "sha256": sha256_file(path)})
    return {
        "returncode": result.returncode,
        "stdout_sha256": sha256_bytes(
            normalize_text(result.stdout, output_root).encode("utf-8")
        ),
        "stderr_sha256": sha256_bytes(
            normalize_text(result.stderr, output_root).encode("utf-8")
        ),
        "files": files,
    }


def run_command(argv: list[str], cwd: Path) -> subprocess.CompletedProcess[str]:
    return subprocess.run(
        argv,
        cwd=cwd,
        text=True,
        capture_output=True,
        check=False,
    )


def write_golden(path: Path, manifest: dict) -> None:
    path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")


def check_golden(path: Path, manifest: dict) -> int:
    expected = json.loads(path.read_text())
    if expected == manifest:
        print("golden manifest matches")
        return 0
    print("golden manifest mismatch")
    print("expected:")
    print(json.dumps(expected, indent=2, sort_keys=True))
    print("actual:")
    print(json.dumps(manifest, indent=2, sort_keys=True))
    return 1


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("mode", choices=["write", "check"])
    parser.add_argument("--fixture", default="fixtures/case-a")
    parser.add_argument("--output-root", default="out")
    parser.add_argument("--golden", default="fixtures/case-a.golden.json")
    parser.add_argument(
        "--cmd",
        nargs=argparse.REMAINDER,
        default=[sys.executable, "report_gen.py", "--in", "input", "--out", "../out"],
    )
    args = parser.parse_args()
    fixture = Path(args.fixture).resolve()
    output_root = Path(args.output_root).resolve()
    golden = Path(args.golden)
    argv = args.cmd[1:] if args.cmd[:1] == ["--"] else args.cmd

    if output_root.exists():
        shutil.rmtree(output_root)
    output_root.mkdir(parents=True)

    result = run_command(argv, cwd=fixture)
    manifest = build_manifest(result, output_root)
    if args.mode == "write":
        write_golden(golden, manifest)
        print(f"wrote {golden}")
        return 0
    if not golden.exists():
        print(f"missing {golden}; run write mode first")
        return 2
    return check_golden(golden, manifest)


if __name__ == "__main__":
    raise SystemExit(main())

Run write mode once against the untouched messy command. Run check mode after every subsequent source change. A mismatched hash means the observable contract moved.

python golden_manifest.py write --output-root out
python golden_manifest.py check --output-root out

Keep those two commands in the repository README. Do not keep golden files only inside chat logs. The tape belongs beside the fixture, inside version control.

Numbered workflow

1. Inventory the entry point

Find the command operators actually run in this repository. Write that command down as a single argv list. Do not start inside a random helper module.

2. Freeze a fixture

Copy one real input tree into the fixtures/case-a folder. Strip secrets, credentials, and machine-specific absolute paths. Keep the fixture small enough for a fast local run.

3. Declare the output root

Choose one directory the command is allowed to write. Point the command at that directory with an explicit flag. Delete the directory between runs so leftovers cannot hide.

4. Record the golden manifest

Execute the command from a clean working directory. Persist exit code, streams, and per-file SHA-256 digests. Commit the fixtures/case-a tree and golden.json together.

5. Make the smallest internal change

Pick one duplicated concern, not a layer rewrite. Path joining is a typical first extraction in generators. Leave control flow, I/O order, and file names untouched.

6. Re-run the tape

Rebuild the output directory from the same fixture. Fail the change if any digest or stream differs. Only then consider a second extraction in another commit.

Example messy generator

The example CLI writes a JSON summary and a text report. It duplicates path joins and mixes listing with rendering. Do not treat this sample as a real customer codebase.

# report_gen.py — messy baseline, example only
from __future__ import annotations

import argparse
import json
from pathlib import Path


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--in", dest="input_dir", required=True)
    parser.add_argument("--out", dest="output_dir", required=True)
    args = parser.parse_args()
    input_dir = Path(args.input_dir)
    output_dir = Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    names = sorted(p.name for p in input_dir.iterdir() if p.is_file())
    summary_path = output_dir / "summary.json"  # duplicated join style
    report_path = output_dir / "report.txt"

    summary = {"count": len(names), "files": names}
    summary_path.write_text(json.dumps(summary, indent=2) + "\n")
    lines = [f"count={len(names)}"] + [f"- {name}" for name in names]
    report_path.write_text("\n".join(lines) + "\n")
    print(f"wrote {summary_path.name} and {report_path.name}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The smallest safe change extracts one path helper. File names, JSON keys, and report wording stay identical. The golden manifest should remain unchanged after extraction.

# report_gen.py — after one extraction, example only
from __future__ import annotations

import argparse
import json
from pathlib import Path


def under(root: Path, *parts: str) -> Path:
    return root.joinpath(*parts)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--in", dest="input_dir", required=True)
    parser.add_argument("--out", dest="output_dir", required=True)
    args = parser.parse_args()
    input_dir = Path(args.input_dir)
    output_dir = Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    names = sorted(p.name for p in input_dir.iterdir() if p.is_file())
    summary_path = under(output_dir, "summary.json")
    report_path = under(output_dir, "report.txt")

    summary = {"count": len(names), "files": names}
    summary_path.write_text(json.dumps(summary, indent=2) + "\n")
    lines = [f"count={len(names)}"] + [f"- {name}" for name in names]
    report_path.write_text("\n".join(lines) + "\n")
    print(f"wrote {summary_path.name} and {report_path.name}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Stop after that helper lands. Do not rename JSON keys in the same commit. Do not restyle the report text while extracting paths.

After the harness stays green

Cheap model edits still need an external oracle. Run the checker on a clean machine before requesting edits.

MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use the free server when local disks inject extra files.

If you try that loop, paste the manifest diff, not the tree. Ask the model for one helper extraction, nothing larger.

Decision table

Choose the oracle from the actual observable surface. Do not default to function tests because they look tidy.

Situation	Oracle to pin	Smallest next change
CLI writes an output tree	Golden file manifest	One path or format helper
Library with a stable import surface	Exported symbol snapshot	One leaf function
HTTP service with JSON clients	Response fixture bytes	One handler branch
Core already covered by units	Keep those units	Skip this tape

Use this table before opening an editor. If two oracles apply, pick the one operators already trust. Extra oracles can wait for a later commit.

Limits of this oracle

The tape cannot see pure in-memory refactors that preserve bytes. That is acceptable when operators only care about files. It is insufficient when you must prove internal state.

Nondeterministic clocks will break SHA-256 comparison. So will random UUIDs and unordered set iteration. Normalize or freeze those sources before recording goldens.

Network calls do not belong in this fixture loop. Stub them at the process boundary, or the tape becomes flaky. Secrets must never enter fixtures/case-a or golden.json.

Byte identity is a harsh contract for rendered HTML. Tiny whitespace shifts will fail check mode. Prefer this method for data files, not layout-sensitive markup.

Parallel writers can race inside the output directory. Serialize the command under characterization, then keep it serialized. A racy golden file teaches the wrong lesson.

Who should skip this approach

Skip it on greenfield code with a designed module surface. Write ordinary unit tests there and avoid a process tape. The extra harness would hide a better design.

Skip it when the command deletes unknown user files. A recursive output wipe is part of this workflow. That is unsafe against a home directory or shared disk.

Skip it for long-running daemons without a batch entry point. You need one finite invocation and a closed output tree. Attach a different oracle for resident services.

Skip it if legal review forbids copying production fixtures. Synthetic inputs are fine when they still exercise the contract. Empty toys that never hit branches are not fine.

Close the loop with one commit rule

One commit records the fixture and the golden manifest. The next commit contains only the helper extraction. A third commit starts only after check mode stays quiet.

That split keeps blame useful during later review. It also makes a bad extraction cheap to revert. The output tree either matches, or the change does not ship.