Pin the CLI Tape Before You Edit a Messy Repo

Do not refactor a messy repo by reading it. Record a CLI golden tape before any source change. Edit one file only after replay stays green.

Hidden branches often survive a visual code review. Exit codes can drift without a failing test. A large cleanup then ships silent behavior changes.

Pin observables, not opinions

A messy entrypoint still has one honest contract. That contract is whatever callers already observe today. Internal structure is not the caller contract yet.

Capture four fields for every frozen fixture. Store the process exit code as an integer. Store stdout text, stderr text, and output hashes.

Do not pin function names in this pass. Do not pin import graphs in this pass. Those signals help later, not first.

Why a tape beats inspection

Characterization tests freeze behavior, including present bugs. Feathers documented this approach for legacy systems. Approval tests store the same idea as golden files.

You do not need a correct specification today. You need a replayable oracle for every later edit. The oracle then makes each later patch falsifiable.

Guessing intent from a tangled script is cheap. Proving no caller-visible drift is not cheap. The committed golden tape is that behavioral proof.

Labeled sample under test

The script below is only a labeled sample. This listing does not represent captured production traffic. Use it as a stand-in messy CLI.

#!/usr/bin/env python3
"""sample_messy_cli.py — labeled sample, not live production code."""
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def _load(path: Path) -> dict:
    if not path.exists():
        return {"rows": [], "mode": "empty"}
    raw = path.read_text(encoding="utf-8")
    if path.suffix == ".json":
        return json.loads(raw)
    rows = [line.strip() for line in raw.splitlines() if line.strip()]
    return {"rows": rows, "mode": "text"}


def _score(rows: list[str]) -> int:
    total = 0
    for row in rows:
        if row.startswith("#"):
            continue
        total += len(row)
        if "TODO" in row:
            total += 10
    return total


def main(argv: list[str]) -> int:
    if len(argv) < 3:
        sys.stderr.write("usage: sample_messy_cli.py IN OUT\n")
        return 2
    src = Path(argv[1])
    dest = Path(argv[2])
    payload = _load(src)
    score = _score(payload.get("rows", []))
    if payload.get("mode") == "empty":
        sys.stdout.write("empty\n")
        dest.write_text("empty\n", encoding="utf-8")
        return 0
    digest = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()[:12]
    dest.write_text(f"{score}:{digest}\n", encoding="utf-8")
    sys.stdout.write(f"score={score}\n")
    if score == 0:
        sys.stderr.write("warn: zero score\n")
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The sample mixes file IO with scoring logic. Empty inputs follow a different return path here. Zero scores return a non-zero process exit.

That mix is why visual inspection fails. One casual rename can hit three observables. The golden tape must see all three.

Artifact: golden tape runner

The runner executes frozen fixtures against the sample. The first run writes tapes/cli_golden.json on disk. Later runs compare live output to that file.

This listing is a proposed local harness. Run it only against your frozen fixtures. Do not treat later hashes as measured product data.

#!/usr/bin/env python3
"""characterize_cli.py — proposed golden tape runner."""
from __future__ import annotations

import argparse
import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
CLI = ROOT / "sample_messy_cli.py"
TAPE = ROOT / "tapes" / "cli_golden.json"
FIXTURES = ROOT / "fixtures"


def _hash_file(path: Path) -> str:
    if not path.exists():
        return "missing"
    return hashlib.sha256(path.read_bytes()).hexdigest()


def _run_case(name: str, src: Path, out: Path) -> dict:
    out.unlink(missing_ok=True)
    proc = subprocess.run(
        [sys.executable, str(CLI), str(src), str(out)],
        capture_output=True,
        text=True,
        check=False,
    )
    return {
        "name": name,
        "exit_code": proc.returncode,
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "out_hash": _hash_file(out),
    }


def _cases() -> list[dict]:
    work = ROOT / ".tape_work"
    work.mkdir(exist_ok=True)
    results = []
    for src in sorted(FIXTURES.glob("*")):
        dest = work / f"{src.stem}.out"
        results.append(_run_case(src.name, src, dest))
    return results


def record() -> int:
    TAPE.parent.mkdir(parents=True, exist_ok=True)
    payload = {"cases": _cases()}
    TAPE.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
    print(f"wrote {TAPE}")
    return 0


def verify() -> int:
    if not TAPE.exists():
        print("missing tape; run --record first", file=sys.stderr)
        return 2
    expected = json.loads(TAPE.read_text(encoding="utf-8"))
    live = {"cases": _cases()}
    if live != expected:
        print("CLI tape drift")
        print(json.dumps({"expected": expected, "live": live}, indent=2, sort_keys=True))
        return 1
    print("CLI tape green")
    return 0


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--record", action="store_true")
    parser.add_argument("--verify", action="store_true")
    args = parser.parse_args()
    if args.record:
        return record()
    if args.verify:
        return verify()
    parser.print_help()
    return 2


if __name__ == "__main__":
    raise SystemExit(main())

Commands for the proposed harness follow.

mkdir -p fixtures tapes
printf 'alpha\nTODO beta\n' > fixtures/notes.txt
printf '%s\n' '{"rows":["# skip","ok"],"mode":"json"}' > fixtures/rows.json
: > fixtures/blank.txt
python characterize_cli.py --record
python characterize_cli.py --verify

notes.txt exercises the text path and TODO bonus. rows.json exercises JSON loading and comment skips. blank.txt exercises a zero score and exit 1.

Keep fixture bytes frozen after the record step. Commit the tape together with the fixtures. Never commit credential material inside either file.

A green verify means every observable matched. A red verify means some caller-visible field drifted. Do not interpret red as a style failure.

Numbered workflow

1. Inventory the entrypoint

Name the command that users already run. List the required files and flags next. Ignore private helpers during this inventory step.

Write that exact command into the runner. Do not change any CLI flags yet. The first tape must match behavior today.

2. Freeze fixtures on disk

Pick three to seven representative fixture inputs. Cover empty, normal, and warning paths explicitly. Store them under fixtures/ as real bytes.

Do not generate fixtures from live production secrets. Strip credentials before any tape record step. Committed tapes should stay safe to publish.

3. Record the golden tape

Run python characterize_cli.py --record as shown. Inspect the JSON tape before you commit. Confirm exit codes match known operator knowledge.

If an exit code surprises you, stop immediately. The surprise is a finding, not tape noise. Document that finding beside the tape file.

4. Prove replay on a clean tree

Delete every temporary output file before replay. Run python characterize_cli.py --verify next. Require a zero exit from the verify command.

Then change nothing and verify a second time. Two greens without edits show harness stability. Instability here means the fixtures are flaky.

5. Change one file only

Choose a single module for the edit. Prefer a leaf helper over the CLI entrypoint. Keep the public CLI flags untouched here.

Pick one concern as the whole patch. Extract _score into scorelib.py as one change. Or replace one magic constant inside that helper.

Do not also delete branches in the same patch. One concern stays one concern. The tape cannot explain a mixed diff.

Run verify immediately after that one file. If the tape is red, revert the file.

If the tape is green, stop the cycle. Start a new cycle only after review.

6. Refresh the tape only on purpose

Do not auto-accept every observed tape drift. Any observed drift counts as a behavior change. Refresh the tape only when the change is intended.

Record a one-line reason with the tape commit. Name the observable you meant to change. Leave every other recorded tape field identical.

Smallest safe change, in practice

Consider moving _score out of the script. That is one concern and one file. The CLI wrapper should stay stable under verify.

# scorelib.py — proposed extraction after the tape is green
def score(rows: list[str]) -> int:
    total = 0
    for row in rows:
        if row.startswith("#"):
            continue
        total += len(row)
        if "TODO" in row:
            total += 10
    return total

Wire the new import in sample_messy_cli.py only. Do not rename flags in the same patch. Do not reformat unrelated functions in the same patch.

Then run the verify command one more time. Green means the extraction preserved all observables. Red means the move altered scoring or IO.

That sequence is the whole refactor loop. Repeat the loop as tape, one file, verify. Start the next loop only after green.

Where a coding model belongs

A model is useful after the tape exists. It is not a substitute for the tape. Prompt it to touch one file, not the tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Use those only as a drafting surface for the one-file patch. If verify goes red, paste the tape diff into the prompt and keep the rest of the tree closed.

Do not ask the model to clean the whole repo. That prompt ignores the recorded CLI contract. Ask it to extract one helper and keep CLI output stable.

The local harness remains the only judge. The model only proposes a candidate patch. Keep or revert solely from the verify result.

Networked generation does not replace frozen fixture bytes. Local verify remains the only merge gate.

Decision table

Signal	Action	Stop condition
Tape missing	Record, then inspect JSON	Unknown exit codes
Verify green, no edit	Change exactly one file	Multi-file itch
Verify red after edit	Revert the file	Fix-forward urge
Intended behavior change	Update one fixture, re-record	Unrelated field drift
Model patch spans many files	Reject the patch	Calendar pressure

Read each table row from left to right. Do not skip the listed stop condition. The stop condition is the actual safety mechanism.

Limitations

This golden tape does not prove correctness. It only proves stability against frozen fixtures. Unseen inputs can still break the CLI.

Stdout timestamps will poison the golden tape. Non-deterministic hashes will also poison the tape. Live network calls will poison the tape.

Stub time, clocks, and network before recording anything. Hash only files the CLI is meant to write. Drop volatile log lines if you must, and document the drop.

The runner shown uses exact JSON equality checks. Exact equality stays strict for a reason. Looser matchers hide real observable tape drift.

This method also does not measure model quality. No latency numbers are claimed in this article. No product quota claims appear here either.

Who should not use this approach

Do not use this as a substitute for typed APIs. Library authors need unit tests around public functions. A CLI tape is too coarse for that.

Do not use this on systems with live side effects. Payment charges and emails are not golden-file safe. Add fakes first, or skip the method.

Do not use this to justify a rewrite. A green tape only permits a small edit. It does not permit a new architecture in one pass.

Teams without fixture hygiene should wait first. If inputs cannot be sanitized, do not record. A tape that contains secrets is an incident.

What to do next

Inventory one messy command on a local clone. Freeze three representative fixtures on local disk. Record the tape, extract one helper, then verify.

Stop at the first green one-file patch. That already is a complete refactor cycle.

Patch size is not the goal here. Observable stability is the only real goal.