Record Four CLI Channels Before You Change One Flag

Do not refactor a messy CLI as the first move.
Freeze four output channels into a ledger first.
Then change one flag or one helper function.

A larger patch is an untested behavior rewrite.
Argv parsing, env reads, and file writes hide contracts.
Unit tests on internal functions miss those contracts.

Why four channels, not one assertion

Stdout is not the whole CLI contract.
Stderr, exit codes, and file writes also bind users.
Scripts, CI jobs, and operators depend on all four.

A green unit test can still break a cron wrapper.
That wrapper may key off exit code two.
It may also parse a warning line on stderr.

The ledger in one sentence

A CLI ledger is a JSON record per fixture.
Each record stores inputs, outputs, and a tree hash.
Inputs are argv, env subset, stdin, and cwd files.

Outputs are exit code, stdout, stderr, and output files.
The tree hash covers only files the command may write.
Ignore caches, pyc files, and unrelated temp noise.

Worked example: a messy invoice CLI

The following script is a labeled example, not production code.
It parses flags by hand and writes a side-effect file.
Treat it as the brownfield module under freeze.

#!/usr/bin/env python3
"""invoice_cli.py — messy on purpose. Example only."""
import os
import sys
from pathlib import Path


def main(argv):
    currency = os.environ.get("INVOICE_CURRENCY", "USD")
    if argv and argv[0] in ("-h", "--help"):
        sys.stdout.write("usage: invoice_cli.py TOTAL [TAX]\n")
        return 0
    if argv and argv[0] in ("-v", "--verbose"):
        sys.stderr.write("verbose: parsing totals\n")
        argv = argv[1:]
    if len(argv) < 1:
        sys.stderr.write("error: missing TOTAL\n")
        return 2
    try:
        total = float(argv[0])
        tax = float(argv[1]) if len(argv) > 1 else 0.0
    except ValueError:
        sys.stderr.write("error: TOTAL and TAX must be numbers\n")
        return 3
    grand = round(total * (1.0 + tax), 2)
    line = f"{currency} {grand:.2f}\n"
    sys.stdout.write(line)
    Path("invoice.out").write_text(line, encoding="utf-8")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))

This CLI has six observable behaviors worth freezing.
Help, missing TOTAL, bad numbers, tax paths, and verbose mode.
Currency comes from the environment, not from argv.

Spot-check one path before recording the full matrix.

python3 invoice_cli.py 100 0.1
# expected shape from this example: stdout "USD 110.00\n"

Step 1: List the fixture matrix

Write the matrix before any model drafts a patch.
Keep the set small and high-signal on purpose.
Six to ten rows beat a hundred vague tests.

id	argv	env	stdin	expect_exit
help	--help	{}	empty	0
missing	(none)	{}	empty	2
badnum	abc	{}	empty	3
default_tax	100	{}	empty	0
tax	100 0.1	{}	empty	0
verbose	-v 100	{}	empty	0
eur	100	INVOICE_CURRENCY=EUR	empty	0

Each row becomes one golden JSON file.
Do not generate expected stdout from memory.
Run the current binary and store what it emits.

Step 2: Record the four channels

The runner below is a proposal you can execute locally.
It isolates cwd, captures streams, and hashes invoice.out.
It does not mock the filesystem inside the CLI.

#!/usr/bin/env python3
"""record_ledger.py — proposal for a CLI characterization runner."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

FIXTURES = [
    {"id": "help", "argv": ["--help"], "env": {}, "stdin": ""},
    {"id": "missing", "argv": [], "env": {}, "stdin": ""},
    {"id": "badnum", "argv": ["abc"], "env": {}, "stdin": ""},
    {"id": "default_tax", "argv": ["100"], "env": {}, "stdin": ""},
    {"id": "tax", "argv": ["100", "0.1"], "env": {}, "stdin": ""},
    {"id": "verbose", "argv": ["-v", "100"], "env": {}, "stdin": ""},
    {"id": "eur", "argv": ["100"], "env": {"INVOICE_CURRENCY": "EUR"}, "stdin": ""},
]


def hash_tree(root: Path, names: list[str]) -> str:
    h = hashlib.sha256()
    for name in sorted(names):
        path = root / name
        payload = path.read_bytes() if path.exists() else b""
        h.update(name.encode())
        h.update(payload)
    return h.hexdigest()


def run_one(cli: Path, fix: dict, work: Path) -> dict:
    env = os.environ.copy()
    env.update(fix["env"])
    proc = subprocess.run(
        [sys.executable, str(cli), *fix["argv"]],
        cwd=work,
        input=fix["stdin"].encode(),
        capture_output=True,
        env=env,
        check=False,
    )
    out_file = work / "invoice.out"
    return {
        "id": fix["id"],
        "argv": fix["argv"],
        "env": fix["env"],
        "stdin": fix["stdin"],
        "exit_code": proc.returncode,
        "stdout": proc.stdout.decode("utf-8", "replace"),
        "stderr": proc.stderr.decode("utf-8", "replace"),
        "fs_hash": hash_tree(work, ["invoice.out"]),
        "invoice_out": out_file.read_text("utf-8") if out_file.exists() else None,
    }


def main() -> int:
    cli = Path("invoice_cli.py").resolve()
    out_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "ledger")
    out_dir.mkdir(exist_ok=True)
    for fix in FIXTURES:
        work = Path("work") / fix["id"]
        work.mkdir(parents=True, exist_ok=True)
        for leftover in work.glob("*"):
            leftover.unlink()
        record = run_one(cli, fix, work)
        (out_dir / f"{fix['id']}.json").write_text(
            json.dumps(record, indent=2, sort_keys=True) + "\n",
            encoding="utf-8",
        )
        print(f"wrote {out_dir}/{fix['id']}.json exit={record['exit_code']}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it once against the unpatched CLI binary.

python3 record_ledger.py ledger
git add ledger/*.json invoice_cli.py record_ledger.py
git commit -m "freeze cli ledger before parse_args extract"

That commit is the freeze, with no refactor inside.
The golden files are the oracle, not a chat transcript.

Expected shape for the tax row in this example:

{
  "id": "tax",
  "argv": ["100", "0.1"],
  "env": {},
  "stdin": "",
  "exit_code": 0,
  "stdout": "USD 110.00\n",
  "stderr": "",
  "invoice_out": "USD 110.00\n"
}

Replace any hand-waved hash with the value the recorder writes.
Do not paste a hash from memory into the golden file.

Step 3: Compare with an allowed-diff table

Characterization without a policy still invites noisy diffs.
Decide which channels may move for this change.
Put the policy in a table, not in review comments.

Channel	Freeze commit	One-flag change	Parser extract
stdout bytes	lock	lock	lock
stderr bytes	lock	lock	lock
exit code	lock	lock	lock
fs_hash	lock	lock	lock
invoice.out body	lock	lock	lock
help text	lock	may change if the flag is help	lock
source of parse_args	free	free	expected to change

The compare script should fail closed on drift.
Any unlocked channel still needs an explicit allow list.
Silence is not permission to change a channel.

#!/usr/bin/env python3
"""compare_ledger.py — fail closed unless a channel is allowed."""
from __future__ import annotations

import json
import sys
from pathlib import Path

# Proposal: this extract allows no channel drift.
ALLOWED = set()


def load(dir_path: Path) -> dict:
    records = {}
    for path in sorted(dir_path.glob("*.json")):
        records[path.stem] = json.loads(path.read_text(encoding="utf-8"))
    return records


def main() -> int:
    golden = load(Path("ledger"))
    actual = load(Path("ledger_actual"))
    failed = 0
    keys = ("exit_code", "stdout", "stderr", "fs_hash", "invoice_out")
    for fid, old in golden.items():
        new = actual.get(fid)
        if new is None:
            print(f"FAIL {fid}: missing actual record")
            failed += 1
            continue
        for key in keys:
            if key in ALLOWED:
                continue
            if old.get(key) != new.get(key):
                print(f"FAIL {fid}.{key}")
                failed += 1
    print(f"failures={failed}")
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Re-run the recorder into ledger_actual after the patch.

python3 record_ledger.py ledger_actual
python3 compare_ledger.py

Zero failures is the only merge signal that matters.
Review comments are not a substitute for that exit code.

Hashing rules that keep the ledger stable

Hash file names and bytes, not mtime or inode.
Sort names so the hash does not depend on readdir order.
Missing files hash as empty bytes, not as a crash.

Do not hash the entire work directory by default.
Name the output paths in a short allow list.
Expand that list only when the CLI grows a new artifact.

Step 4: Make the smallest source change

The first allowed edit is a parser extract.
Do not add flags or change money formatting.
Move argv handling into parse_args and keep main stable.

def parse_args(argv):
    verbose = False
    if argv and argv[0] in ("-v", "--verbose"):
        verbose = True
        argv = argv[1:]
    return verbose, argv


def main(argv):
    currency = os.environ.get("INVOICE_CURRENCY", "USD")
    if argv and argv[0] in ("-h", "--help"):
        sys.stdout.write("usage: invoice_cli.py TOTAL [TAX]\n")
        return 0
    verbose, argv = parse_args(argv)
    if verbose:
        sys.stderr.write("verbose: parsing totals\n")
    if len(argv) < 1:
        sys.stderr.write("error: missing TOTAL\n")
        return 2
    # remainder unchanged from the freeze commit

That edit is one behavior-preserving structural change.
The ledger must stay byte-identical on every fixture.
If help text drifts, the change is already too large.

Step 5: Only then consider a flag change

A later change may add a currency flag.
Update the allowed-diff table before that patch.
Help stdout may change under an explicit allow rule.

Every other fixture must remain byte-identical after it.
Add one new ledger row for the currency flag.
Keep the INVOICE_CURRENCY environment row fully locked.

Two currency sources are a product decision, not a surprise.
Record the decision in the fixture matrix first.
Then implement the flag against that new row only.

What a failed ledger row means

A red compare run is diagnostic data, not a style debate.
Read the failing key before you edit source again.
The key name tells you which contract moved.

1. Open golden and actual JSON for the same fixture id.
2. Compare exit_code first because parsers often shift it.
3. Byte-diff stdout with cmp or git diff --no-index.
4. Byte-diff stderr next; verbose flags leak on that channel.
5. Compare fs_hash when streams match but a file path changed.
6. If only help text moved, the patch already exceeded the budget.

cmp -l ledger/tax.json ledger_actual/tax.json
git diff --no-index ledger/verbose.json ledger_actual/verbose.json

Do not loosen ALLOWED to silence a surprise key.
Loosen a channel only when the change list names it.
Then add or refresh the one fixture that should move.

Where a free coding model fits

A model is useful after the ledger exists, not before.
It can draft the recorder and the parse_args extract.
It must not invent expected stdout, stderr, or exit codes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.
Those two facts are the only product claims used here.

The free server option is a disposable place to run the ledger.
Keep golden files in git and keep secrets out of fixture env.
Do not upload production invoices or customer totals.

If the recorder needs a clean machine, the free server option can run it.
Ask the model for a patch smaller than the test surface.
Reject any diff that touches more than parse-and-dispatch code.

If the model rewrites rounding, the ledger should fail.
Byte identity is the review tool, not the model summary.

Limitations

This method does not prove functional correctness at all.
It proves the current messy behavior did not drift.
Wrong rounding stays wrong until you add a true oracle.

It is weak against time, randomness, and network calls.
Clock stamps, UUIDs, and DNS errors break hashes.
Stub those sources or exclude them from the tree hash.

It is also weak against unbounded file trees.
Hash only the paths the CLI is documented to write.
A recursive hash of cwd will flake on extra files.

Parallel tests need separate work directories per fixture.
Shared invoice.out in one cwd is a race.
The runner isolates by fixture id for that reason.

Who should not use this

Do not use a CLI ledger as a substitute for typed APIs.
Library refactors need import-surface tests, not argv fixtures.
GUI apps need a different oracle than stdout bytes.

Do not use it when the CLI must change behavior now.
A lock on stdout will fight a required format migration.
Split the work: freeze, extract, then change with a new row.

Do not feed the model production data as stdin fixtures.
Redact amounts, names, and hostnames before recording.
A characterization suite is still a data store.

Close

Freeze argv, env, stdin, stdout, stderr, exit, and side effects.
Commit that ledger before any structural edit lands.
Then extract one function or add one flag, not both.

If you run the example, keep the JSON ledger in git.
Models stay optional once those golden files exist.