Pin Exit Codes Before You Extract One CLI Flag

Do not extract a CLI flag from messy code first. Freeze the exit codes and stderr tokens first. Characterization tests make that freeze cheap and repeatable.

Messy CLIs fail refactors in three silent ways. Exit codes drift while callers keep old scripts. Stderr tokens vanish when a logger gets cleaned.

Stdout JSON can also shuffle keys after a tiny helper extract. Scripts that parse both streams then fail in CI only. You need a contract that is smaller than the full transcript.

The contract that matters

A CLI contract is not the source layout. It is the process boundary callers already depend on.

Pin these four process fields before any extract.

1. Record the process exit code for each fixture input.
2. Record stable stderr tokens, not full log lines.
3. Record stdout JSON keys, types, and null policy.
4. Record presence or absence of extra files on disk.

The table below is a working decision grid. Fill it from captured runs, not from memory.

Input fixture	Exit	Stderr tokens	Stdout keys	Extra files
mixed.csv	0	wrote, rows	errors, rows	none
empty.csv	2	empty, input	(no json)	none
bad.csv	3	parse, failed	errors	none
missing path	2	not, found	(no json)	none

Do not pin timestamps, PIDs, or absolute paths. Those values still move without a behavior change. Tokenize stderr and strip the volatile fields first.

Choose fixtures from callers, not functions

Read the scripts that wrap the CLI today. Copy each wrapper argument list into fixture files. Ignore internal functions until the gold stays green.

1. Take one success path from a wrapper script.
2. Take one empty-file path from a cron job.
3. Take one parse-failure path from a support ticket.
4. Take one missing-path path from a shell check.

Four fixtures beat twenty shallow ones in review. Each fixture must change at least one gold field. Duplicate exits with identical tokens waste review time.

Artifact: a recorder, then tests

The listing below is a self-contained example, not a production trace. It records exit codes, stdout bytes, and stderr tokens. Run it against the current messy binary before edits.

# record_cli.py
# Proposal: capture process-boundary contracts for one messy CLI.
from __future__ import annotations

import json
import os
import re
import subprocess
from pathlib import Path

TOKEN_RE = re.compile(r"[A-Za-z]{3,}")
SKIP_TOKENS = {"pid", "elapsed", "timestamp", "debug"}


def tokenize(stderr: str) -> list[str]:
    words = [w.lower() for w in TOKEN_RE.findall(stderr)]
    return [w for w in words if w not in SKIP_TOKENS]


def run_case(bin_argv: list[str], cwd: Path) -> dict:
    proc = subprocess.run(
        bin_argv,
        cwd=cwd,
        capture_output=True,
        text=True,
        env={**os.environ, "TZ": "UTC", "PYTHONHASHSEED": "0"},
    )
    stdout = proc.stdout.strip()
    keys: list[str] | None
    if stdout:
        payload = json.loads(stdout)
        keys = sorted(payload.keys()) if isinstance(payload, dict) else ["<non-object>"]
    else:
        keys = None
    return {
        "argv": bin_argv[1:],
        "exit": proc.returncode,
        "stderr_tokens": tokenize(proc.stderr),
        "stdout_keys": keys,
    }


def main() -> None:
    root = Path(__file__).resolve().parent
    cases = [
        ["python", "legacy_report.py", "fixtures/mixed.csv"],
        ["python", "legacy_report.py", "fixtures/empty.csv"],
        ["python", "legacy_report.py", "fixtures/bad.csv"],
        ["python", "legacy_report.py", "fixtures/missing.csv"],
    ]
    gold = [run_case(argv, root) for argv in cases]
    Path("cli_gold.json").write_text(json.dumps(gold, indent=2) + "\n")
    print(f"wrote {len(gold)} cases to cli_gold.json")


if __name__ == "__main__":
    main()

Commit the captured gold file into git next. Treat it as the error contract for the process. Do not pretty-print it by hand later.

Turn that gold file into tests next. The test must fail on exit drift. It must also fail on missing stderr tokens.

# test_cli_contract.py
# Proposal: characterization tests. Do not assert on source shape.
from __future__ import annotations

import json
from pathlib import Path

from record_cli import run_case

GOLD = json.loads(Path("cli_gold.json").read_text())
ROOT = Path(__file__).resolve().parent


def test_exit_codes_match_gold() -> None:
    for case in GOLD:
        argv = ["python", "legacy_report.py", *case["argv"]]
        got = run_case(argv, ROOT)
        assert got["exit"] == case["exit"], (case["argv"], got["exit"])


def test_stderr_tokens_cover_gold() -> None:
    for case in GOLD:
        argv = ["python", "legacy_report.py", *case["argv"]]
        got = run_case(argv, ROOT)
        missing = set(case["stderr_tokens"]) - set(got["stderr_tokens"])
        assert not missing, (case["argv"], missing)


def test_stdout_keys_match_gold() -> None:
    for case in GOLD:
        argv = ["python", "legacy_report.py", *case["argv"]]
        got = run_case(argv, ROOT)
        assert got["stdout_keys"] == case["stdout_keys"], case["argv"]

These tests are characterization tests, not design tests. They lock observed behavior at the process boundary. They do not bless the current module shape.

Sample messy module under test

The snippet below is labeled as an example. It mixes parsing, logging, and exit codes. That mix is why the flag extract waits.

# legacy_report.py
# Example only. Do not treat this as production code.
import json
import sys
from pathlib import Path


def main(argv: list[str]) -> int:
    if not argv:
        print("usage: report <file>", file=sys.stderr)
        return 2
    path = Path(argv[0])
    if not path.exists():
        print("not found: input path", file=sys.stderr)
        return 2
    text = path.read_text()
    if not text.strip():
        print("empty input: no rows", file=sys.stderr)
        return 2
    rows = []
    errors = []
    for line in text.splitlines():
        parts = line.split(",")
        if len(parts) < 2:
            errors.append(line)
            continue
        rows.append({"id": parts[0], "n": parts[1]})
    if errors and not rows:
        print("parse failed: bad csv", file=sys.stderr)
        print(json.dumps({"errors": errors}))
        return 3
    print("wrote rows to stdout", file=sys.stderr)
    print(json.dumps({"rows": rows, "errors": errors}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))

The new flag extract should not touch those print calls. It should only split the argument vector. Process boundary tests make that extract limit visible.

Workflow: tests first, one flag second

You must follow these steps in listed order. Skip a step and the extract will lie.

1. Inventory every caller that reads exit codes today.
2. Add fixtures that cover zero, empty, and parse failure.
3. Run the recorder script on an unchanged tree.
4. Commit cli_gold.json with no code edits yet.
5. Add the contract tests and watch them pass.
6. Draft extra cases only from new traces.
7. Extract one flag after the suite is green.
8. Re-run the suite and stop if gold moves.

Step six is where a coding model can help. It should not rewrite the messy module yet.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Point the model at traces and the gold file only.

Ask the model for missing fixtures, not for a new architecture. Paste the gold file and one failing command. Require every new test to replay run_case.

Discard assertions that mention functions you plan to extract. Keep the model away from helper names.

Dirty local shells inject extra environment variables into runs. Those variables change logs and sometimes exit codes. When the tree looks dirty, use a free server.

The server job must stay boring and deterministic. Use the same fixtures and frozen hash seed. Compare cli_gold.json byte for byte after the run.

Read a failed characterization before you edit

A red test is a map, not a prompt. You should read the failing assertion name first. Then inspect only that one gold field.

If exit codes differ, stop the extract now. Outside callers still branch on those exit numbers. Restore the original mapping before any other fix.

If stderr tokens are missing, search the messy module for that word. You likely rewrote a log line during extract. Put the missing token back and then retry.

If stdout keys differ, print both key lists. A helper that drops nulls will fail this check. Keep the null policy in the original printer.

Do not regenerate gold to silence the suite. Fresh gold files hide the regression from reviewers. Only regenerate when the product owner changes the contract.

The smallest safe change

Only one behavior may move, and it must be additive. Extract a strict flag that currently lives inline. Do not rename stderr tokens in the same patch.

# Proposal: smallest extract after gold is pinned.
# legacy_report.py still owns printing and exit codes.

def parse_strict(argv: list[str]) -> tuple[bool, list[str]]:
    strict = False
    rest: list[str] = []
    for item in argv:
        if item == "--strict":
            strict = True
            continue
        rest.append(item)
    return strict, rest

Wire parse_strict in one call site only. Leave JSON printing in the original function body. Leave exit code mapping in the original function.

Re-run the characterization suite after that extract. Exit codes must match the gold file exactly. Stderr token sets must still cover gold.

Stdout keys must stay sorted and equal. Any drift means the extract was too large.

If strict is new behavior, add a new gold case. Do not edit old cases to make the extract pass. Old callers did not pass that flag before.

Keep the commands below in the pull request.

export TZ=UTC PYTHONHASHSEED=0
python record_cli.py
python -m pytest test_cli_contract.py -q
git diff -- cli_gold.json

The gold diff must be empty for old fixtures. A non-empty diff means the extract was not smallest. Revert the extract and split the patch.

What the model must not do

A free model will offer a parser rewrite. It will also offer a logging cleanup. Both edits move stderr tokens and exit codes together.

Reject every combined diff from the model. Characterization tests exist to make that rejection cheap. If the model output changes gold, the output is out of scope.

Also reject tests that import those private helpers. Process boundary tests should spawn the real CLI. Helper tests come after the extract, not before.

Decision: extract or stop

Use this table after the suite runs. Do not negotiate the table rows during review.

Observation	Action
gold diff empty, tests green	Extract one flag, then stop
exit code changed	Revert extract, restore mapping
stderr token missing	Restore the log token, rerun
stdout keys moved	Keep printer in original function
model rewrote three files	Discard the diff, retry tests-only
new flag needs a new case	Append gold, do not edit old rows

The table is the review script for this patch. Paste the table in the pull request. Reviewers then check one row, not the whole module.

Limitations

This workflow does not freeze clocks or time-dependent reports. Freeze clocks or drop time tokens first. It does not freeze byte-for-byte log lines.

It does not replace tests for new flags. New behavior needs intent tests after gold. It does not prove thread safety or signal handling.

Loose tokenization can hide a real behavior regression. A swapped word order may still pass. Add a short ordered-token check for fatal paths if needed.

def test_fatal_token_order() -> None:
    case = next(c for c in GOLD if c["exit"] == 3)
    argv = ["python", "legacy_report.py", *case["argv"]]
    got = run_case(argv, ROOT)
    # Ordered prefix on the fatal path only.
    assert got["stderr_tokens"][:2] == case["stderr_tokens"][:2]

Gold files rot as soon as fixtures move. Keep fixtures in the same commit as gold. Do not regenerate gold in CI on main.

A free server does not match every laptop filesystem. You still need to strip path tokens. If your CLI writes absolute paths, normalize them in the recorder.

Who should skip this

Skip this if you do not have a runnable CLI. Skip this if callers never read exit codes. Skip this if the tool is still private and unreleased.

Do not use characterization tests to bless a rewrite. They only tell you what callers already survived. Greenfield CLIs need intent tests, not gold files.

Do not send secrets to any model. Strip tokens, hostnames, and file contents before prompts. The gold file should hold keys and tokens only.

Close

Pin the process contract before the code extract. Pin exit codes first and stderr tokens second. Extract one flag only after those pins hold.

The test suite is the refactor permission slip. No green gold means no extract today. Keep free models away from the rewrite itself.

Try this first on one messy reporter. Start with four fixtures and an empty gold diff.