Build a Flag Matrix Before You Extract a Resolver

Do not split a messy CLI parser first. Pin the flag matrix that callers already depend on. Then extract one resolver. Leave every other branch frozen.

A messy CLI is a behavior surface, not a cleanup target. Exit codes, stdout bytes, and stderr tokens are the contract. Helpers are not.

This workflow is a proposal. It uses an illustrative parser, not a live dump. Run it on a copy. Do not claim green until the matrix matches.

The failure that keeps repeating

AI diffs love argparse rewrites. They rename flags. They swap defaults. They merge env and file config in a new order.

Callers do not feel the new structure. They feel a changed port. They feel a silent default. They feel a missing error line.

Characterization tests freeze that surface. They do not freeze class names. They freeze combinations.

What you pin

Pin three outputs per combination. Exit code. SHA-256 of stdout. A short stderr token list.

Do not pin full stderr dumps if timestamps leak. Do not pin absolute paths. Do not pin help wrapping from argparse versions.

combo_id | argv              | env           | exit | stdout_sha | stderr_tokens
A1       | --port 8080       |               | 0    | <hash>     | []
A2       | --port 0          |               | 2    | <hash>     | ["invalid-port"]
A3       |                   | PORT=9090     | 0    | <hash>     | []
A4       | --port 8080       | PORT=9090     | 0    | <hash>     | []
A5       | --config bad.json |               | 2    | <hash>     | ["config-parse"]

That table is the artifact. Code later must reproduce it. A model patch later must not change it.

Step 1 — Inventory the real combinations

List flags that production scripts already pass. List env keys those scripts export. List config files they point at.

Keep the list short. Twelve to twenty rows beat two hundred. Cover defaults, overrides, and the known invalid cases.

Skip decorative flags with no callers. Skip help text. Skip version banners unless install scripts parse them.

rg -n --hidden -g '!node_modules' -- 'argparse|click|typer|sys.argv' .
rg -n -- 'os.environ|getenv|PORT|CONFIG' .

Record the current precedence in one sentence. Example: CLI flag beats env. Env beats config file. Config beats hardcoded default.

If nobody can state precedence, stop. You do not have a refactor. You have archaeology.

Step 2 — Isolate one entry command

Pick the command users actually run. Not an internal function. Not a test helper.

python -m messy_tools serve --port 8080

Wrap it so tests can inject argv and env. Do not import private classes yet. The process boundary is the oracle.

# illustrative harness, not executed against a private repo
import hashlib
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]

def run_combo(argv, env_extra):
    env = os.environ.copy()
    for key in ("PORT", "CONFIG_PATH"):
        env.pop(key, None)
    env.update(env_extra)
    proc = subprocess.run(
        [sys.executable, "-m", "messy_tools", *argv],
        cwd=ROOT,
        env=env,
        capture_output=True,
        text=True,
        check=False,
    )
    stdout_sha = hashlib.sha256(proc.stdout.encode()).hexdigest()
    tokens = sorted(
        t for t in ("invalid-port", "config-parse", "bind-failed")
        if t in proc.stderr
    )
    return proc.returncode, stdout_sha, tokens

Label this harness illustrative until you run it. First run writes the matrix. Later runs assert it.

Step 3 — Capture, then lock

First capture must use the messy code as-is. No formatter pass. No import sort. No “small cleanup”.

CASES = [
    ("A1", ["serve", "--port", "8080"], {}),
    ("A2", ["serve", "--port", "0"], {}),
    ("A3", ["serve"], {"PORT": "9090"}),
    ("A4", ["serve", "--port", "8080"], {"PORT": "9090"}),
    ("A5", ["serve", "--config", "fixtures/bad.json"], {}),
]

def test_flag_matrix_matches_lockfile(tmp_path):
    lock = ROOT / "tests" / "locks" / "serve_flag_matrix.json"
    observed = {}
    for case_id, argv, env_extra in CASES:
        code, sha, tokens = run_combo(argv, env_extra)
        observed[case_id] = {
            "exit": code,
            "stdout_sha": sha,
            "stderr_tokens": tokens,
        }
    if not lock.exists():
        lock.write_text(__import__("json").dumps(observed, indent=2) + "\n")
        raise AssertionError("lockfile written; rerun to assert")
    expected = __import__("json").loads(lock.read_text())
    assert observed == expected

Commit the lockfile alone. That commit is the characterization baseline. Reviewers can read rows. They cannot read intent from a 900-line parser.

Step 4 — Prove one override rule

Add a focused assertion for the row that usually breaks. Flag versus env is the usual fight.

def test_cli_port_beats_env():
    code, sha_flag, _ = run_combo(["serve", "--port", "8080"], {"PORT": "9090"})
    code_env, sha_env, _ = run_combo(["serve"], {"PORT": "9090"})
    assert code == 0 and code_env == 0
    assert sha_flag != sha_env

If those hashes match, the flag is dead. Stop extracting helpers. Fix the contract first, or document the dead flag.

Step 5 — Name the smallest safe change

After the matrix is green, pick one resolver. One function. One return value. No file move. No package rename.

Good change: extract resolve_port(argv_ns, env, file_cfg) -> int. Bad change: rewrite the parser class. Bad change: switch argparse to click. Bad change: “clean the module”.

Write the target signature before any model sees the file.

def resolve_port(cli_port, env_port, file_port, default=8080):
    """CLI int beats env int beats file int beats default."""
    for raw in (cli_port, env_port, file_port, default):
        if raw is None or raw == "":
            continue
        port = int(raw)
        if port < 1 or port > 65535:
            raise ValueError("invalid-port")
        return port
    raise ValueError("invalid-port")

Keep the old call site wired. Do not delete branches in the same patch. The matrix must stay green with only this function new.

Where a free coding model can enter

The matrix is the gate. A model does not replace it. A model may draft the one-function extract after the lockfile exists.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use those only as a patch draft channel. Do not treat them as a test runner for your lockfile.

Send the resolver signature. Send the messy function body. Send three failing-adjacent rows from the matrix. Do not send the whole repo. Do not ask for a framework swap.

Paste the returned function into a new module. Keep the CLI entrypoint untouched. Re-run the matrix locally. Reject the patch if any row moves.

If you already keep this matrix in CI, a free remote session can draft that single helper. Skip the session while the lockfile is still missing.

Step 6 — Re-run, then stop

pytest tests/test_serve_flag_matrix.py -q

Green means the extract preserved the surface. It does not mean the parser is clean. Stop the patch there.

A second extract needs a second commit. New helper. Same lockfile. Same rows. No bonus rename.

If stdout hashes drift, print a hexdump of the two streams. Most drifts are trailing newlines. Some are log timestamps. Fix the harness, not the parser, when the bytes are noise.

python -m messy_tools serve --port 8080 | sha256sum
PORT=9090 python -m messy_tools serve | sha256sum

Decision table for the extract

Use this table before you open an editor.

Signal                         | Extract now? | Why
Lockfile missing               | No           | No oracle
Help text changed              | No           | Unstable surface
One resolver, same lock hashes | Yes          | Smallest safe change
Parser library swap            | No           | Contract rewrite
Env/flag hashes now equal      | No           | Precedence bug
New flag added in same patch   | No           | Mixed intent

If two columns say No, do not negotiate. The extract waits.

Limitations

This matrix does not see bind timing. It does not see SO_REUSEADDR. It does not see IPv6 versus IPv4.

It misses config files outside the fixture set. It misses locale-dependent number parsing. It misses Windows path quoting.

SHA-256 of stdout is brittle on logs. Prefer structured stdout. If the CLI prints banners, strip them in the harness before hashing.

Subprocess tests are slow under large env copies. Keep the row count small. Do not turn this into a fuzzer in the same patch.

A free model can invent a prettier precedence. Pretty is not pinned. The lockfile is pinned. Discard style-only diffs.

Who should not use this

Do not use this on a greenfield CLI with zero callers. Write real unit tests. Do not snapshot accidents.

Do not use this for secret flags. Hashes of stdout can leak tokens. Redact. Keep secrets out of lockfiles.

Do not use this when the CLI must change behavior. Characterization freezes old bugs. A planned break needs new rows, not a silent extract.

Do not use this as permission to delete tests. The matrix is a fence. It is not a specification of good design.

What “done” means

Done is a lockfile commit plus one resolver commit. Both keep the same hashes. Reviewers can read the table in under a minute.

The messy parser can stay messy. Callers keep their flags. You changed one function. That is the refactor.