Start at the process edge
Tangled CLIs punish function tests with no seams. The process edge still tells a stable story.
Record argv, stdout, stderr, and exit codes. Keep those golden files green during every extract.
Unit tests can wait until a pure seam exists. The binary contract is the first honest oracle.
Why function tests fail on day one
A messy repo hides I/O inside module import. Constructors open files, sockets, and log handles.
Function tests then boot the entire side-effect graph. The first assertion never reaches the target line.
Process-level pins skip that import graph entirely. They treat the script as a black-box command.
That match is what operators actually observe. Refactors that preserve it are user-visible safe.
Four fields worth pinning
Pin four fields for every fixture case.
- The argv vector after the interpreter.
- Raw stdout bytes, not decoded text.
- Raw stderr bytes, including warnings.
- The integer process exit code.
Store each field as a separate golden file. Bytes beat unicode snapshots for encoding bugs.
Name files from a stable fixture identifier. Never name golden files from wall-clock timestamps.
Artifact: a record and replay harness
The harness below is a labeled worked example. It is not a production test framework.
Place edgepin.py next to the messy script. Keep fixtures in goldens/ as committed bytes.
Restore writable inputs before every subprocess. Import caches die with each subprocess run.
#!/usr/bin/env python3
"""Process-edge golden recorder. Example only."""
from __future__ import annotations
import argparse
import hashlib
import json
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent
GOLDEN_DIR = ROOT / "goldens"
SEED = ROOT / "inventory.seed.json"
DB = ROOT / "inventory.json"
def fixture_id(argv: list[str]) -> str:
blob = json.dumps(argv, separators=(",", ":")).encode("utf-8")
return hashlib.sha256(blob).hexdigest()[:12]
def restore_seed() -> None:
if SEED.exists():
DB.write_bytes(SEED.read_bytes())
def run_target(target: str, argv: list[str], stdin: bytes) -> dict:
restore_seed()
proc = subprocess.run(
[sys.executable, target, *argv],
input=stdin,
capture_output=True,
cwd=ROOT,
)
return {
"argv": argv,
"returncode": proc.returncode,
"stdout": proc.stdout,
"stderr": proc.stderr,
}
def write_golden(fid: str, result: dict) -> None:
case = GOLDEN_DIR / fid
case.mkdir(parents=True, exist_ok=True)
(case / "meta.json").write_text(
json.dumps(
{"argv": result["argv"], "returncode": result["returncode"]},
indent=2,
),
encoding="utf-8",
)
(case / "stdout.bin").write_bytes(result["stdout"])
(case / "stderr.bin").write_bytes(result["stderr"])
def read_golden(fid: str) -> dict:
case = GOLDEN_DIR / fid
meta = json.loads((case / "meta.json").read_text(encoding="utf-8"))
return {
"argv": meta["argv"],
"returncode": meta["returncode"],
"stdout": (case / "stdout.bin").read_bytes(),
"stderr": (case / "stderr.bin").read_bytes(),
}
def compare(fid: str, result: dict) -> list[str]:
gold = read_golden(fid)
diffs: list[str] = []
if result["returncode"] != gold["returncode"]:
diffs.append(f"exit {result['returncode']} != {gold['returncode']}")
if result["stdout"] != gold["stdout"]:
diffs.append("stdout bytes differ")
if result["stderr"] != gold["stderr"]:
diffs.append("stderr bytes differ")
return diffs
def verify_all(target: str) -> int:
failed = 0
for meta in sorted(GOLDEN_DIR.glob("*/meta.json")):
fid = meta.parent.name
gold = read_golden(fid)
result = run_target(target, gold["argv"], b"")
diffs = compare(fid, result)
if diffs:
print("DRIFT", fid, "; ".join(diffs))
failed += 1
else:
print("OK", fid)
return 1 if failed else 0
def main() -> int:
parser = argparse.ArgumentParser(description="Pin CLI process edges")
parser.add_argument("mode", choices=["record", "verify"])
parser.add_argument("--target", default="inventory_cli.py")
parser.add_argument("--stdin-file", default="")
parser.add_argument("--all", action="store_true")
parser.add_argument("argv", nargs=argparse.REMAINDER)
args = parser.parse_args()
argv = args.argv[1:] if args.argv[:1] == ["--"] else args.argv
stdin = Path(args.stdin_file).read_bytes() if args.stdin_file else b""
GOLDEN_DIR.mkdir(exist_ok=True)
if args.all:
if args.mode != "verify":
print("--all only valid with verify", file=sys.stderr)
return 2
return verify_all(args.target)
fid = fixture_id(argv)
result = run_target(args.target, argv, stdin)
if args.mode == "record":
write_golden(fid, result)
print(f"recorded {fid} exit={result['returncode']}")
return 0
if not (GOLDEN_DIR / fid / "meta.json").exists():
print(f"missing golden {fid}", file=sys.stderr)
return 2
diffs = compare(fid, result)
if diffs:
print("DRIFT", fid, "; ".join(diffs))
return 1
print("OK", fid)
return 0
if __name__ == "__main__":
raise SystemExit(main())
A messy target worth pinning
The script below is tangled on purpose. Import time mutates a module-level path cache.
Formatting and filesystem writes share one function. That mix is the extract hazard.
#!/usr/bin/env python3
"""Messy inventory CLI. Example subject, not production code."""
from __future__ import annotations
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent
DB = ROOT / "inventory.json"
CACHE = {"loaded": False, "rows": []}
def _boot() -> None:
if CACHE["loaded"]:
return
if not DB.exists():
DB.write_text("[]", encoding="utf-8")
CACHE["rows"] = json.loads(DB.read_text(encoding="utf-8"))
CACHE["loaded"] = True
_boot()
def fmt_sku(sku: str, qty: int) -> str:
return f"{sku.upper()}:{qty:04d}"
def main(argv: list[str]) -> int:
if not argv or argv[0] in {"-h", "--help"}:
sys.stdout.write("usage: inventory_cli.py list|add <sku> <qty>\n")
return 0
cmd = argv[0]
if cmd == "list":
for row in CACHE["rows"]:
sys.stdout.write(fmt_sku(row["sku"], row["qty"]) + "\n")
return 0
if cmd == "add" and len(argv) == 3:
sku, qty_s = argv[1], argv[2]
try:
qty = int(qty_s)
except ValueError:
sys.stderr.write("qty must be int\n")
return 2
CACHE["rows"].append({"sku": sku, "qty": qty})
DB.write_text(json.dumps(CACHE["rows"]), encoding="utf-8")
sys.stdout.write(fmt_sku(sku, qty) + "\n")
return 0
sys.stderr.write("unknown command\n")
return 1
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
Commit inventory.seed.json beside the script. The harness copies it over inventory.json.
printf '%s\n' '[{"sku":"aa","qty":3},{"sku":"bb","qty":10}]' > inventory.seed.json
cp inventory.seed.json inventory.json
Numbered workflow
1. Freeze the entry file
Do not rename the script on day one. Do not move files during the first pin.
Commit inventory_cli.py as the frozen target. The harness must call that exact path.
2. Freeze locale and clock sources
CLI output often depends on cwd and env. Unpinned locales change date and number formats.
Export a fixed locale before record and verify.
export LC_ALL=C.UTF-8
export TZ=UTC
export PYTHONHASHSEED=0
Do not pin user home directories in goldens. Fail the harness if HOME leaks into stdout.
# Labeled example: add this guard after capture.
FORBIDDEN = (str(Path.home()).encode("utf-8"),)
edge = result["stdout"] + result["stderr"]
if any(part in edge for part in FORBIDDEN):
raise SystemExit("home path leaked into process edge")
3. Collect real invocations
Mine shell history for actual command lines. Prefer production argv over imagined happy paths.
Cover help, list, add, and one error. Four fixtures beat twenty synthetic unit cases.
Pass target argv after a bare --. That keeps --help off the harness parser.
python3 edgepin.py record --target inventory_cli.py -- --help
python3 edgepin.py record --target inventory_cli.py -- list
python3 edgepin.py record --target inventory_cli.py -- add zz 7
python3 edgepin.py record --target inventory_cli.py -- add zz no
python3 edgepin.py record --target inventory_cli.py -- boom
Each record writes one hashed golden directory. Inspect goldens/*/meta.json before the refactor.
4. Verify on a clean tree
Replay must start from the committed seed. The harness restores that seed per process.
python3 edgepin.py verify --target inventory_cli.py -- --help
python3 edgepin.py verify --target inventory_cli.py -- list
python3 edgepin.py verify --target inventory_cli.py -- boom
python3 edgepin.py verify --all --target inventory_cli.py
A non-zero harness exit means drift. Do not extract code while drift exists.
A single verify command reduces operator error. Reset still happens inside each spawn.
5. Classify each golden
Build a small decision table from goldens.
| Fixture | Exit | stdout shape | stderr shape | Safe extract? |
|---|---|---|---|---|
| help | 0 | usage line | empty | yes, help text |
| list | 0 | SKU lines | empty | yes, formatter |
| add valid | 0 | one SKU line | empty | no, writes DB |
| add bad qty | 2 | empty | qty error | maybe, parser |
| unknown | 1 | empty | unknown cmd | yes, dispatch |
Extract only rows marked yes or maybe. Leave rows that write files inside main.
6. Make the smallest safe extract
Move fmt_sku first and nothing else. It takes two values and returns text.
It performs no filesystem or process I/O. That is the only extract class this step allows.
# new file: sku_format.py
def fmt_sku(sku: str, qty: int) -> str:
return f"{sku.upper()}:{qty:04d}"
Keep the CLI import one line long. Do not rename arguments in the same patch.
from sku_format import fmt_sku
7. Re-run every pin
python3 edgepin.py verify --all --target inventory_cli.py
Green goldens mean the extract preserved behavior. Red goldens mean the extract is not safe.
Restore the function and shrink the patch. Never stack a second extract on red pins.
When stdout bytes drift
Do not re-record goldens as the first move. Classify the byte diff with a hex dump.
python3 - <<'PY'
from pathlib import Path
a = Path("goldens/REPLACE/stdout.bin").read_bytes()
b = Path("/tmp/new-stdout.bin").read_bytes()
print("len", len(a), len(b))
print(a[:80])
print(b[:80])
PY
Trailing newlines are the most common false drift. JSON key order is the second common cause.
Fix the CLI to emit stable bytes first. Re-record only when the old bytes were wrong.
Help text wrapping can change with argparse. Log timestamps on stderr will always drift.
Strip clocks before you write stderr goldens. Do not pin absolute paths from stack traces.
Normalize stack paths to repo-relative strings first. Then compare the normalized bytes.
Stdin is a fifth pin when needed
Some CLIs read stdin for bulk commands. Add a stdin.bin next to stdout goldens then.
The example harness already takes --stdin-file. Record stdin bytes beside argv in meta.json.
Verify must replay the same stdin blob. Skip this pin when the CLI never reads stdin.
After traces exist, draft the next extract
A coding model helps only after goldens exist. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option. Feed traces plus one candidate function, not the repo.
Ask for a purity check against the table. Reject any suggestion that opens files or sockets.
The free server option can run edgepin.py remotely. Use it when the local environment already mutates paths.
Limitations
This method ignores hidden in-process module state. Two runs can share an exit code.
Internal caches can still rot between commands. Golden files do not prove thread safety.
They do not prove disk-full error paths. Add fixtures are order sensitive by design.
Reset inventory.json before every verify run. Bytes comparison fails on unordered JSON dumps.
Canonicalize JSON if the CLI prints objects. Otherwise object key order will flap.
Who should skip this approach
Skip it when a real unit suite exists. Skip it when the product is a library.
Skip it for safety-critical control-system software. Skip it when output includes raw secrets.
Goldens committed to git would leak them. Skip it when the CLI is not user-facing.
Pin the HTTP surface instead of argv. This workflow only gates process-edge contracts.
What this workflow refuses to do
It refuses to rename while goldens are red. It refuses to extract functions that write files.
It refuses to trust green unit tests alone. The process edge remains the merge gate.
Four fixtures and one extract beat a rewrite. Stop when the next function still performs I/O.
Top comments (0)