Cap Snapshot Churn: A Failure-First Characterization Harness for Messy Repos

Coverage is a weak gate for a refactor in a repo you do not understand. Snapshot churn is a stronger one.

Here is the problem. A messy repo has behavior you cannot name yet. You want to change one function, but you cannot prove you changed only one thing.

This workflow captures the failure surface first. Then it caps how much that surface may move per commit.

1. Why failure records beat return values

Most untested code paths are error paths. They run on bad input, bad config, or missing files.

Return-value tests miss those paths. There is no return value to assert on, only an exit code and a log line.

Failure records are cheap to capture. They also move less often than happy-path logs, which carry IDs, timings, and ordering noise.

So the harness records five fields per case: exit code, a normalized stderr hash, the first three stderr lines, a timeout marker when applicable, and the case name. That is the whole tape.

2. Build the input matrix first

Do not start with tests for logic you have not read. Start with a matrix of inputs you can run today.

1. List every entry point: CLI command, cron job, import script, worker.
2. Give each entry point one good input and two or three bad ones.
3. Keep the first matrix under twenty cases. You can grow it later.
4. Store it as data, not as code, so reviewers can read it.

[
  {"name": "missing-config", "argv": ["python", "-m", "app.cli", "run", "--config", "nope.toml"]},
  {"name": "bad-json", "argv": ["python", "-m", "app.cli", "import", "fixtures/broken.json"]},
  {"name": "bad-out-dir", "argv": ["python", "-m", "app.cli", "export", "--out", "/nope/x.json"]},
  {"name": "happy-path", "argv": ["python", "-m", "app.cli", "run", "--config", "fixtures/ok.toml"]}
]

Keep the happy-path case. A failure-only tape will happily approve a refactor that breaks every success path.

3. The capture script

The following is a proposal harness, standard library only. Run it against a scratch checkout before you trust it.

#!/usr/bin/env python3
# capture_failures.py -- proposal, not yet run against your repo
import hashlib, json, re, subprocess, sys
from pathlib import Path

NOISE = [
    (re.compile(r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?\b"), "<TS>"),
    (re.compile(r"\b0x[0-9a-fA-F]+\b"), "<ADDR>"),
    (re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"), "<UUID>"),
    (re.compile(r"/tmp/[A-Za-z0-9_.-]+"), "<TMP>"),
    (re.compile(r"\b\d+(?:\.\d+)?(?:ms|s|us)\b"), "<DUR>"),
    (re.compile(r"\b\d+\b"), "<N>"),
]

def normalize(text: str) -> str:
    for pattern, token in NOISE:
        text = pattern.sub(token, text)
    return "\n".join(line.rstrip() for line in text.splitlines() if line.strip())

def record(case: dict) -> dict:
    try:
        proc = subprocess.run(
            case["argv"], capture_output=True, text=True,
            timeout=case.get("timeout", 60), cwd=case.get("cwd", "."),
        )
        code, err = proc.returncode, proc.stderr
    except subprocess.TimeoutExpired:
        code, err = "timeout", "TIMEOUT"
    err = normalize(err)
    return {
        "case": case["name"],
        "code": code,
        "err_sha": hashlib.sha256(err.encode()).hexdigest()[:16],
        "err_head": err.splitlines()[:3],
    }

def main() -> int:
    cases = json.loads(Path(sys.argv[1]).read_text())
    tape = [record(c) for c in cases]
    Path(sys.argv[2]).write_text(json.dumps(tape, indent=2) + "\n")
    print(f"recorded {len(tape)} cases")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Run it twice on the same commit. If the two tapes differ, your normalizer is incomplete and the budget below is meaningless.

python capture_failures.py cases.json tape_a.json
python capture_failures.py cases.json tape_b.json
diff tape_a.json tape_b.json && echo "tape is stable"

Never ship raw stderr off your machine. Stderr carries tokens, paths, and customer identifiers more often than people expect.

4. Set a churn budget

The budget is the number of changed records a single commit may produce. Everything else is the smallest-change loop.

Changed records	Proposed action	Rationale
0	merge after review	pure structural change
1-2	merge with written reason	a real behavior edge moved
3-5	require second reviewer	blast radius is widening
more than 5	revert, split the commit	you are editing logic, not shape

Those thresholds are a starting policy, not measured results. Tune them once you have ten real refactor commits of your own to look at.

#!/usr/bin/env python3
# churn_check.py -- exits 1 when the tape moves past budget
import json, sys
from pathlib import Path

old = {r["case"]: r for r in json.loads(Path(sys.argv[1]).read_text())}
new = {r["case"]: r for r in json.loads(Path(sys.argv[2]).read_text())}
budget = int(sys.argv[3])

changed = [k for k in sorted(set(old) | set(new)) if old.get(k) != new.get(k)]
for k in changed:
    print(f"CHANGED {k}\n  was {old.get(k)}\n  now {new.get(k)}")
print(f"{len(changed)} changed, budget {budget}")
sys.exit(1 if len(changed) > budget else 0)

5. The smallest-change loop

1. Capture the tape on HEAD and commit it beside the code it guards.
2. Make one change. Extract a function. Move a module. Do not edit logic.
3. Capture again into a second file.
4. Run the churn check with your budget.
5. Explain every changed record in the commit message.
6. If you cannot explain one, revert and split the commit.

Step 6 is where most teams quit. Stay with it. An unexplained log-line change is the common shape of a silent regression.

6. Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two parts of this loop are tedious rather than hard, and that is where the operator-supplied free model access helps. Paste the tape diff and ask for a first-pass classification of each changed record as expected or suspicious. It is triage input, not a verdict. You still own the final call.

The free server option, also operator-supplied, is useful for step 1. Run the capture matrix in a clean checkout so a stale virtualenv or a local dotfile does not leak into the tape.

Two limits apply. A remote environment drifts from your laptop, especially on locale and patch versions. Pin the runtime in both places, or capture locally and treat the remote run as a second opinion. A model also does not know your domain. It can be confidently wrong about a log line that means money.

7. Who should not use this

• Teams with no CI. A budget nobody enforces is a comment.
• Services whose stderr contains secrets or personal data.
• Code with genuinely non-deterministic output. Use distribution checks instead of hashes.
• Products where error text is a public contract. Each wording change is a breaking change.

8. What the budget does not prove

A green churn check proves the tape moved less than a threshold. It does not prove correctness.

Keep hand-written invariant tests for money, permissions, and deletion. The tape guards shape. Those tests guard meaning.

Takeaway

Capture failures, normalize them, cap the churn, then change one thing at a time. Start with four cases today.

python capture_failures.py cases.json tape_head.json
# make one structural change
python capture_failures.py cases.json tape_next.json
python churn_check.py tape_head.json tape_next.json 2