The Change Ladder: Rank Candidate Diffs by Blast Radius Before You Refactor

Most messy-repo refactors fail on sequencing, not on skill. The safe order is small and boring. Pin one behavior surface, then climb a ladder of changes. Each rung has a known blast radius and a known gate.

This post gives you two artifacts. First, a stdlib-only golden-trace recorder. Second, a five-rung change ladder. Both are cheap to add and cheap to delete. The code below is an illustrative example, not an executed benchmark.

1. Stop measuring lines. Measure surface.

A coverage percentage tells you which lines ran. It never tells you what the program promised. Messy repos often have high line coverage on paths nobody calls.

Ask a narrower question instead. What does one shipped entry point accept, print, write, and exit with? That answer is checkable before and after any refactor.

Pick one surface per refactor. Good candidates: one CLI subcommand, one python -m module, one cron script, one HTTP handler. Bad candidates: "the whole package", "everything under core/", "the API".

2. Record golden traces from the outside

The recorder runs your entry point the way a stranger would. It fixes the environment. It captures stdout, stderr, exit code, and every file the process wrote.

# charlock.py - stdlib only. Example; not executed in CI for this post.
import hashlib, json, os, pathlib, subprocess, sys, tempfile

def tree_hash(root):
    h = hashlib.sha256()
    for p in sorted(pathlib.Path(root).rglob("*")):
        if p.is_file():
            h.update(str(p.relative_to(root)).encode())
            h.update(p.read_bytes())
    return h.hexdigest()

def sha(text):
    return hashlib.sha256(text.encode()).hexdigest()

def run_case(case, repo):
    work = tempfile.mkdtemp(prefix="charlock-")
    env = {
        "PATH": os.environ.get("PATH", ""),
        "PYTHONHASHSEED": "0",
        "TZ": "UTC",
        "LC_ALL": "C",
        "HOME": work,
        "TMPDIR": work,
    }
    proc = subprocess.run(
        [sys.executable, "-m", case["module"], *case["args"]],
        cwd=repo, env=env, capture_output=True, text=True,
        timeout=case.get("timeout", 60),
    )
    return {
        "exit": proc.returncode,
        "stdout_sha": sha(normalize(proc.stdout, work)),
        "stdout_head": proc.stdout.splitlines()[:5],
        "stderr_sha": sha(normalize(proc.stderr, work)),
        "written_sha": tree_hash(work),
    }

The case list is data, not code. Keep it short enough to review in one screen.

[
  {"name": "report-empty-account", "module": "legacy.cli", "args": ["report", "--id", "0"]},
  {"name": "report-one-row", "module": "legacy.cli", "args": ["report", "--id", "1"]},
  {"name": "report-missing-flag", "module": "legacy.cli", "args": ["report"]}
]

Note the third case. Error paths are behavior too. Cheap refactors break exit codes before they break output text.

3. Normalize three things. Not thirty.

Golden traces rot for boring reasons: the clock, temp paths, and random identifiers. Fix those three and stop.

def normalize(text, work):
    text = text.replace(work, "<TMP>")
    return "\n".join(line.rstrip() for line in text.splitlines())

Every extra normalizer is a hole in the net. It hides drift you wanted to see.

Adopt one rule: no normalizer without a failing case first. If a trace drifts and you cannot name the cause, delete that case. An unstable golden file is worse than no golden file.

4. Run every case twice before you trust it

python charlock.py record --out golden.json
python charlock.py check golden.json   # run 1
python charlock.py check golden.json   # run 2

Both runs must exit zero. If run 2 differs from run 1, the case is nondeterministic. Fix the seed or drop the case. Never "just re-record".

The comparison itself is trivial and worth writing yourself.

def compare(golden, fresh):
    rows = []
    for name in sorted(golden):
        status = "OK" if golden[name] == fresh.get(name) else "DRIFT"
        rows.append((name, status))
    return rows

Print those rows before you read any diff. The table decides whether you continue.

5. The change ladder

Rank candidate diffs by blast radius, not by elegance. Then merge one rung at a time.

Rung	Change	Expected golden diff	Gate	Usual failure
0	Rename a local variable	None	None	None
1	Extract a private function in the same module	None	1 reviewer	Closure or self capture
2	Move a function to a new module, keep the old import path	None	1 reviewer plus import smoke run	Circular import
3	Change an internal signature, update all call sites	None	2 reviewers, 2 clean runs	One missed caller
4	Change a return shape or output format	Expected, reviewed diff	2 reviewers plus a golden update in its own commit	A silent downstream consumer

Three ladder rules. One rung per commit. One rung per branch. Revert the rung, never the branch.

Never write a new golden file inside a rung commit. That hides the exact diff you built the harness to expose.

Rung 4 is where a refactor becomes a product change. Say so in the PR title.

6. Where a free model fits, and where it does not

Rungs 1 and 3 are mechanical and repetitive. Drafting them is a reasonable job for a code model. The model proposes one candidate. The golden check decides whether it survives.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator of this account reports that MonkeyCode offers free model access and a free server option; I have not independently verified quotas, model names, or hardware, so re-check the current documentation before depending on either.

Two honest uses in this workflow. First, ask a free model for a single rung-1 diff on the one file you already pinned, then run the golden check on it. Second, if the entry point needs a service your laptop cannot reproduce, host the recorder on the free server option and keep the golden file in the repo. The harness stays the authority in both cases. If you want a starting point, the free model access is the cheapest way to try the ladder on a repo you are afraid to touch.

7. Limits, and who should skip this

• No runnable entry point means no surface to pin. Skip the method entirely.
• Live third-party services make traces flap. Pin a fake at the boundary or drop the case.
• Hash-based traces cannot see performance regressions. Use timing separately.
• Storing raw stdout can leak secrets. Store hashes and a five-line head only.
• New code still needs real unit tests. Golden traces describe old behavior, not intended behavior.
• Repos under a few thousand lines with passing tests usually do not need this ceremony.

8. Pre-merge checklist

1. One entry point chosen and named in the PR description.
2. Case list under twenty entries, including one error path.
3. Two consecutive clean check runs on the base branch.
4. Exactly one ladder rung in the diff.
5. No new golden file in the same commit as the code change.
6. Revert command written in the PR before review starts.

Small rungs, one surface, two clean runs. That is the whole method, and it survives messy repos.