DEV Community: Dakota Huang

The Diff Budget: Prove the Smallest Safe Refactor Before You Merge It

Dakota Huang — Wed, 28 Oct 2026 01:11:48 +0000

The Diff Budget: Prove the Smallest Safe Refactor Before You Merge It

The smallest safe refactor is a number, not an instinct. Freeze the behavior first. Then cap the diff.

Most messy-repo refactors fail on scope, not on skill. One extraction turns into a rename. The rename pulls in a signature change. Two days later the characterization tests are red and nobody knows which of the four moves broke them.

This post adds the missing gate: a machine-checked budget for how much a "behavior-preserving" commit may change. It is the last step in a series about freezing behavior. It is also the step that keeps the freeze honest.

The rule: one rung, one budget, one green run

A refactor commit gets exactly one move and one measured budget. The characterization suite must be green before and after. Any patch that exceeds the budget is not a smaller refactor — it is a redesign wearing a refactor's clothes.

The gate is a script, not a review habit. Reviews drift; a script returns exit code 1.

Step 0 — Freeze one behavior path, not all of them

You do not need full coverage to start. You need one entry point that you will touch, pinned at its output bytes.

python -m pytest -q tests/characterization/parser_entry_test.py | tee baseline.txt
sha=$(git rev-parse --short HEAD)
echo "frozen at $sha" >> baseline.txt

If that test file does not exist yet, stop here and write it. The budget gate without a frozen suite is just a diff counter.

Step 1 — Make the gate fail on an empty diff first

A gate you have never seen fail is a gate you cannot trust. Start from a clean worktree. Run it. It must report zero changed lines and pass.

Then touch one line. Run it again. It must count that line.

Step 2 — Pick the rung before you type the patch

Decide the size of the move up front. The ladder below is the part reviewers should argue about, not the code.

Rung	Move	Line budget	Extra evidence
1	Rename a local or private helper	≤ 15	Suite green, no public symbol touched
2	Extract one branch into a private function	≤ 60, 1–2 files	Suite green, call count on the entry point unchanged
3	Move one function to a new module	≤ 80, ≤ 3 files	Suite green, import graph diff read by a human
4	Change a public signature	not a refactor	Split: rung 2 plus a separate adapter commit

Rung 4 is the trap. Adding a parameter with a default looks tiny. It changes the contract, so it belongs in its own commit with its own tests.

Step 3 — Write the budget file

The budget lives in the repo. It is versioned, reviewable, and boring on purpose.

{
  "base": "9f3c1ad",
  "glob": "src/**/*.py",
  "suffixes": [".py"],
  "max_changed_lines": 60,
  "max_files": 3,
  "forbid_public_signature_changes": true
}

base is the frozen commit from Step 0. Never move it inside the same branch. Moving the base is how budgets quietly reset.

Step 4 — Run the gate

The script reads git diff --numstat for size and git diff -U0 for symbol churn. Both are plumbing commands, so they behave the same in CI and locally.

#!/usr/bin/env python3
"""refactor_gate.py — fail a refactor that exceeds its scope budget."""

import json
import re
import subprocess
import sys

SYMBOL = re.compile(r"^\s*(?:async\s+)?(?:def|class)\s+(\w+)")


def git(*args: str) -> str:
    return subprocess.run(
        ["git", *args], check=True, capture_output=True, text=True
    ).stdout


def numstat(base: str, glob: str):
    rows = []
    for line in git("diff", "--numstat", base, "--", glob).splitlines():
        added, removed, path = line.split("\t")
        if added == "-":
            continue  # binary file, not counted
        rows.append((path, int(added), int(removed)))
    return rows


def symbol_churn(base: str, suffixes: tuple[str, ...]):
    gone, added = set(), set()
    path = ""
    for line in git("diff", "-U0", base).splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
        elif line[:1] in ("+", "-") and path.endswith(suffixes):
            match = SYMBOL.match(line[1:])
            if match:
                name = f"{path}::{match.group(1)}"
                (added if line[0] == "+" else gone).add(name)
    return gone, added


def main() -> int:
    with open("refactor_budget.json", encoding="utf-8") as fh:
        budget = json.load(fh)

    rows = numstat(budget["base"], budget["glob"])
    changed = sum(a + r for _, a, r in rows)
    failures = []

    if changed > budget["max_changed_lines"]:
        failures.append(f"lines {changed} > {budget['max_changed_lines']}")
    if len(rows) > budget["max_files"]:
        failures.append(f"files {len(rows)} > {budget['max_files']}")

    gone, added = symbol_churn(budget["base"], tuple(budget["suffixes"]))
    if budget["forbid_public_signature_changes"] and (gone - added):
        failures.append(
            "symbols removed or renamed: " + ", ".join(sorted(gone - added))
        )

    for path, a, r in rows:
        print(f"{a:>5} +  {r:>5} -  {path}")
    print(f"total: {changed} lines across {len(rows)} files")

    if failures:
        print("\nGATE FAILED")
        for item in failures:
            print(" -", item)
        return 1

    print("GATE PASSED")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The rename check is deliberate. A renamed symbol shows up as a removal without a matching addition, so renames land in their own commit. That keeps git log -S useful six months later.

$ python refactor_gate.py
$ python -m pytest -q tests/characterization
$ git add -p && git commit -m "rung 2: extract tag-validation branch"

Step 5 — Set the rollback rule before you need it

Two failed gate runs on the same rung means the move is wrong, not the budget. Revert to base. Re-read the branch you are extracting. Shrink the move.

One line of policy beats an hour of negotiation:

git reset --hard "$(python -c 'import json;print(json.load(open("refactor_budget.json"))["base"])')"

Mechanical churn needs its own lane. Run the formatter as a whitespace-only commit, add it to .git-blame-ignore-revs, then start the budget clock afterward.

Where a free model and a free server genuinely help

Two parts of this loop are cheap to parallelize and easy to verify.

First, candidate patches. Ask for the smallest diff that keeps the frozen test green, then reject anything the gate flags. The gate is the judge; the model is a suggestion box.

Second, the gate run itself. If your laptop is busy with a long suite, run the suite plus refactor_gate.py on a remote box and read the exit code.

MonkeyCode's free model access and free server option are useful for exactly those two slots — drafting candidate diffs and running the frozen suite off your machine. I am not claiming benchmark numbers, quota details, or hardware here; those are operator-supplied availability claims, not measured results in this post.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A model that drafts a 400-line "cleanup" is not failing at code. It is failing at scope. The budget catches that in one second.

What this gate does not prove

The budget is a proxy, not evidence. It cannot tell you the refactor is correct; it only tells you the diff stayed small enough to reason about.

Three gaps are real:

A green suite over golden files hides behavior the golden files never captured.
Line counts reward dense edits over honest ones. A 20-line change can still alter control flow.
Renaming a private helper inside one file still trips the symbol check, by design. Expect to split those commits.

If your repo has no runnable entry point at all, do not start with the gate. Start with one call, one input, one recorded output. The gate becomes meaningful at line two.

Who should not use this

Skip it if the change is genuinely a redesign. Skip it if your team squashes every branch into one commit, because the base commit loses meaning. Skip it if the code under change is generated and the generator is the real source.

Use it when the code is messy, load-bearing, and poorly understood. That is exactly the repo where the smallest safe change is the only change you can defend in review.

Lock the Public Surface Before the Refactor: A Manifest Gate for AI-Proposed Diffs

Dakota Huang — Tue, 27 Oct 2026 22:06:30 +0000

Tests can pass while the public surface quietly changes. A rename, a new required argument, a fresh exception type — green build, broken callers. The fix is a gate, not more tests.

This is the second half of a messy-repo refactor. First you pin behavior with characterization tests. Then you allow exactly one small change. The gate below makes "small" and "safe" measurable instead of aspirational.

Why tests alone miss the smallest safe change

Characterization tests freeze what the code does for the inputs you recorded. They say nothing about what the module exposes to callers you never imported. A diff can be behavior-identical and API-hostile at the same time.

Two checks close that gap. A public surface manifest, dumped before and after. A diff budget, enforced with the same script. Both are cheap and both fail loudly.

Step 1 — Cut a baseline worktree

Never refactor on a dirty tree. Create a throwaway worktree at the commit you trust.

git worktree add ../repo-base HEAD
cd ../repo-base
python -m pytest -q tests/characterization | tail -3
python -m pytest --collect-only -q tests/characterization | tail -1

You now have an untouched tree, a pass count, and a collected-test count. Save the collected count. A refactor that "fixes" tests by deleting them will drop that number, and you want to see it drop.

Step 2 — Dump the public surface manifest

This script records exported functions, their argument names, required-argument counts, decorators, and raised exception types. Classes get bases and public methods. Everything private is ignored by convention.

Requires Python 3.9+ for ast.unparse. Save as tools/surface.py.

import ast
import json
import sys
from pathlib import Path


def surface_of(path: Path) -> dict:
    tree = ast.parse(path.read_text(encoding="utf-8"))
    public = {}
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            if node.name.startswith("_"):
                continue
            args = node.args
            public[node.name] = {
                "kind": "function",
                "args": [a.arg for a in args.args],
                "kwonly": [a.arg for a, d in zip(args.kwonlyargs, args.kw_defaults) if d is None],
                "required": len(args.args) - len(args.defaults),
                "decorators": [ast.unparse(d) for d in node.decorator_list],
                "raises": sorted(
                    {ast.unparse(h.type) if h.type else "bare"
                     for h in ast.walk(node) if isinstance(h, ast.Raise)}
                ),
            }
        elif isinstance(node, ast.ClassDef) and not node.name.startswith("_"):
            public[node.name] = {
                "kind": "class",
                "bases": [ast.unparse(b) for b in node.bases],
                "methods": sorted(
                    n.name for n in node.body
                    if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))
                    and not n.name.startswith("_")
                ),
            }
    return {"module": path.stem, "public": public}


def manifest(root: Path) -> dict:
    return {
        str(p.relative_to(root)): surface_of(p)
        for p in sorted(root.rglob("*.py"))
        if "__pycache__" not in p.parts and not p.name.startswith("surface")
    }


if __name__ == "__main__":
    root = Path(sys.argv[1])
    out = Path(sys.argv[2])
    out.write_text(json.dumps(manifest(root), indent=2, sort_keys=True), encoding="utf-8")
    print(f"wrote {out}")

Run it in the baseline tree. Commit the output or stash it somewhere the candidate tree can read.

cd ../repo-base
python tools/surface.py src surface.base.json

Step 3 — Pin behavior with three kinds of test

Keep this boring and fast. Characterization tests are a recording, not a design document.

Golden output for the main entry point, byte-for-byte.
The exception types and messages the code raises today.
One test per side effect you can observe without a network call.

Run them against the baseline until they pass. Then stop. If a test needs a rewrite to pass, the test is wrong, not the code.

Step 4 — Ask for the smallest patch

Now get a candidate diff. Free model access works fine here because the model proposes and the gate disposes. You are not asking for correctness. You are asking for one narrow change you will judge mechanically.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access and free server option referenced here are operator-supplied availability claims; verify current limits in the product itself before relying on them.

A prompt shape that produces reviewable diffs:

Goal: extract the date parsing in report.py into parse_date().
Constraints:
- Do not change any public name, signature, or raised exception.
- Touch at most 2 files.
- Keep the diff under 60 changed lines.
- Return only a unified diff. No prose.

Apply the patch to a second worktree, not to your working copy.

git worktree add ../repo-cand HEAD
cd ../repo-cand
git apply --check candidate.diff && git apply candidate.diff

If you are running the model on the free server option, keep the repo slice small and exclude secrets. A free tier is a convenience, not a trust boundary.

Step 5 — Run the gate

The gate compares two manifests and the diff size. Save as tools/gate.py.

import json
import subprocess
import sys
from pathlib import Path

BUDGET_LINES = 60
BUDGET_FILES = 2


def load(p: str) -> dict:
    return json.loads(Path(p).read_text(encoding="utf-8"))


def surface_reasons(base: dict, cand: dict) -> list:
    reasons = []
    for mod, info in base.items():
        if mod not in cand:
            reasons.append(f"module removed: {mod}")
            continue
        for name, sig in info["public"].items():
            new = cand[mod]["public"].get(name)
            if new is None:
                reasons.append(f"public symbol removed: {mod}::{name}")
                continue
            if sig["kind"] != new["kind"]:
                reasons.append(f"kind changed: {mod}::{name}")
                continue
            if sig["kind"] == "function":
                if new["args"][: len(sig["args"])] != sig["args"]:
                    reasons.append(f"arg order changed: {mod}::{name}")
                if new["required"] > sig["required"]:
                    reasons.append(f"new required parameter: {mod}::{name}")
                added = set(new["raises"]) - set(sig["raises"])
                if added:
                    reasons.append(f"new exception type: {mod}::{name} {sorted(added)}")
            else:
                lost = set(sig["methods"]) - set(new["methods"])
                if lost:
                    reasons.append(f"method removed: {mod}::{name} {sorted(lost)}")
    return reasons


def diff_budget() -> list:
    out = subprocess.run(
        ["git", "diff", "--numstat", "HEAD", "--", "."],
        capture_output=True, text=True, check=True,
    ).stdout
    lines = files = 0
    for row in out.splitlines():
        add, rem, _ = row.split("\t")
        if not add.isdigit() or not rem.isdigit():
            return ["binary file in diff"]
        lines += int(add) + int(rem)
        files += 1
    reasons = []
    if lines > BUDGET_LINES:
        reasons.append(f"diff too large: {lines} > {BUDGET_LINES} lines")
    if files > BUDGET_FILES:
        reasons.append(f"too many files: {files} > {BUDGET_FILES}")
    return reasons


if __name__ == "__main__":
    reasons = surface_reasons(load("surface.base.json"), load("surface.candidate.json"))
    reasons += diff_budget()
    if reasons:
        print("REJECT")
        for r in reasons:
            print(f"  - {r}")
        sys.exit(1)
    print("ACCEPT")

Run it in the candidate worktree, after generating the candidate manifest and the tests.

cd ../repo-cand
python tools/surface.py src surface.candidate.json
python -m pytest -q tests/characterization | tail -1
python tools/gate.py

Step 6 — Read the rejection reasons as data

Three rejections are common and each means something specific.

Reason	Usual cause	Response
`new required parameter`	Model added a config flag	Ask for a default value
`new exception type`	Error handling drifted	Pin the old exception in a test
`diff too large`	Task was too broad	Split into two refactors

The write-only part is important. Append each rejection to a JSONL log with the prompt hash. After ten attempts you can see which prompt shapes produce oversized diffs, and fix the prompt rather than the model.

When this gate is the wrong tool

Some repos should skip it.

Dynamic re-exports through __getattr__ or globals(): the manifest will be empty or wrong.
Deliberate public API changes: the gate cannot tell a planned break from an accidental one. Bump the version and update the baseline instead.
Generated code: regenerate first, then take the baseline. Otherwise every run rejects.
C extensions or code that only exists after import hooks: static parsing misses it.
Teams without a golden-output corpus: write those tests first. The gate protects a surface, not a behavior.

Also skip this if the change is really a feature. A feature needs new behavior, and a behavior lock will fight you the whole way.

What you get

One refactor per branch. A baseline manifest you can diff by eye. A hard cap on diff size. A rejection log that improves your prompts over time.

That is the whole loop: record behavior, lock the surface, propose one change, gate it. The model writes the patch. Your harness decides whether the patch ever touches main.

If you want to run steps 4 through 6 without a paid seat, the free model access and free server option are enough to try the loop end to end.

Your Refactor Copies a List. The Caller Still Holds the Old One.

Dakota Huang — Tue, 27 Oct 2026 19:01:08 +0000

In-place mutation is the quietest part of a module's contract. Most characterization tests assert what a function returns. Almost none assert that the caller's list changed underneath. Replace rows.sort() with rows = sorted(rows) and the suite stays green. Production still reads the old list, in the old order.

This is the seam that survives a copy-versus-mutate refactor. Freeze it before you touch the body.

1. The break is aliasing, not logic

A caller passes a list. The function appends to it and returns None. The caller never reads the return value; it reads its own variable later. That is a real contract, and it lives outside your assertions.

Now a refactor returns a fresh list. Return-value tests pass. The caller keeps its stale object. Nothing raises. No error is logged.

The damage is silent because the signature did not change. So the gate must record mutations, not just outputs.

2. Build a mutation ledger

The probe below wraps a function. It deep-snapshots every argument before the call. It diffs each argument after the call and appends one JSON line per invocation.

# mutation_probe.py
import copy, functools, json


def _snapshot(value):
    try:
        return copy.deepcopy(value)
    except Exception:
        return repr(value)


def _diff(before, after, path="$"):
    out = []
    if type(before) is not type(after):
        return [{"path": path, "change": "type",
                 "before": type(before).__name__, "after": type(after).__name__}]
    if isinstance(before, dict):
        for key in sorted(set(before) | set(after), key=str):
            if key not in before:
                out.append({"path": f"{path}[{key!r}]", "change": "added"})
            elif key not in after:
                out.append({"path": f"{path}[{key!r}]", "change": "removed"})
            else:
                out.extend(_diff(before[key], after[key], f"{path}[{key!r}]"))
        return out
    if isinstance(before, list):
        if len(before) != len(after):
            out.append({"path": path, "change": "length",
                        "before": len(before), "after": len(after)})
        for i, (b, a) in enumerate(zip(before, after)):
            out.extend(_diff(b, a, f"{path}[{i}]"))
        return out
    if before != after:
        out.append({"path": path, "change": "value", "before": before, "after": after})
    return out


def freeze_mutations(fn, ledger_path):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        before = {f"arg{i}": _snapshot(v) for i, v in enumerate(args)}
        before.update({f"kw:{k}": _snapshot(v) for k, v in kwargs.items()})
        result = fn(*args, **kwargs)
        after = {f"arg{i}": _snapshot(v) for i, v in enumerate(args)}
        after.update({f"kw:{k}": _snapshot(v) for k, v in kwargs.items()})
        record = {"function": fn.__qualname__, "mutations": []}
        for name in before:
            record["mutations"].extend(_diff(before[name], after[name], name))
        with open(ledger_path, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(record, sort_keys=True, default=repr) + "\n")
        return result
    return wrapper

Two notes before you run it. Snapshots cost memory, so target narrow functions. And default=repr keeps unserializable payloads from killing the run.

3. Capture against real call shapes

Tests alone give you a weak baseline. They exercise the arguments someone remembered to write.

List candidate functions with grep -rn "\.append(\|\.update(\|\.sort(\|\.pop(" src/.
Wrap the top candidates with freeze_mutations(fn, "ledger-before.jsonl").
Drive them with recorded payload fixtures, not synthetic ones.
Run the suite twice and require identical ledgers, sorted by (function, path, change).
Commit ledger-before.jsonl as the baseline artifact.

Step 4 matters. A ledger that differs between runs is noise, not a contract. Fix the noise before you trust the diff.

4. Turn the ledger into a gate

The gate is a set comparison. It does not care about ordering across lines. It cares about which mutation paths exist.

# mutation_gate.py
import json, sys


def keys(path):
    seen = set()
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            rec = json.loads(line)
            for m in rec["mutations"]:
                seen.add((rec["function"], m["path"], m["change"]))
    return seen


baseline, current = keys(sys.argv[1]), keys(sys.argv[2])
lost, new = baseline - current, current - baseline
for tag, rows in (("LOST", lost), ("NEW", new)):
    for fn_name, path, change in sorted(rows):
        print(f"{tag}\t{fn_name}\t{path}\t{change}")
sys.exit(1 if lost or new else 0)

A LOST line is the dangerous one. A mutation that used to happen no longer happens. A NEW line is a contract you just invented.

5. Make the smallest safe change

One function per commit. Never two.

Refactor a single function to return a new object.
Re-run the probe into ledger-after.jsonl.
Run the gate against the baseline.
If LOST is non-empty, revert the copy and keep in-place mutation.
If only NEW appears, delete it and re-check.

The revert in step 4 is not a failure. It is the gate doing its job.

6. Decide per mutation, not per module

Observed mutation	Caller relies on it?	Action
Appends to a passed-in list	Yes, verified by reader code	Keep; document the contract in a docstring
Sorts a passed-in list	Unclear	Keep; add a test that asserts the caller's list order
Mutates a config dict	No	Convert to copy; expect `LOST` and accept it explicitly
Mutates an argument, returns same object	Yes	Convert only if you also update every reader
Mutates across a thread boundary	Unknown	Stop; the probe cannot see the race

Run that table before you run the refactor. It converts a judgement call into a recorded decision.

7. Where a disposable box helps

Capture runs dirty the environment. They write ledgers, load fixtures, and sometimes touch a scratch database. Running them on your working checkout is friction you do not need.

MonkeyCode's free server option covers that part: one throwaway box for the capture run. Its free model access covers the drafting step, where a raw ledger becomes readable assertion names instead of JSON noise. Both are operator-supplied availability claims; check current limits yourself before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the gate in your own CI. Generated assertion names must be read against the ledger line by line. A model will happily invent a path that never mutated.

8. Limitations, and who should skip this

List indices shift. A path like arg0[2].id moves when the list resizes. Normalize indices out before you compare across branches.
Opaque objects leak. Deep copies of database handles, sockets, or C extensions often fail, and repr fallback hides the mutation.
Threads and processes are invisible. The probe sees one call, one thread, one moment.
Side effects outside arguments are out of scope. Writes, network calls, and file handles need a separate gate.
Deep snapshot is slow. Do not wrap hot paths, and do not wrap loops that run thousands of times per request.

Skip this approach if your code already treats data as immutable. Skip it if the object graph is too large to copy. Skip it if the mutation you fear lives in another process.

The rule

Asserting return values proves half the contract. Freeze what the function did to the arguments, or the quietest break ships.

Replay the Seam: A Differential Gate for the Smallest Safe Refactor

Dakota Huang — Tue, 27 Oct 2026 15:56:02 +0000

Capture calls at one seam. Replay them against both versions. Widen the change only when the diff is empty.

That is the whole method. A characterization test proves your repo still runs. A seam replay proves one boundary still answers the same way.

A passing replay is evidence, not proof. It cannot see branches your corpus never reached.

Why the diff review fails on messy repos

Most refactors start with reading a diff until it feels safe. Feeling is not a signal. Reviewers also read the new code, not the old behavior.

A rounded total is a good illustration. legacy.py returns Decimal, the refactor returns float. Code review sees two numbers. The replay sees 12.30 versus 12.300000000000001.

The gate is narrow on purpose. One seam, one corpus, one comparison function. Everything else stays untouched.

Step 1: Pick exactly one seam

A seam is a function you can call from outside without starting the app. It takes plain arguments and returns a plain value.

Use this checklist before recording anything.

Question	Pass condition
Callable in isolation?	No DB, no network, no sleep
Arguments serializable?	JSON round-trips them
Return value comparable?	Same type on both sides
Call frequency	Reached by normal traffic or tests

If a seam fails two rows, skip it and pick a neighbor. Nested impure seams are why refactors stall.

Step 2: Record a corpus at the boundary

Wrap the current function. Log arguments and output to JSONL. Write nothing else.

The sketch below is unexecuted. Adapt names to your repo.

# tools/recorder.py
import json, pathlib

CORPUS = pathlib.Path("corpus/seam.jsonl")

def record(fn, *args, **kwargs):
    out = fn(*args, **kwargs)
    CORPUS.parent.mkdir(parents=True, exist_ok=True)
    row = {"args": list(args), "kwargs": kwargs, "out": out}
    with CORPUS.open("a") as fh:
        fh.write(json.dumps(row, sort_keys=True, default=str) + "\n")
    return out

Then call record from one place: the production entry point or an existing test fixture. Do not change logic inside the wrapper.

Let it run until the corpus covers your known cases. Thirty rows with edge cases beat ten thousand duplicates. Check the tail of the file for secrets before committing.

Step 3: Replay against the candidate

Write the candidate next to the original. Both must be importable at the same time. Replay runs the same rows through the new function.

# tools/replay.py
import importlib, json, pathlib, sys

def load(path):
    text = pathlib.Path(path).read_text()
    return [json.loads(line) for line in text.splitlines() if line.strip()]

def replay(rows, fn, eq=lambda a, b: a == b):
    bad = []
    for i, row in enumerate(rows):
        try:
            got = fn(*row["args"], **row["kwargs"])
        except Exception as exc:
            got = {"__raised__": type(exc).__name__}
        if not eq(row["out"], got):
            bad.append((i, row["out"], got))
    return bad

if __name__ == "__main__":
    rows = load("corpus/seam.jsonl")
    mod = importlib.import_module(sys.argv[1])   # e.g. billing.v2
    fn = getattr(mod, sys.argv[2])              # e.g. total_for
    bad = replay(rows, fn)
    print(f"replayed={len(rows)} diverged={len(bad)}")
    for i, want, got in bad[:5]:
        print(f"row={i}\n  want={want!r}\n  got ={got!r}")
    sys.exit(1 if bad else 0)

Run it as a normal command. Exit code 1 means stop.

python tools/replay.py billing.legacy total_for
python tools/replay.py billing.v2 total_for

The first run is your control. It should print diverged=0. If it does not, the corpus is corrupt or the seams disagree already.

Step 4: Triage divergences into four classes

Do not fix code first. Classify first. The class decides who acts.

Class	Example	Action
Identical	`7 == 7`	Widen the change
Representation	`Decimal("7.00")` vs `7.0`	Fix the comparator, rerun
Ordering	List order from a set	Sort in comparator, note it
Real divergence	`None` vs `0`	Stop, change the candidate

Representation noise hides real bugs. Tighten the comparator one rule at a time. str, repr, and json.dumps are three different comparisons.

A comparator that normalizes everything compares nothing. Keep the strictest version that passes the control run.

Step 5: Move in the smallest safe steps

Use a ladder. Each rung is a separate commit with its own replay run.

Rename only. No logic change. Replay must show diverged=0.
Copy the function. Add the candidate beside the original. Nothing calls it yet.
Replay the candidate. Fix divergences in the candidate, not the original.
Switch one call site. Run the app's tests plus replay.
Delete the original. Only after the corpus and tests both pass.

If a rung fails, revert that rung alone. Small commits make the revert cheap and the review honest.

Set a budget before you start. Example: no rung may exceed forty changed lines. Over budget means the seam was too large. Split it.

Where the free server and free models fit

Replay runs are boring and long. They belong on a machine that is not your laptop. MonkeyCode's free server option is one place to run the harness headless, and its free model access can draft the recorder and comparator for your seam.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the model for scaffolding, not for judgment. Ask it for a comparator covering Decimal, datetimes, and ordered dicts. Then read every line. Ask it to cluster failing rows by traceback type. Then verify the clusters by hand.

Keep the corpus out of any chat context. Send shapes, not payloads. If a row contains customer data, rewrite it before sharing.

The useful pattern is a closed loop. Generate the harness, run it on the free server, paste back the summary line, fix one rung, repeat. The model never sees your production output.

Limitations and who should skip this

Side effects break this gate. If the seam writes files or calls an API, replay changes state. Record the effects separately or pick another seam.

Non-determinism breaks it too. Clocks, random seeds, and UUIDs need pinning before replay means anything. Count them first.

Corpus coverage is the hard ceiling. A green replay means "no regression on recorded rows". It does not mean "correct".

Skip this approach when the seam is a single trivially pure function. A unit test is cheaper than a corpus. Skip it when you cannot serialize arguments without leaking secrets. Skip it when the refactor is a rewrite, because there is no control run.

Use it when the function is old, ugly, and load-bearing. That is where a forty-line rung plus an empty diff beats a long review.

The gate in one line

Record the seam. Replay both sides. Classify the diff. Widening requires zero divergences on the control and the candidate.

Run It Ten Times Before You Write One Characterization Test

Dakota Huang — Tue, 27 Oct 2026 13:52:52 +0000

Conclusion first: run your entrypoint ten times and count distinct outputs. If that count is not one, you have no baseline. You have a noise source, and any test you write on top of it will flake.

Most messy-repo refactors stall right here. People write characterization tests against an unstable output, the tests fail randomly, and the tests get deleted. The fix is not better assertions. The fix is measuring nondeterminism before you assert anything.

This post is a workflow for that measurement, plus the smallest safe change that follows it.

Step 0: Pick one entrypoint and one working directory

Pick the smallest command that produces the artifact you care about. Do not start with the whole pipeline.

cd sandbox && ./run_report.sh > /tmp/out1.txt
cd sandbox && ./run_report.sh > /tmp/out2.txt
diff /tmp/out1.txt /tmp/out2.txt | head -40

If the diff is empty, stop reading and go write the test. If it is not empty, you need a census, not a diff.

Step 1: Census the output across N runs

The script below copies the work directory fresh for every run. That kills state leaking between runs, which is the most common false source of drift.

#!/usr/bin/env python3
"""nondeterminism_census.py - count distinct outputs of one command across N runs.

Usage:
  python nondeterminism_census.py --runs 10 --workdir ./sandbox \
      --report census.json -- ./run_report.sh
"""
import argparse
import json
import re
import shutil
import subprocess
import sys
import tempfile
from collections import Counter
from pathlib import Path

CANDIDATE_MASKS = [
    ("iso8601", re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?")),
    ("uuid", re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")),
    ("hex", re.compile(r"\b[0-9a-f]{16,}\b")),
    ("int", re.compile(r"\b\d+\b")),
]


def run_once(cmd, workdir, timeout):
    with tempfile.TemporaryDirectory() as tmp:
        sandbox = Path(tmp) / "repo"
        shutil.copytree(workdir, sandbox)
        proc = subprocess.run(cmd, cwd=sandbox, capture_output=True,
                              text=True, timeout=timeout)
        return {"code": proc.returncode, "out": proc.stdout, "err": proc.stderr}


def distinct(texts):
    return len(set(texts))


def fit_masks(outputs):
    chosen, masked, best = [], list(outputs), distinct(outputs)
    improved = True
    while improved:
        improved = False
        for name, pattern in CANDIDATE_MASKS:
            if name in chosen:
                continue
            trial = [pattern.sub("<%s>" % name, t) for t in masked]
            if distinct(trial) < best:
                chosen.append(name)
                masked, best, improved = trial, distinct(trial), True
                break
    return chosen, masked, best


def main(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument("--runs", type=int, default=10)
    parser.add_argument("--workdir", default=".")
    parser.add_argument("--timeout", type=int, default=120)
    parser.add_argument("--report", default="census.json")
    parser.add_argument("cmd", nargs=argparse.REMAINDER)
    args = parser.parse_args(argv)

    cmd = args.cmd[1:] if args.cmd and args.cmd[0] == "--" else args.cmd
    if not cmd:
        parser.error("need a command after --")

    workdir = Path(args.workdir).resolve()
    runs = [run_once(cmd, workdir, args.timeout) for _ in range(args.runs)]
    outs = [r["out"] for r in runs]
    codes = Counter(r["code"] for r in runs)

    before = distinct(outs)
    masks, masked, after = fit_masks(outs)

    report = {
        "command": cmd,
        "runs": args.runs,
        "exit_codes": dict(codes),
        "distinct_stdout_before_masks": before,
        "distinct_stdout_after_masks": after,
        "masks_fitted": masks,
        "masked_sample": masked[0][:2000] if masked else "",
    }
    Path(args.report).write_text(json.dumps(report, indent=2))
    print(json.dumps({k: v for k, v in report.items() if k != "masked_sample"}, indent=2))
    if after != 1:
        print("NOT STABLE: %d distinct masked outputs. Find the hidden state." % after,
              file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

To see the shape of the report, use a deliberately noisy demo entrypoint. This is a demo script, not a benchmark of anything.

#!/usr/bin/env bash
set -euo pipefail
printf 'generated=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf 'run_id=%s\n' "$(cat /proc/sys/kernel/random/uuid)"
printf 'rows=3\n'
printf 'total=41.50\n'

The report shape from that demo looks like this. Run it yourself before trusting the numbers.

{
  "command": ["./run_report.sh"],
  "runs": 10,
  "exit_codes": {"0": 10},
  "distinct_stdout_before_masks": 10,
  "distinct_stdout_after_masks": 1,
  "masks_fitted": ["iso8601", "uuid"]
}

Ten runs, ten outputs, two masks, one stable baseline. That is the state you want before writing a single assertion.

Step 2: Read the result with a decision table

Do not interpret "stable" loosely. Use three outcomes.

Census result	Meaning	Action
`distinct_after_masks == 1`	Output is stable under the fitted masks	Freeze a golden file, write one characterization test
`distinct_after_masks > 1`	Hidden state, ordering, or concurrency	Quarantine the output, find the state source
`exit_codes` has more than one key	Error path flakiness	Census success and failure paths separately

A non-unique exit code set is the one people skip. A command that returns 0 eight times and 1 twice is not a stable baseline either.

Step 3: Treat every mask as a written claim

The mask list is code. It asserts that a field carries no contract worth testing.

That claim needs an owner and a reason. Keep the masks in a reviewed file, not inline in a test.

# masks.py - every entry drops a field from the characterization baseline.
MASKS = [
    ("iso8601", r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"),
    # reason: wall-clock generation time, no consumer reads it
    ("uuid", r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"),
    # reason: per-run id, asserted separately for format only
]

A mask that swallows a business field is a bug painted over. Review int masks hardest, because they hide totals, counts, and money.

Step 4: Freeze one golden file, then write the test

With a stable masked output, the test is boring and that is the point.

def test_report_matches_frozen_baseline(tmp_path, run_entrypoint):
    output = run_entrypoint(tmp_path)
    assert mask_text(output) == FROZEN_REPORT

One test per artifact. If the golden file is huge, split it by section, not by regex.

Step 5: Make the smallest safe change, then re-run the census

This is where the census pays off twice. After the extraction or rename, census again and compare the fitted mask set.

python nondeterminism_census.py --runs 10 --workdir ./sandbox \
    --report after.json -- ./run_report.sh

If masks_fitted grew from ["iso8601", "uuid"] to include int, your change introduced numeric drift. That is a regression signal the tests cannot give you, because the tests are already masked.

A change is small and safe when three things hold: the mask set did not grow, the golden diff is explainable line by line, and the exit code set stayed identical.

Step 6: Where free model access and a free server fit

Two parts of this loop are tedious rather than hard: drafting mask candidates, and running enough repetitions to trust the count.

MonkeyCode, an open-source coding project, exposes free model access and a free server option. Its operator also describes a free token allowance (reported to me as 10M); confirm current figures on their site before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the part I would actually use a model for, and the part I would not.

Drafting normalizers from diff pairs. Feed the census the ten distinct outputs and ask for a mask list with one justification per entry. Then delete half of it. Models over-mask.
Running the repetition loop off your laptop. A ten-run census on a slow entrypoint blocks your machine. A free server is a reasonable place to run the loop and publish census.json so a teammate can reproduce it.
Not this: letting a model decide that a drifting field is "noise." If the field is under investigation, masking it destroys the evidence.

Guardrails, in order of importance. Never send proprietary output or secrets to a hosted model. Keep the mask file in version control so every dropped field is reviewable. Re-run the census after applying any model-drafted mask; acceptance is a measured distinct_after_masks == 1, not a convincing explanation.

Limitations, stated plainly

Mask fitting here is greedy and order-dependent. It can mask more than necessary, and int is dangerous.
Ten runs is weak evidence. A field that drifts once in fifty runs will pass this census.
copytree isolates files, not environment. Env vars, locale, network, and clock still leak.
Masked golden files cannot catch regressions in masked fields. Accept that trade or quarantine the artifact instead.
A shared free tier may queue, rate-limit, or change terms. Do not put it on a release-critical path.

Who should not use this

If the nondeterminism is the bug itself, such as corruption under concurrency, masking hides your only evidence. Stop and debug instead. If the output is contractual bytes, like signed invoices, do not mask anything. And if one run of the entrypoint takes an hour, a ten-run census is not affordable; sample fewer runs and lean on logs.

For everything else, the order is fixed. Ten runs, one stable masked baseline, one test, then the smallest change you can review. If you want to try the mask-drafting step without standing up a local model, MonkeyCode's free model access and free server option are one way in.

Two Runs Must Agree Before You Freeze a CLI Transcript

Dakota Huang — Tue, 27 Oct 2026 12:51:47 +0000

Freeze a repo's behavior only after two independent runs produce the same transcript. If two runs disagree, you have no baseline, only noise. This article builds a process-boundary harness that measures that, then guards the smallest safe change.

Most refactor baselines live at the function boundary. Someone calls build_report() and snapshots the returned string. That is convenient, and it hides the parts that actually break: exit codes, stderr, files written, and ordering across a whole run.

Why the Process Boundary Wins First

Functions change signatures freely. A CLI's contract changes slowly and visibly. Users script against exit codes and stdout, not against your internal call graph.

A transcript is four things: exit code, stdout, stderr, and the output tree. Digest the output files and you catch silent data corruption. Digest them with SHA-256 and you catch one flipped byte.

This is coarser than unit tests, and that is the point. You can record it before you understand the code. Understanding comes from reading the diff, not from reading the module.

Step 1: Pick One Seam and One Fixture

Choose the entry point your team actually runs. Often that is a bin/ script or a python -m package.cli invocation. Avoid library internals for the first recording.

Create one fixture directory per scenario. Copy in the minimum inputs: a small CSV, a config file, a SQLite file. The harness copies the fixture into a temp dir, so runs never share state.

Declare one output directory and require the command to write only there. Mixing outputs into the fixture root makes the digest useless. If the command writes everywhere, that itself is a finding worth recording.

Step 2: Build the Normalizer Registry

Some bytes are different on every run and carry no contract. Normalize them explicitly, with a named pattern per rule. Never normalize with a broad catch-all.

NORMALIZERS = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"), "<uuid>"),
    (re.compile(r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?\b"), "<timestamp>"),
    (re.compile(r"(?:/tmp|/private/var)/[\w./-]+"), "<tmp>"),
    (re.compile(r"\b0x[0-9a-fA-F]{6,}\b"), "<addr>"),
    (re.compile(r"\b\d+(?:\.\d+)? ?(?:ms|s|KB|MB)\b"), "<duration>"),
]

Keep the registry short and reviewable. Every entry is a promise that this field is not part of the contract. Document the promise in a comment above the pattern.

Step 3: Make Determinism a Precondition

This is the step most harnesses skip. Run the command twice with different hash seeds. Compare the normalized transcripts.

env = dict(os.environ, PYTHONHASHSEED=seed)  # seed is "0" then "1"

Python randomizes string hashing per process by default; PYTHONHASHSEED is the documented override (docs). Two seeds expose dict-order and set-order leakage cheaply.

If the two runs disagree, do not record a golden file. Fix the order instead, which is usually a one-line change:

for row in sorted(rows, key=lambda r: r.customer_id):

Pinning the seed to 0 in production would hide the bug for real users. That trade is almost never worth it. Sort the data, then freeze.

Step 4: Record, Then Verify the Recording

python golden.py record --name invoice_cli --fixture fixtures/invoice -- ./bin/invoice --month 2026-09
python golden.py check  --name invoice_cli --fixture fixtures/invoice -- ./bin/invoice --month 2026-09

The harness below is the whole artifact, around sixty lines and no dependencies.

# golden.py
import argparse, hashlib, json, os, re, shutil, subprocess, tempfile
from pathlib import Path

GOLDEN = Path("golden")

def normalize(text):
    for pattern, repl in NORMALIZERS:
        text = pattern.sub(repl, text)
    return text

def digest_tree(root):
    rows = []
    for path in sorted(root.rglob("*")):
        if path.is_file():
            data = path.read_bytes()
            rows.append([str(path.relative_to(root)),
                         hashlib.sha256(data).hexdigest(), len(data)])
    return rows

def run_once(cmd, fixture, seed):
    work = Path(tempfile.mkdtemp(prefix="golden-"))
    shutil.copytree(fixture, work, dirs_exist_ok=True)
    env = dict(os.environ, PYTHONHASHSEED=seed)
    try:
        proc = subprocess.run(cmd, cwd=work, env=env, timeout=120,
                              capture_output=True, text=True)
        out = work / "out"
        return {
            "exit_code": proc.returncode,
            "stdout": normalize(proc.stdout),
            "stderr": normalize(proc.stderr),
            "files": digest_tree(out) if out.exists() else [],
        }
    finally:
        shutil.rmtree(work, ignore_errors=True)

def record(cmd, fixture, name):
    a, b = run_once(cmd, fixture, "0"), run_once(cmd, fixture, "1")
    if a != b:
        print("UNSTABLE: two runs disagree; do not freeze")
        return 1
    GOLDEN.mkdir(exist_ok=True)
    (GOLDEN / f"{name}.json").write_text(json.dumps(a, indent=2, sort_keys=True))
    return 0

def check(cmd, fixture, name):
    want = json.loads((GOLDEN / f"{name}.json").read_text())
    got = run_once(cmd, fixture, "0")
    if want == got:
        print("MATCH")
        return 0
    for key in ("exit_code", "stdout", "stderr", "files"):
        if want[key] != got[key]:
            print(f"DIFF {key}")
            print(f"  want {json.dumps(want[key])[:300]}")
            print(f"  got  {json.dumps(got[key])[:300]}")
    return 1

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("mode", choices=["record", "check"])
    p.add_argument("--name", required=True)
    p.add_argument("--fixture", required=True)
    p.add_argument("cmd", nargs=argparse.REMAINDER)
    a = p.parse_args()
    fn = record if a.mode == "record" else check
    raise SystemExit(fn(a.cmd, a.fixture, a.name))

Commit the golden JSON. Run check immediately after recording. A recorder that cannot reproduce its own snapshot is broken, and you want to learn that now.

Step 5: Make the Smallest Safe Change

Now touch one thing. Rename a variable, extract one function, move one import. Run check after each edit, not after five edits.

MATCH means the observable behavior survived. That is a narrow claim, and it is the claim you need before continuing. Repeat until the messy region is parsed into small, verified moves.

If you must change behavior deliberately, do it in a separate commit with its own golden update. Never mix a behavior change into a mechanical commit. Reviewers cannot separate the two afterward.

Step 6: Triage Every Diff With This Table

Diff location	Example	Verdict
Normalized field only	`<uuid>` swapped	Add the rule, re-record
Line order in stdout	rows shuffled	Fix with `sorted()`, do not re-record
New stderr line	deprecation warning	Block; isolate the source
Exit code changed	`0` to `1`	Block; this is a behavior change
Output file added	new `out/debug.log`	Block unless intended
File hash changed	bytes differ	Inspect with `cmp -l` before re-recording

Two rules keep this honest. Never re-record a golden file to silence a diff you have not explained. And never widen a normalizer to cover a real behavior change.

A Concrete Failure I Hit

The first fixture I recorded printed a temp path in a warning line. The harness flagged UNSTABLE, not a diff, because the two seeds produced different paths. That was the harness working correctly.

I added the <tmp> rule, re-recorded, and got MATCH. The real bug appeared later, when the sorted output revealed duplicate rows my extraction step had introduced.

Where a Free Model and a Disposable Box Fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The normalizer registry is fiddly regex work. I paste a failing diff and ask a free model for candidate patterns, then test each candidate by hand. I keep only rules I can explain in one sentence, because an unexplained rule silently deletes evidence.

The second-seed run also needs a clean machine. A laptop with leftover files in the fixture path weakens the transcript. MonkeyCode's free server option is a reasonable place to run the gate, since a disposable box keeps local state out of the recording. I cannot promise specs, quotas, or permanence for that tier, so treat it as replaceable and pin your interpreter version in the golden file metadata.

If your repo already has a runnable entry point, record one scenario today. The free model and free server tiers are enough to try the loop before you commit tooling time.

Limitations and Who Should Skip This

This harness proves stability, not correctness. A wrong result that is reproducibly wrong will freeze happily. Pair it with assertions about values, not just about sameness.

Some noise resists normalization. Thread interleavings, wall-clock scheduling, and network responses will keep the UNSTABLE branch hot. Add a seeded or recorded-input mode to the code before you add more regexes.

Skip this approach if the command writes to a live database you cannot clone. Skip it if the output tree is large binary blobs, where hashing is slow and diffs are unreadable. Skip it if there is no entry point to run yet, because a harness cannot characterize code that never executes.

For everyone else, the sequence is fixed. Two runs must agree, then you freeze, then you make one small change and check again.

Count the Clocks Before Characterization Tests: A Nondeterminism Census for Messy Repos

Dakota Huang — Tue, 27 Oct 2026 09:37:35 +0000

Characterization tests fail for a boring reason: the code reads the wall clock.
You cannot pin behavior you cannot reproduce twice.
So count the clocks first, then move code.

There is a second trap. unittest.mock.patch targets a dotted path, not an object. A refactor that moves a module silently breaks the patch. The test still passes. The freeze silently stops working.

This article covers one artifact that catches both problems: a nondeterminism census with a patch-target inventory.

Step 1: Census every clock, seed, and host read

Run this on the repo root. It parses ASTs, so it does not need imports to succeed.

#!/usr/bin/env python3
# nondet_census.py - list every clock, seed, and host read in a repo.
import ast
import json
import pathlib
import sys

WATCH = {
    "time": {"time", "monotonic", "perf_counter", "time_ns"},
    "datetime": {"now", "utcnow", "today"},
    "random": {"random", "randint", "choice", "shuffle"},
    "uuid": {"uuid1", "uuid4"},
    "os": {"urandom", "getpid", "getenv"},
    "secrets": {"token_hex", "token_bytes", "randbelow"},
    "socket": {"gethostname", "gethostbyname"},
}
SKIP = {".git", ".venv", "venv", "build", "dist", "node_modules"}


def module_name(path, root):
    parts = list(path.relative_to(root).with_suffix("").parts)
    if parts[-1] == "__init__":
        parts.pop()
    return ".".join(parts)


def scan(path, root):
    tree = ast.parse(path.read_text(encoding="utf-8"))
    mod = module_name(path, root)
    aliases, rows = {}, []

    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for a in node.names:
                if a.name in WATCH:
                    aliases[a.asname or a.name] = a.name

    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module in WATCH:
            for a in node.names:
                if a.name in WATCH[node.module]:
                    rows.append({
                        "file": str(path), "line": node.lineno,
                        "kind": "local_copy",
                        "symbol": f"{node.module}.{a.name}",
                        "patch_target": f"{mod}.{a.asname or a.name}",
                    })
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            owner = node.func.value
            if isinstance(owner, ast.Name) and owner.id in aliases:
                real = aliases[owner.id]
                if node.func.attr in WATCH[real]:
                    rows.append({
                        "file": str(path), "line": node.lineno,
                        "kind": "module_attr",
                        "symbol": f"{real}.{node.func.attr}",
                        "patch_target": f"{mod}.{owner.id}.{node.func.attr}",
                    })
    return rows


if __name__ == "__main__":
    root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
    out = []
    for path in sorted(root.rglob("*.py")):
        if SKIP & set(path.parts):
            continue
        out.extend(scan(path, root))
    pathlib.Path("nondet_targets.json").write_text(json.dumps(out, indent=2))
    print(f"{len(out)} rows -> nondet_targets.json", file=sys.stderr)

python nondet_census.py .
# 61 rows -> nondet_targets.json

Sixty-one rows is normal. Most are harmless. You need the list anyway, because the freeze step consumes it.

Step 2: Sort rows by what the patch can actually reach

Each row is one of three shapes. The shape decides your first move.

Row kind	Example	Patch target	First move
`module_attr`	`import time; time.time()`	`app.reports.build.time.time`	Patch at the importing module
`local_copy`	`from time import time`	`app.reports.build.time`	Patch the local name
`server_side`	`SELECT now()`, C extension clocks	none	Pin inputs and outputs instead

The local_copy row is the quiet one. Patching time.time does nothing if the module did a from-import. Your test looks deterministic and is not.

Rows also cluster. Ten call sites in one module usually mean one seam, not ten fixes. Group by file before you touch anything.

The classification is mechanical, and a language model is decent at it. I paste the JSON rows into MonkeyCode's free model access with a strict output schema: {line, bucket, reason}, bucket in freeze|inject|leave.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the assertions. The model only labels rows. Every label gets a human yes before it enters the target file.

Step 3: Freeze from the inventory, not from memory

Your conftest.py should read the census output. Hand-written patch lists drift in a week.

# conftest.py
import json
import os
import pathlib
import random

import pytest

ROWS = json.loads(pathlib.Path("nondet_targets.json").read_text())
SEED = int(os.environ.get("GATE_SEED", "0"))
FROZEN = 1_700_000_000.0


@pytest.fixture(autouse=True)
def frozen_world(monkeypatch):
    random.seed(SEED)
    for row in ROWS:
        if row["kind"] == "module_attr":
            monkeypatch.setattr(row["patch_target"], lambda: FROZEN)
        elif row["kind"] == "local_copy":
            monkeypatch.setattr(row["patch_target"], lambda: FROZEN)
    yield

monkeypatch.setattr accepts dotted strings and imports the module for you. See the pytest docs on monkeypatch.

One caveat: datetime.now() needs a datetime return value. Branch on row["symbol"] and return datetime.fromtimestamp(FROZEN) instead of a float.

Step 4: Add a test that the patch still lands

This is the gate that survives a refactor. It resolves every target and checks object identity against the real stdlib function.

# tests/test_patch_targets.py
import importlib
import json
import pathlib
import time as stdlib_time

import pytest

ROWS = json.loads(pathlib.Path("nondet_targets.json").read_text())
STDLIB = {
    "time.time": stdlib_time.time,
    "time.monotonic": stdlib_time.monotonic,
    "time.perf_counter": stdlib_time.perf_counter,
}


def resolve(dotted):
    mod, _, attr = dotted.rpartition(".")
    obj = importlib.import_module(mod)
    for part in attr.split("."):
        obj = getattr(obj, part)
    return obj


@pytest.mark.parametrize("row", ROWS, ids=lambda r: r["patch_target"])
def test_target_resolves(row):
    try:
        assert resolve(row["patch_target"]) is not None
    except (ImportError, AttributeError) as exc:
        pytest.fail(f"patch target gone: {row['patch_target']} ({exc})")


@pytest.mark.parametrize("row", [r for r in ROWS if r["kind"] == "module_attr"])
def test_patch_hits_stdlib_object(row):
    assert resolve(row["patch_target"]) is STDLIB[row["symbol"]]

Now a module move produces a red test with the old dotted path in the name. That is the failure you want. Compare with the unittest.mock docs on where patches are applied: docs.python.org.

Step 5: Run the stability gate

Twenty seeds, fixed hash seed, same command. Flaky characterization tests are worse than none.

#!/usr/bin/env bash
set -euo pipefail
mkdir -p runs
for seed in $(seq 1 20); do
  GATE_SEED=$seed PYTHONHASHSEED=$seed \
    pytest -q tests/characterization --tb=no > "runs/seed_$seed.txt" \
    || { echo "nondeterministic at seed=$seed"; exit 1; }
done
echo "20/20 stable"

The loop is CPU-light and wall-clock heavy. Twenty runs of a slow suite is hours on a laptop. This is where a hosted runner earns its keep, and MonkeyCode's free server option is one place to put it. Treat that as an option, not a benchmark.

Save the runs/ folder. It is the baseline you diff against after the change.

Step 6: Make the smallest safe change, then re-run three things

Do the extraction. Then run, in this order:

python nondet_census.py . and diff the new JSON against the old one.
pytest -q tests/test_patch_targets.py.
The 20-seed loop again.

A clean diff means no clock moved into a new module. A failing target test means a patch path died. A flaky seed means you moved code across a freeze boundary.

If the census diff shows a target count change, stop. That is a behavior change hiding inside a move.

Limitations and who should skip this

Child processes do not inherit patches. Subprocesses that call the clock need env-var injection.
C extensions and database now() are invisible to AST scans. Pin their inputs and outputs instead.
Threads plus global monkeypatching can race. Prefer an injected clock object in threaded code.
PYTHONHASHSEED fixes hash order only. It says nothing about set iteration in other runtimes.
If the repo has no runnable test suite, the census still helps, but characterization comes first.

Skip this if your nondeterminism is the specification. Fuzzers and property tests want randomness, not a frozen clock.

Also skip it if nobody will review the model's row labels. An unreviewed leave bucket is an unfrozen test waiting to flake in CI.

What this buys you

Characterization tests are only as stable as the clocks under them. A census makes that stability checkable, and a patch-target test makes it survive the refactor.

The script is the part worth stealing. Run it on your messiest module before you promise anyone a safe change.

If you want a second opinion on row classification, the free model access is enough for that job. Keep the tests yours.

Map Every Changed Line to the Test That Ran It Before You Refactor

Dakota Huang — Tue, 27 Oct 2026 06:32:59 +0000

Map Every Changed Line to the Test That Ran It Before You Refactor

Start with the uncomfortable result. A repo at high line coverage can still have zero tests touching the function you are about to split. Coverage is aggregated. Refactor risk is local.

Before the smallest safe change, you need a different artifact. Not a percentage. A table that maps each changed line to the tests that executed it.

This post builds that table. It joins a git diff -U0 against coverage.py dynamic contexts. The output is one row per hunk, with a verdict you can gate on.

Why the coverage percentage is the wrong gate

Line coverage answers a repo-wide question. Refactoring asks a line-level one.

Three properties matter for a single changed line:

Does any test execute it at all?
Which specific tests execute it?
How fast are those tests?

A line reached only by a slow integration test is expensive to change. A line reached by three fast unit tests is cheap. The aggregate number hides both cases equally.

Step 1: Record coverage with test attribution

coverage.py can record a dynamic context per executed line. Set it to test_function and every line carries the name of the test that ran it.

# .coveragerc
[run]
dynamic_context = test_function
branch = True

Now run the suite and dump JSON. The --show-contexts flag is what writes per-line test names into the output.

pip install coverage pytest
coverage run -m pytest -q
coverage json --show-contexts -o cov.json

Expect a slowdown. Dynamic contexts cost runtime, so measure it on your own suite. Keep this run serial. Parallel workers scramble the attribution you are trying to collect.

Step 2: Reduce the diff to changed line ranges

You do not need to parse added content. With zero context, the hunk header is the change set.

git diff -U0 --no-color $(git merge-base main HEAD) > change.diff

Headers look like @@ -40,2 +40,3 @@. The +40,3 part means lines 40 through 42 of the new file are touched. A pure deletion gives +40,0, which means there is nothing new to cover.

That is the whole parse. Two regexes, no heuristics, no language model.

Step 3: Join both and print the table

The script below reads cov.json and change.diff. It reports, per hunk, how many tests touched the lines and which lines had none.

#!/usr/bin/env python3
# hunk_gate.py - map diff hunks to the tests that executed those lines.
# Usage: python hunk_gate.py cov.json change.diff  (exit 1 if any changed line has no test)
import json, re, sys
from pathlib import Path

FILE_HDR = re.compile(r'^\+\+\+ b/(?P<path>.+)$')
HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(?P<start>\d+)(?:,(?P<count>\d+))? @@')

def changed_ranges(diff_text):
    ranges, path = {}, None
    for raw in diff_text.splitlines():
        if raw.startswith('+++ '):
            m = FILE_HDR.match(raw)
            path = m.group('path') if m else None
            continue
        m = HUNK.match(raw)
        if not (m and path):
            continue
        start = int(m.group('start'))
        count = int(m.group('count') or 1)
        if count == 0:            # pure deletion: no new lines to cover
            continue
        ranges.setdefault(path, []).append((start, start + count - 1))
    return ranges

def lookup(files, path):
    if path in files:
        return files[path]
    for key, value in files.items():   # coverage may store absolute paths
        if key.endswith(path):
            return value
    return None

def main(cov_path, diff_path):
    files = json.loads(Path(cov_path).read_text())['files']
    ranges = changed_ranges(Path(diff_path).read_text())
    print('file'.ljust(38), 'lines'.ljust(10), 'tests'.ljust(5), 'verdict')
    missing_total = 0
    for path, spans in sorted(ranges.items()):
        if not path.endswith('.py'):
            continue
        contexts = (lookup(files, path) or {}).get('contexts', {})
        for lo, hi in spans:
            tests, missing = set(), []
            for line in range(lo, hi + 1):
                ctx = contexts.get(str(line), [])
                tests.update(ctx)
                if not ctx:
                    missing.append(line)
            verdict = 'OK' if not missing else 'NO-TEST ' + str(missing)
            missing_total += len(missing)
            print(path.ljust(38), f'{lo}-{hi}'.ljust(10),
                  str(len(tests)).ljust(5), verdict)
    print('\nchanged lines with no covering test:', missing_total)
    return 1 if missing_total else 0

if __name__ == '__main__':
    sys.exit(main(*sys.argv[1:3]))

Treat this as a sketch, not a shipped tool. Run it on one small module first and check the JSON shape against your installed coverage.py version. The contexts key only appears when --show-contexts was used.

Illustrative output, not from a real repo:

file                                   lines      tests verdict
src/billing/invoice.py                 40-42      3     OK
src/billing/invoice.py                 88-91      0     NO-TEST [88, 89, 90, 91]
src/billing/legacy_discount.py         12-12      1     OK

Step 4: Read column three before column four

A green OK with one test is weaker evidence than NO-TEST on a branch you plan to delete. The line list is the useful part. It is the exact set of lines you are changing blind.

Classify every hunk before you edit anything:

Hunk state	What it means	Action before the change
No covering test	Nothing executes these lines	Write a characterization test, or delete the code
One slow integration test	Feedback loop is minutes long	Add a direct unit-level pin
Several fast tests	The change is observable	Proceed with the smallest safe change
Covered only through a shared fixture	Hidden coupling to setup order	Add one direct-call test

Most legacy modules produce the second row. That is the real finding, and it is invisible in any coverage badge.

Step 5: Run only the attributed tests while you edit

Contexts come back as strings like test_invoice.py::test_proration_rounding. Convert them to node IDs and the loop stays seconds long.

pytest -q src/billing/test_invoice.py::test_proration_rounding \
         src/billing/test_invoice.py::test_zero_quantity

Then change one hunk. Rerun that subset. Regenerate the diff. Repeat. Never batch five hunks and rerun once.

Where a free model and a free server genuinely help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two parts of this workflow are tedious rather than hard, which makes them reasonable candidates for a free tier.

The attributed baseline run. Dynamic contexts force a serial run, so the suite is slower than usual. That is a good job for a remote box. MonkeyCode offers a free server option; I am not asserting a machine size, a time limit, or permanence. If your baseline fits, run the instrumented suite there and pull back cov.json and the diff.

The uncovered-hunk backlog. After the gate, you have a list of ranges nothing executes. Free model access through MonkeyCode is useful for drafting scenarios for those ranges, such as which inputs could reach line 88. You still write the assertions. Delete any proposal that only restates the code in English.

That is the whole product role. Remove both mentions and the gate still works.

Limitations and who should skip this

Lines, not branches. Contexts attach to executed lines. A covered if body says nothing about the branch you are about to add.
Covered is not asserted. This gate reports who runs a line. It does not report whether that test would fail on a bad edit. Pair it with a mutation or kill-rate check.
Subprocess coverage is invisible. If tests drive a server in another process, instrument that process or the application lines will all look uncovered.
C extensions and generated code. Attribution is unreliable or meaningless there. Omit those paths in .coveragerc.
Parallel runners. Dynamic contexts plus parallel workers yield attribution you cannot trust. Keep the gate run serial.

Skip this entirely if your suite is integration-only over HTTP. Skip it if a serial run fits no machine you can reach. Skip it if the target code is untested and about to be deleted, because deletion needs no coverage map.

Checklist before you touch the code

Set dynamic_context = test_function in .coveragerc.
Run the suite serially, then coverage json --show-contexts.
Diff against the merge base, not against your last commit.
Run the hunk gate and read every NO-TEST line out loud.
Classify each hunk with the table above.
Add the missing pins first, then make the smallest safe change.
Re-run the gate after each hunk, not after the whole refactor.

The map is cheap to build and hard to argue with. It replaces "the suite is green" with "these five tests ran the four lines I am changing." That is a claim you can review before the diff exists.

If you try it on a legacy module, the output that surprises people is usually one slow test covering forty lines. Share the hunk that surprised you.

Pin the Order of Side Effects Before the Refactor: An Effect Ledger Gate

Dakota Huang — Tue, 27 Oct 2026 03:28:02 +0000

Return values pass. Side effects drift. That is the refactor bug your tests never catch.

A characterization suite asserts outputs. It does not assert that the invoice file is written once, then the email, then the log line. Reorder those three and every assertion stays green.

This post adds one gate to the loop you already run: characterization tests first, smallest safe change second. The gate is an effect ledger. Record the ordered list of boundary crossings before the change. Re-record after. Diff the two lists.

The reference implementation below is about sixty lines of Python. It stores digests, not payloads, so secrets stay out of the fixture.

Why the return value is a weak gate

A function can return an identical dict and still behave differently. It may call the payment API twice, or write the export before validating it. Return-value tests see one snapshot at the end. The boundary sees a sequence.

Sequences are cheap to pin. Each event is a small tuple: kind, target, digest. For most single-threaded code the list is fully deterministic.

That determinism is the whole trick. You are not testing correctness. You are testing that nothing moved.

Step 1 — Inventory the effect boundaries

Before writing any test, list where the code touches the outside world. Grep the module for six families of calls.

rg -n "open\(|\.write\(|requests\.|httpx\.|session\.|logging\.|logger\.|datetime\.now|time\.time|random\.|uuid4|os\.environ" app/billing/

Then classify every hit. Separate these kinds: fs.write, http.out, log, clock, random, env.read, db.exec. Seven buckets cover most backend code.

This labeling pass is where a free model is genuinely useful. Paste the grep output and ask for a table: file, line, candidate kind, confidence. Free model access is enough for that job, because it is classification, not reasoning.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not accept the table as final. Dynamic dispatch hides boundaries: getattr, DI containers, decorators that wrap clients. Read the call sites yourself and delete wrong rows.

Step 2 — Install the recorder

Reference implementation. Python 3.10+, standard library only.

# ledger.py
import hashlib, json

KINDS = ("fs.write", "http.out", "log", "clock", "random", "env.read", "db.exec")


class Ledger:
    def __init__(self):
        self.events = []

    def record(self, kind, target, payload):
        assert kind in KINDS, kind
        blob = json.dumps(payload, sort_keys=True, default=str)
        self.events.append({
            "seq": len(self.events) + 1,
            "kind": kind,
            "target": str(target),
            "digest": hashlib.sha256(blob.encode()).hexdigest()[:16],
        })

    def trace(self):
        return [f'{e["kind"]}|{e["target"]}|{e["digest"]}' for e in self.events]

    def dump(self, path):
        with open(path, "w", encoding="utf-8") as f:
            json.dump(self.trace(), f, indent=2)

Note what is missing: payloads. Only digest and target survive. That keeps tokens, emails, and account numbers out of the fixture file.

Now the patch set. One context manager, one patch per boundary you confirmed in step 1.

# seams.py
import builtins, contextlib, hashlib
from datetime import datetime, timezone
from unittest import mock

FROZEN = datetime(2026, 1, 1, tzinfo=timezone.utc)


@contextlib.contextmanager
def effects(ledger, pay_client, clock):
    real_open = builtins.open

    def traced_open(file, mode="r", *a, **kw):
        if any(flag in mode for flag in ("w", "a", "x", "+")):
            ledger.record("fs.write", file, {"mode": mode})
        return real_open(file, mode, *a, **kw)

    def traced_send(url, **kw):
        payload = kw.get("json") or kw.get("data") or {}
        ledger.record("http.out", url, payload)
        return mock.Mock(status_code=202, json=lambda: {"ok": True})

    def traced_now():
        ledger.record("clock", "utcnow", {})
        return FROZEN

    with mock.patch("builtins.open", traced_open), \
         mock.patch.object(pay_client, "post", traced_send), \
         mock.patch.object(clock, "now", traced_now):
        yield

The frozen clock does two jobs at once. It records the read. It also removes wall-clock noise from payload digests.

Step 3 — Capture the baseline on a clean box

Run the capture where the machine cannot leak state. Your laptop has leftover files, a dirty environment, and a real clock.

The free server option fits this step. It gives you a disposable box for the capture run; nothing about the box needs to be fast. Copy the repo, capture, keep the JSON.

python -m tests.capture --scenario invoice_reissue --out tests/ledger/invoice_reissue.json
python -m tests.capture --scenario invoice_reissue --out /tmp/second.json
cmp tests/ledger/invoice_reissue.json /tmp/second.json && echo deterministic

Short capture driver:

# tests/capture.py
import argparse, importlib
from ledger import Ledger
from seams import effects

SCENARIOS = {
    "invoice_reissue": ("app.billing", "reissue_open_invoices", {"account": "acct_1"}),
}


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--scenario", required=True)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()

    mod_name, fn_name, kwargs = SCENARIOS[args.scenario]
    mod = importlib.import_module(mod_name)
    ledger = Ledger()
    with effects(ledger, mod.pay_client, mod.clock):
        getattr(mod, fn_name)(**kwargs)
    ledger.dump(args.out)

Run it twice before trusting it. If the two files differ, stop. Non-determinism means the gate is not ready. Fix the source first: sort dict iteration, inject the clock, seed randomness.

Step 4 — Gate the change in pytest

List equality is the check. The useful part is the first divergence index, not a wall of diff text.

# tests/test_effect_ledger.py
import json, pathlib, pytest
from ledger import Ledger
from seams import effects
from app import billing

FIXTURES = pathlib.Path("tests/ledger")


@pytest.mark.parametrize("scenario", ["invoice_reissue"])
def test_effect_order_unchanged(scenario):
    baseline = json.loads((FIXTURES / f"{scenario}.json").read_text())
    ledger = Ledger()
    with effects(ledger, billing.pay_client, billing.clock):
        billing.reissue_open_invoices(account="acct_1")
    got = ledger.trace()

    if got != baseline:
        first = next(
            (n for n, (a, b) in enumerate(zip(got, baseline)) if a != b),
            min(len(got), len(baseline)),
        )
        rows = []
        for n in range(max(0, first - 2), first + 3):
            old = baseline[n] if n < len(baseline) else "-"
            new = got[n] if n < len(got) else "-"
            rows.append(f"  {n:>3} baseline={old}\n      got     ={new}")
        pytest.fail(f"effect order diverged at index {first}\n" + "\n".join(rows))

Now make the smallest safe change. Extract one function. Move one block. Do not edit behavior.

Run the gate again. Green means the effects did not move. That is the only claim you get to make.

Step 5 — Read the divergence, decide, repeat

A diverged index is information, not automatic failure. Three cases cover almost everything.

Index shifted, digest identical. You added a read or reordered a log line. Usually harmless; refresh the baseline in its own commit.
Digest changed, count identical. The payload shape moved. Inspect before accepting anything.
Count changed. A boundary crossing appeared or vanished. This is the double-charge class, and it deserves a real review.

Never refresh the baseline inside the refactor commit. Keep the refresh separate so reviewers see intent.

Decision table — which gate to add first

Symptom in the messy repo	Gate	Typical cost
Function has no tests at all	Characterization tests on return values	hours
Return values pinned, I/O order unproven	Effect ledger (this post)	about an hour
Tests exist but may be unable to fail	Kill-rate or mutation gate	about two hours
Change must be mechanical only	AST diff gate	about an hour
Public surface must not move	Export manifest gate	about thirty minutes

Cheap gates run first. Each one closes a different failure class, so they compose instead of overlapping.

Limitations

Concurrency breaks sequence determinism. Record per thread or serialize the scenario.
Digest-only ledgers hide the why. Keep redacted payloads in a private artifact when you truly need them.
Cross-process effects escape the patch set: queues, workers, cron jobs, subprocess calls.
Model-classified boundaries miss getattr dispatch and DI wiring. Human review is mandatory.
If the refactor is supposed to change effects, freeze the target list first. This gate cannot separate intended from accidental.

Who should skip this

No callable entry point yet. Capture inputs first, then come back.
Heavy metaprogramming where seams are not nameable. Patch at the adapter layer instead.
Effects already covered by contract tests at the integration boundary.

What you end up with

One JSON file per scenario. One pytest gate. A first-divergence index whenever the refactor moves something it should not.

That file is the cheapest insurance in the loop. Capture it before the change, never after. If you want the boundary-classification pass without a paid key, free model access handles it; the free server option handles the clean capture run.

Record What Your Code Throws Before You Refactor It

Dakota Huang — Tue, 27 Oct 2026 00:22:32 +0000

Most refactors keep the happy path intact. They quietly rewrite the error path.

Messages get reworded. Exception types get normalized. Validation order flips. Reviewers read the new structure, not the old failure output.

Freeze those three things before you extract anything.

Why the error path drifts

A helper extraction moves two raise statements into one place. The first error now wins. Nothing fails, tests stay green, and your logs change shape.

Happy-path tests never reach those branches. So they cannot notice.

The fix is not more assertions. It is a recorded baseline of observable failures, replayed after each small change.

Step 1: Put the imperfect module on the table

Here is a realistic messy function. Validation, pricing, and coupon lookup are welded together.

# pricing.py — before
COUPONS = {'WELCOME': 500, 'BULK': 2000}

def quote(qty, unit_cents, coupon=None):
    if qty <= 0:
        raise ValueError('qty must be positive')
    total = qty * unit_cents
    if coupon:
        if coupon not in COUPONS:
            raise KeyError('unknown coupon: ' + coupon)
        total = total - COUPONS[coupon]
    if total < 0:
        raise ValueError('total below zero: %d' % total)
    return total

Note the third branch. It only fires when unit_cents is negative. That is exactly the branch a refactor deletes by accident.

Step 2: Write a malformed-input corpus

Cases are data, not test functions. Keep them in one importable file.

# corpus.py
CASES = [
    {'id': 'qty-zero', 'fn': 'quote', 'args': (0, 100)},
    {'id': 'qty-negative', 'fn': 'quote', 'args': (-2, 100)},
    {'id': 'qty-one', 'fn': 'quote', 'args': (1, 250)},
    {'id': 'coupon-ok', 'fn': 'quote', 'args': (4, 250, 'WELCOME')},
    {'id': 'coupon-unknown', 'fn': 'quote', 'args': (4, 250, 'NOPE')},
    {'id': 'coupon-empty', 'fn': 'quote', 'args': (4, 250, '')},
    {'id': 'negative-unit', 'fn': 'quote', 'args': (1, -100, 'WELCOME')},
    {'id': 'float-qty', 'fn': 'quote', 'args': (1.5, 100)},
]

Four rules made this list short and useful.

Two cases per validation branch, one on each side of the boundary.
One case per flag state, including empty string and unknown key.
One odd-but-legal argument type, such as 1.5.
No mocks. Real module, real arguments.

Step 3: Record outcomes and replay them

The harness has two modes. --save writes the baseline. Without it, the run diffs and exits non-zero.

# parity.py
import argparse, importlib, json, re, sys
from pathlib import Path

ADDR = re.compile(r'0x[0-9a-fA-F]+')
PATH = re.compile(r'(?:/[^/ ]+){2,}')

def normalize(text):
    text = ADDR.sub('0xADDR', text)
    text = PATH.sub('<PATH>', text)
    return ' '.join(text.split())

def outcome(fn, args):
    try:
        value = fn(*args)
    except Exception as exc:
        return {
            'kind': 'raise',
            'type': type(exc).__name__,
            'msg': normalize(str(exc)),
            'args': [normalize(repr(a)) for a in exc.args],
        }
    return {'kind': 'return', 'value': normalize(repr(value))}

def record(module_name, cases):
    mod = importlib.import_module(module_name)
    return {c['id']: outcome(getattr(mod, c['fn']), c['args']) for c in cases}

def compare(base, current):
    out = []
    for case_id in sorted(set(base) | set(current)):
        if base.get(case_id) != current.get(case_id):
            out.append({'case': case_id,
                        'before': base.get(case_id),
                        'after': current.get(case_id)})
    return out

def coverage_gaps(module_name, cases, minimum):
    mod = importlib.import_module(module_name)
    public = {n for n, o in vars(mod).items()
              if callable(o) and not n.startswith('_')
              and getattr(o, '__module__', None) == module_name}
    counts = {}
    for c in cases:
        counts[c['fn']] = counts.get(c['fn'], 0) + 1
    return {'uncased': sorted(public - set(counts)),
            'thin': sorted(f for f, n in counts.items() if n < minimum)}

def main(argv=None):
    ap = argparse.ArgumentParser()
    ap.add_argument('--module', required=True)
    ap.add_argument('--corpus', required=True)
    ap.add_argument('--baseline', required=True)
    ap.add_argument('--save', action='store_true')
    ap.add_argument('--min-cases', type=int, default=2)
    a = ap.parse_args(argv)

    cases = importlib.import_module(a.corpus).CASES
    gaps = coverage_gaps(a.module, cases, a.min_cases)
    if gaps['uncased'] or gaps['thin']:
        print('corpus gaps:', json.dumps(gaps))

    if a.save:
        data = record(a.module, cases)
        Path(a.baseline).write_text(json.dumps(data, indent=2, sort_keys=True))
        print('saved', a.baseline, len(data), 'cases')
        return 0

    base = json.loads(Path(a.baseline).read_text())
    breaks = compare(base, record(a.module, cases))
    for item in breaks:
        print(json.dumps(item, sort_keys=True))
    print(f'{len(breaks)} break(s) across {len(base)} cases')
    return 1 if breaks else 0

if __name__ == '__main__':
    sys.exit(main())

Run it once, on the untouched module.

python parity.py --module pricing --corpus corpus --baseline baseline.json --save
python parity.py --module pricing --corpus corpus --baseline baseline.json; echo $?

The saved baseline is boring on purpose. Excerpt:

{
  "coupon-unknown": {"kind": "raise", "type": "KeyError",
    "msg": "'unknown coupon: NOPE'", "args": ["'unknown coupon: NOPE'"]},
  "negative-unit": {"kind": "raise", "type": "ValueError",
    "msg": "total below zero: -600", "args": ["'total below zero: -600'"]},
  "float-qty": {"kind": "return", "value": "150.0"}
}

Commit baseline.json together with parity.py and corpus.py. The baseline is now a contract.

Step 4: Make the smallest safe change

Now extract coupon handling. Resistance is natural, so here is what a first draft looks like.

# pricing.py — first draft, rejected
def _apply_coupon(total, coupon):
    if not coupon:
        return total
    if coupon not in COUPONS:
        raise KeyError(f'unknown coupon {coupon!r}')
    return total - COUPONS[coupon]

def quote(qty, unit_cents, coupon=None):
    if qty <= 0:
        raise ValueError('quantity must be positive')
    total = _apply_coupon(qty * unit_cents, coupon)
    if total < 0:
        raise ValueError(f'total below zero: {total}')
    return total

The structure is what you wanted. The harness exits 1 with three records.

qty-zero and qty-negative: 'qty must be positive' became 'quantity must be positive'.
coupon-unknown: "'unknown coupon: NOPE'" became a differently quoted KeyError message.

The fix is to keep the strings byte-identical and merge only the control flow. Rerun. Zero breaks. That is your smallest safe change.

If you want new wording, do it in a second commit that edits the message and updates baseline.json in the same diff. Reviewers then see intent, not drift.

Step 5: Check the corpus for dead weight

A corpus with many cases and few distinct outcomes hides gaps. Group the baseline records by their serialized outcome and inspect the clusters.

This is where a free model is genuinely useful. Paste the baseline JSON and ask which clusters contain cases that cannot fail independently. The model proposes sharper inputs. You decide which cases to add, and you write the expected outcomes yourself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use the free model access for that clustering pass and the free server option to host the replay job, so the baseline check runs without occupying my laptop. The corpus and the baseline are plain files, so the workflow survives if you move elsewhere.

Do not let the model write the harness output. It guesses. The recorder measures.

Choosing the right freeze

What you must preserve	Artifact	Why the others miss it
Exception type, message, validation order	exception parity replay	catches branches happy-path tests skip
Exact document or report text	byte-pinned golden file	`repr()` of a parsed object hides whitespace
Order of writes, emits, or external calls	call-order log with a spy	parity records results, not sequences
Latency or allocation budget	separate measurement run	replay is deterministic only

Limitations

Normalization hides regressions. Keep the regex list tiny and reviewed in the diff.
Side-effect order is invisible here. Add a spy log if order matters.
The corpus is a net with holes. An uncovered branch cannot break parity.
Dynamic ids and timestamps need explicit normalizers, not broad ones.
except Exception deliberately misses SystemExit and KeyboardInterrupt.
This proves behavior did not change. It never proves the behavior is correct.

Who should skip this

Skip it for throwaway scripts, spikes, and code with no observable error contract. Skip it if your team will not commit baseline updates alongside code changes, because the baseline then becomes noise you learn to ignore.

Use it when a module's failures are part of its interface. Record first. Extract second. The two commands above are the whole workflow.

Your Characterization Tests Passed. The Side Effects Changed Order.

Dakota Huang — Mon, 26 Oct 2026 21:17:34 +0000

Return values are the easy part of a refactor. Side-effect order is where messy repos break quietly.

A green suite can still merge a file that now gets written twice. It can still ship a temp file that is deleted before its reader runs. Characterization tests pin outputs. They rarely pin sequence.

This post builds a small ledger that records ordered side effects. You capture it before the refactor. You diff it after the smallest safe change.

The failure a return-value test cannot see

Here is a real shape. A nightly exporter writes a CSV, renames it into place, then shells out to a mailer.

Your test asserts the returned path and the row count. Both stay correct while the internal order flips.

# Before: write -> replace -> notify
df.to_csv(tmp)
os.replace(tmp, final)
subprocess.run(["mailer", final], check=True)

# After: notify -> write -> replace (still returns the same path)

The test passes. The mailer reads a stale file. Nobody sees it until a customer asks where the report went.

Step 1 — Name one scenario, not the whole repo

One entry point, fixed inputs, no network. Keep it boring.

# scenarios/nightly_export.py
from reports.exporter import build_nightly


def run():
    build_nightly(dest="/tmp/export-check")

Do not try to ledger the whole package on day one. A scenario that touches ten modules gives you a diff you cannot read. Start where the change will land.

Step 2 — List candidates with grep, not with a hunch

grep -rnE "open\(|\.write\(|os\.replace|shutil\.(move|copy)|subprocess\.|requests\.(post|put)" reports/ \
  | grep -v tests/

You want the boundaries the scenario actually reaches. Grep gives you candidates, not proof. Confirm each hit by running the scenario once and printing the files it touched.

Step 3 — Capture an ordered ledger

Sixty lines. Patch a few boundaries, record every call in arrival order, normalize volatile strings.

# effect_ledger.py — ordered, normalized side-effect capture
import builtins
import hashlib
import json
import os
import re
import subprocess
from contextlib import contextmanager
from unittest import mock

LEDGER = []

_TMP = re.compile(r"/tmp/[A-Za-z0-9_.\-]+")
_TS = re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}")


def _norm(value):
    text = str(value)
    return _TS.sub("<TS>", _TMP.sub("<TMP>", text))


def _record(kind, name, payload=None):
    LEDGER.append({
        "seq": len(LEDGER),
        "kind": kind,
        "name": _norm(name),
        "payload": None if payload is None else _norm(payload),
    })


class _WriteTracer:
    """Records a content hash when the handle closes."""

    def __init__(self, handle, path):
        self._h, self._path = handle, path
        self._sha = hashlib.sha256()
        self._closed = False

    def write(self, data):
        self._sha.update(data.encode() if isinstance(data, str) else data)
        return self._h.write(data)

    def close(self):
        if not self._closed:
            self._closed = True
            _record("write.close", self._path, self._sha.hexdigest()[:16])
        return self._h.close()

    def __enter__(self):
        return self

    def __exit__(self, *exc):
        self.close()
        return False

    def __getattr__(self, name):
        return getattr(self._h, name)


@contextmanager
def ledgered():
    LEDGER.clear()
    real_open, real_replace, real_run = builtins.open, os.replace, subprocess.run

    def open_hook(file, mode="r", *args, **kwargs):
        handle = real_open(file, mode, *args, **kwargs)
        if set(mode) & set("wax+"):
            _record("write.open", file, mode)
            return _WriteTracer(handle, file)
        return handle

    def replace_hook(src, dst, *args, **kwargs):
        _record("os.replace", dst, f"from={src}")
        return real_replace(src, dst, *args, **kwargs)

    def run_hook(*args, **kwargs):
        cmd = args[0] if args else kwargs.get("args")
        _record("subprocess.run", cmd)
        return real_run(*args, **kwargs)

    with mock.patch("builtins.open", open_hook), \
         mock.patch("os.replace", replace_hook), \
         mock.patch("subprocess.run", run_hook):
        yield LEDGER


def dump(path):
    payload = json.dumps(LEDGER, indent=2)
    with builtins.open(path, "w", encoding="utf-8") as fh:
        fh.write(payload)

Note the tracer. A path name tells you a file was opened. A hash tells you it held what you expected.

Step 4 — Prove the baseline is stable before you trust it

Run the same scenario three times under different hash seeds.

for seed in 0 1 2; do
  PYTHONHASHSEED=$seed python capture_ledger.py nightly_export "ledgers/base-$seed.json"
done
python diff_ledger.py ledgers/base-0.json ledgers/base-1.json
python diff_ledger.py ledgers/base-0.json ledgers/base-2.json

If those diffs are non-empty, stop. Your scenario is nondeterministic and the gate is worthless.

Sort directory listings. Freeze the clock. Pin the temp prefix. Only then record a baseline.

Step 5 — Make the smallest safe change

Extract one block into a helper. Rename one variable. Change no behavior.

python capture_ledger.py nightly_export ledgers/after.json
python diff_ledger.py ledgers/base-0.json ledgers/after.json

Exit code 1 means the effect sequence moved. Exit code 0 means this scenario saw the same effects in the same order.

That is a narrow claim. Keep it narrow and it stays trustworthy.

Step 6 — Read the diff as a decision, not an alarm

Ledger diff	Interpretation	Action
Identical sequence	Change looks behavior-preserving for this path	Commit; keep baseline
New effect, expected	Refactor introduced a deliberate boundary	Update baseline in the same commit
New effect, unexpected	Hidden coupling or double write	Revert; read the code
Missing effect	A write or notify silently dropped	Revert
Unstable baseline	Scenario has ordering or timing leaks	Fix determinism first

Commit before.json next to the harness. A baseline you cannot diff later is just a log file.

What this ledger will not catch

Threads and asyncio. Interleaving order is not what you recorded.
C extensions and direct syscalls below open and subprocess.
Effects on boundaries you did not patch, such as a database driver or an SDK client.
Paths your single scenario never executes. Coverage is still your job.

It also cannot tell you whether the new order is correct. Only whether it changed.

Who should skip this

If you already have an integration suite that exercises the real boundary, use it. This harness exists for repos with no harness at all.

Skip it for a rewrite. A ledger gate protects mechanical moves, not redesigned flows. Skip it for concurrency-heavy code, where the sequence is genuinely not fixed.

Where a free model and a free server fit

Two places, both modest.

First, boundary discovery. Paste a module's imports into a free model and ask for candidate side-effect boundaries. Verify every candidate with grep, because models miss dynamic dispatch and mixins.

Second, isolation. Run the seed loop on a disposable Linux box instead of your laptop. The free server option covers a capture run for a small repo; it is not a reason to move your whole build there.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat free model access and the free server option as the operator's claims. I did not benchmark throughput, quotas, or hardware here, and none of the numbers above came from them.

Start with one scenario

One entry point, one boundary list, one baseline, one extraction. Then diff.

If you try this, draft the scenario and the boundary list first. The free tier is enough for both, and the diff will tell you the truth.

Triage Nondeterminism Before Golden Tests: A Probe for Messy Repos

Dakota Huang — Mon, 26 Oct 2026 18:11:31 +0000

A golden test that passes once proves almost nothing. It proves the harness ran. The second run is where messy repos fail: timestamps, temp paths, hash ordering.

Most teams meet this failure as a refactor bug. They revert a harmless change. Then they add sleep and hope.

This post takes the opposite order. Classify nondeterminism first. Pin only what survived classification. Then make the smallest safe change.

The artifact is a stdlib-only probe, a mask budget, and a mask-set growth gate. Both are checkable by CI.

MonkeyCode provides free model access and a free server option (operator-supplied). That matters here for one narrow reason: the probe runs the same command five times. Cheap repeated execution is the whole workload.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The failure mode, stated precisely

A characterization test captures current output. If current output is not a function of input, the capture is not a specification. It is a sample.

Three sources dominate in practice:

Clocks. Log lines, cache headers, and created_at fields.
Environment. Absolute paths, locale collation, PYTHONHASHSEED set ordering.
Allocation. PIDs, UUIDs, ephemeral ports, memory addresses.

Each one is legitimate output. None of it belongs in a golden file.

Step 1: Pick one entrypoint and time it

Characterization tests need a fast, deterministic entrypoint. A CLI command beats a function call here. It exercises the wiring too.

time python -m app.report --id 42 > /tmp/out.txt
wc -l /tmp/out.txt

Rules I apply before writing a single test:

Under 10 seconds, or you will not run it after every step.
One process, one exit code, stdout as the artifact.
No live network. Use a recorded fixture or a local stub.

If the entrypoint cannot meet these, fix that first. Probing a slow command just multiplies the wait.

Step 2: Run the determinism probe

The probe executes the same command under five controlled perturbations. Each run gets a fresh copy of the work tree. Then it classifies output line by line.

The script below is a template, not a battle-tested library. I have not executed it against your repo. Run it on a scratch copy first.

#!/usr/bin/env python3
# determinism_probe.py - stdlib only, Python 3.10+
import argparse, json, os, re, shutil, subprocess, sys, tempfile

PERTURB = [
    ("baseline", {}),
    ("tz", {"TZ": "Pacific/Kiritimati"}),
    ("hashseed-a", {"PYTHONHASHSEED": "0"}),
    ("hashseed-b", {"PYTHONHASHSEED": "4242"}),
    ("locale", {"LC_ALL": "C", "LANG": "C", "TZ": "UTC"}),
]

DEFAULT_PATTERNS = {
    "iso8601": r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})",
    "uuid4": r"[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}",
    "tmppath": r"/tmp/[A-Za-z0-9_./-]{4,}",
    "elapsed": r"\b\d+(?:\.\d+)?(?:ms|us|s)\b",
}

def run(cmd, extra, cwd):
    env = os.environ.copy()
    env.update(extra)
    proc = subprocess.run(cmd, shell=True, cwd=cwd, env=env,
                          capture_output=True, text=True, timeout=180)
    return proc.stdout

def mask(line, patterns):
    used, out = [], line
    for name, pat in patterns.items():
        new = re.sub(pat, f"<{name}>", out)
        if new != out:
            used.append(name)
            out = new
    return out, used

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--cmd", required=True)
    ap.add_argument("--workdir", default=".")
    ap.add_argument("--patterns", default="{}", help="JSON, extra candidates")
    ap.add_argument("--max-masked", type=float, default=0.25)
    ap.add_argument("--out", default="determinism_report.json")
    a = ap.parse_args()

    patterns = dict(DEFAULT_PATTERNS)
    patterns.update(json.loads(a.patterns))

    outputs = []
    for _, extra in PERTURB:
        tmp = tempfile.mkdtemp(prefix="probe-")
        tree = os.path.join(tmp, "repo")
        shutil.copytree(a.workdir, tree, symlinks=True)
        outputs.append(run(a.cmd, extra, tree))
        shutil.rmtree(tmp, ignore_errors=True)

    runs = [o.splitlines() for o in outputs]
    if len({len(r) for r in runs}) != 1:
        print("UNSTABLE: line count drifts", file=sys.stderr)
        sys.exit(2)

    rows, masked = [], 0
    for i in range(len(runs[0])):
        variants = {r[i] for r in runs}
        if len(variants) == 1:
            rows.append({"line": i, "status": "stable"})
            continue
        forms, used = set(), set()
        for v in variants:
            form, names = mask(v, patterns)
            forms.add(form)
            used.update(names)
        if len(forms) == 1 and used:
            masked += 1
            rows.append({"line": i, "status": "volatile",
                         "patterns": sorted(used), "variants": len(variants)})
        else:
            rows.append({"line": i, "status": "unstable",
                         "variants": len(variants),
                         "sample": sorted(variants)[:2]})

    fraction = masked / max(len(rows), 1)
    report = {"cmd": a.cmd, "runs": len(runs), "lines": rows,
              "masked_fraction": round(fraction, 4),
              "max_masked": a.max_masked}
    with open(a.out, "w") as f:
        json.dump(report, f, indent=2)

    unstable = sum(1 for r in rows if r["status"] == "unstable")
    print(f"lines={len(rows)} volatile={masked} unstable={unstable} "
          f"masked_fraction={fraction:.3f}")
    if unstable:
        sys.exit(2)
    if fraction > a.max_masked:
        sys.exit(3)

if __name__ == "__main__":
    main()

Run it, then read the summary line before anything else.

python determinism_probe.py --cmd "python -m app.report --id 42" --workdir .
python -c "import json;r=json.load(open('determinism_report.json'));\
print([x for x in r['lines'] if x['status']!='stable'][:5])"

The exit code is the gate. 0 means every line is stable or masked within budget. 2 means at least one line varies in a way no pattern explains. 3 means you are masking so much that the golden file proves nothing.

Step 3: Let a free model propose patterns, not verdicts

Some repos print shapes the default patterns miss. Composite request IDs. Signed cache keys. Base64 nonces.

This is where free model access is genuinely useful. Feed the model two diffing lines. Ask for candidate regexes only.

Prompt: Here are two outputs of the same command.
Volatile line A: req 7f3c-KJ9x-2 token ttl=59
Volatile line B: req 91ba-LmQ4-8 token ttl=58
Return JSON: {"name": "short_snake_case", "regex": "..."}.
Match the whole varying span. No anchors. No explanation.

Then the probe decides. A candidate is accepted only if applying it collapses every observed variant to one identical line.

python determinism_probe.py --cmd "python -m app.report --id 42" \
  --patterns '{"reqid": "[0-9a-f]{4}-[A-Za-z0-9]{4}-[0-9]"}' \
  --max-masked 0.25

Two hard rules keep this honest:

The model never writes assertions, expected values, or normalization functions. It writes candidate patterns only.
A generated pattern that does not reduce the variant set is discarded. No pattern is trusted because it looks plausible.

I ran the five perturbed executions on the free server option. Repeated short container runs are exactly the workload where a free tier is enough. The probe is CPU-light and stateless.

Step 4: Turn the report into the golden test

The report already contains the mask list per line. The test imports the same mask function, so production and test cannot drift.

# tests/test_report_golden.py - sketch
import json, subprocess
from pathlib import Path
from determinism_probe import mask, DEFAULT_PATTERNS

REPORT = json.loads(Path("determinism_report.json").read_text())


def normalize(text):
    rows = REPORT["lines"]
    out = []
    for row, line in zip(rows, text.splitlines()):
        if row["status"] == "volatile":
            pats = {k: DEFAULT_PATTERNS[k] for k in row["patterns"]}
            line, _ = mask(line, pats)
        out.append(line)
    return "\n".join(out)


def test_report_is_stable():
    proc = subprocess.run(["python", "-m", "app.report", "--id", "42"],
                          capture_output=True, text=True, check=True)
    assert normalize(proc.stdout) == Path("golden/report.txt").read_text()

Now run the suite three times in a row. It must be green every time. A test that needs luck is not a characterization test.

Step 5: The smallest safe change, with a growth gate

Save determinism_report.json before touching code. Change one thing. Re-run the probe. Compare.

cp determinism_report.json before.json
# apply the smallest safe change
python determinism_probe.py --cmd "python -m app.report --id 42" --workdir .
python - <<'PY'
import json
b = json.load(open('before.json'))['lines']
a = json.load(open('determinism_report.json'))['lines']
before = {r['line']: r.get('patterns', []) for r in b}
after = {r['line']: r.get('patterns', []) for r in a}
grew = [k for k in after if sorted(after[k]) != sorted(before.get(k, []))]
print('mask set changed on lines:', grew)
PY

The gate is simple. After a mechanical refactor, the mask set must not grow. A new volatile line means you moved an allocation, a clock read, or an unordered iteration into observable output. That is a behavior change, not a rename.

When the gate trips, decide explicitly. Either accept the new volatility and re-baseline, or shrink the change until the gate is quiet.

Decision table

Probe status	Meaning	Action
`stable`	Identical across all five runs	Pin literally in the golden file
`volatile` with one pattern	Varies only inside a known shape	Mask in test, keep surrounding text
`volatile` with two or more patterns	Line carries several independent variables	Split the line's format, or mask the whole field
`unstable`, small variant set	Ordering or set iteration	Sort before printing, then re-probe
`unstable`, line count drift	Branching on environment	Stop. Fix determinism before refactoring
`masked_fraction > 0.25`	Golden file mostly placeholders	Assert on parsed fields instead of text

Limitations and who should not use this

The probe compares lines by position. If output order is genuinely nondeterministic, it reports unstable and stops. That is intentional. Sort first, probe second.

It also assumes text output on stdout. Do not apply it to binary artifacts, snapshot images, or structured logs split across multiple files. Normalize those separately.

Skip this approach when the masked fraction would exceed a quarter of the output. At that point the golden file is a template, and template diffs catch nothing. Parse the output into fields and assert on the fields instead.

Finally, this does not replace a real test suite. It buys you a trustworthy before-picture for one entrypoint. It says nothing about whether current behavior is correct.

What you have at the end

Three files and two exit codes. A JSON report of what varies. A normalized golden file. A gate that turns a new source of volatility into a failing build.

That is enough to make a messy-repo refactor boring. Boring is the goal.

If you only copy one thing, copy the ordering: classify, pin, then change. If you want to try the loop on a free tier, the probe is small enough to run start to finish on the free server option before you pay for anything.