DEV Community: Dakota Wu

Start a Messy-Repo Extract With a Package Boundary Freeze

Dakota Wu — Sun, 20 Sep 2026 11:12:10 +0000

A billing service sat in one Python package for three years, with forty modules sharing helpers through relative imports and a few public functions that plugins actually called. An agent then opened a cleanup pull request that split billing/ into invoices/, tax/, and ledger/ because the folder names looked more professional. Unit tests stayed green, since they imported private helpers the same way the production code did, and the review looked like a tidy architecture win.

Two days later a plugin loader failed in staging because it still imported billing.totals.summarize. The extract had moved the function, deleted the old module, and left no shim. The tests never imported the public path that operators and plugins used. The repo was not missing tests in general; it was missing a freeze of the surfaces that other code was allowed to depend on.

This article treats that failure as a measurement problem, not a taste problem. The workflow below records package boundaries first, then allows only the smallest extract that those records can prove is quiet. Proposals and sample commands are labeled as such; they are a method you can run, not a claim about any particular production system.

Why green tests still ship a broken extract

AI-assisted refactors fail in messy repositories for a boring structural reason. Agents optimize for local readability, while messy packages encode contracts in import paths, __all__ lists, plugin entry points, and call shapes that no unit test names directly. A suite that only exercises internals will endorse a move that production still treats as a public URL.

Current coding-agent reviews often reward a large tree rewrite because the diff looks decisive. That is the same failure mode as shipping work that looks complete while the observable contract never got pinned. If the only evidence is “tests passed,” you have not measured the extract; you have measured the tests you already had.

A boundary freeze is a cheap characterization layer that sits above those tests. It does not prove the new design is better. It only answers whether the extract changed who can call what, and whether any caller still depends on the old path.

Artifact: freeze the public surface, then budget one change

The artifact is a JSON freeze file plus two tests. One test fails if an unexpected public name, export, or import edge appears. The other test fails if you touch more than one module pair in a single extract. Together they force the smallest safe change: move one symbol, keep a shim, and stop.

Proposed layout:

repo/
  billing/                 # the messy package under study
  tools/freeze_boundaries.py
  tests/test_boundary_freeze.py
  tests/test_change_budget.py
  artifacts/boundary_freeze.json

Step 1 — Inventory names and import edges

The following script is a proposed scanner, not a production crawler. It walks one package, records module-level public names, __all__ when present, and import edges that leave the package. Run it against the current tree before anyone moves files.

# tools/freeze_boundaries.py
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path


def iter_py_files(root: Path):
    for path in sorted(root.rglob("*.py")):
        if path.name == "__pycache__":
            continue
        yield path


def module_name(root: Path, path: Path) -> str:
    rel = path.relative_to(root.parent).with_suffix("")
    return ".".join(rel.parts)


def public_names(tree: ast.AST) -> list[str]:
    dunder_all = None
    names: list[str] = []
    for node in tree.body:
        if isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and target.id == "__all__":
                    if isinstance(node.value, (ast.List, ast.Tuple)):
                        dunder_all = [
                            elt.value
                            for elt in node.value.elts
                            if isinstance(elt, ast.Constant) and isinstance(elt.value, str)
                        ]
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            if not node.name.startswith("_"):
                names.append(node.name)
        if isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and not target.id.startswith("_"):
                    names.append(target.id)
    return sorted(set(dunder_all if dunder_all is not None else names))


def outgoing_imports(tree: ast.AST, self_mod: str, pkg: str) -> list[str]:
    edges: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module:
            target = node.module
            if node.level:
                parent = self_mod.split(".")
                target = ".".join(parent[:-node.level] + ([node.module] if node.module else []))
            if not target.startswith(pkg + ".") and target != pkg:
                edges.append(target)
        elif isinstance(node, ast.Import):
            for alias in node.names:
                if not alias.name.startswith(pkg):
                    edges.append(alias.name.split(".")[0])
    return sorted(set(edges))


def freeze(package_dir: Path) -> dict:
    pkg = package_dir.name
    modules = []
    for path in iter_py_files(package_dir):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        mod = module_name(package_dir, path)
        modules.append(
            {
                "module": mod,
                "public": public_names(tree),
                "imports_out": outgoing_imports(tree, mod, pkg),
            }
        )
    return {"package": pkg, "modules": modules}


def main(argv: list[str]) -> int:
    package_dir = Path(argv[1])
    out = Path(argv[2])
    payload = freeze(package_dir)
    out.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    print(f"wrote {out} modules={len(payload['modules'])}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Suggested first run:

python tools/freeze_boundaries.py billing artifacts/boundary_freeze.json
git add artifacts/boundary_freeze.json
git commit -m "chore: freeze billing package boundaries before extract"

Commit the freeze on its own. Mixing the inventory with the extract hides which names moved because of the refactor and which names the scanner newly discovered.

Step 2 — Fail the build when the freeze drifts

The characterization test below loads the committed freeze and compares it to a fresh scan. Equality is the point. If a cleanup diff silently drops billing.totals.summarize from the public list, this test is the first red signal, even when unit tests still import a private helper.

# tests/test_boundary_freeze.py
import json
from pathlib import Path

from tools.freeze_boundaries import freeze

FREEZE = Path("artifacts/boundary_freeze.json")
PACKAGE = Path("billing")


def test_public_surface_matches_committed_freeze():
    committed = json.loads(FREEZE.read_text(encoding="utf-8"))
    current = freeze(PACKAGE)
    assert current["package"] == committed["package"]
    assert current["modules"] == committed["modules"], (
        "Package boundary changed. Update the freeze in a dedicated commit "
        "or keep a shim on the old public path."
    )

Step 3 — Enforce a one-pair change budget

Large extracts hide the breaking rename among dozens of file moves. The second test reads git diff --name-only against the freeze commit and allows at most two production modules: the source file and the destination file. Test files and the freeze artifact itself are exempt. This is a process control, not a type system.

# tests/test_change_budget.py
import os
import subprocess
from pathlib import Path

BASE = os.environ.get("BOUNDARY_BASE", "HEAD1")  # set to the freeze commit in CI


def changed_paths() -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", BASE],
        text=True,
    )
    return [line for line in out.splitlines() if line]


def test_extract_touches_at_most_one_module_pair():
    prod = []
    for path in changed_paths():
        p = Path(path)
        if p.parts[:1] == ("tests",):
            continue
        if path.startswith("artifacts/"):
            continue
        if path.startswith("tools/"):
            continue
        if p.suffix == ".py":
            prod.append(path)
    assert len(prod) <= 2, (
        f"Extract exceeded the one-pair budget: {prod}. "
        "Move one symbol, keep a shim, land, then repeat."
    )

In CI, set BOUNDARY_BASE to the freeze commit SHA rather than HEAD1. The sample default is a placeholder so the file stays runnable in a local clone after you substitute the real ref.

Smallest safe change after the freeze is green

Once the freeze is committed and both tests pass on main, an extract is allowed only if it preserves the old public path. The pattern is mechanical:

Add the new module with the moved function and no other edits.
Keep the old module as a one-line shim that re-exports the same name.
Update the freeze in a follow-up commit that records both modules.
Delete the shim only after grep, plugins, and the freeze agree that nothing imports it.

Proposed shim, labeled as an example:

# billing/totals.py — kept until callers move
from billing.ledger.summarize import summarize

__all__ = ["summarize"]

That is a smaller diff than a folder taxonomy. It is also easier to revert, because failure still points at one symbol. If an agent returns a fifty-file “domain split,” reject the patch and ask for the shim form. The freeze file tells you whether the agent complied.

Decision table for the extract

Observation after the candidate diff	Freeze test	Budget test	Action
Public names unchanged, one pair of modules touched, shim present	pass	pass	Land the extract
Public name disappeared, no shim	fail	either	Restore the old path before any further moves
Public names unchanged, twelve files moved	pass	fail	Split the patch until one pair remains
New public names appear without a freeze update	fail	either	Record the names in a freeze-only commit
Only tests and docs changed	pass	pass	Out of scope for this protocol

Use the table during review. Do not argue about folder beauty until the first two columns are green.

Where a coding assistant belongs in this protocol

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Generating the first freeze by hand is tedious in a forty-file package, and agents are useful for drafting candidate characterization tests from that JSON. They are not useful as the authority that decides the extract is safe. If you try MonkeyCode here, use its free model access and free server option only to propose extra assertions against boundary_freeze.json, then keep the tests that actually fail on a shim-less move.

Treat those drafts as untrusted text until they run. An assistant that rewrites the freeze to match a large cleanup is doing the opposite of characterization. The committed JSON is the spec; the model is a helper that may add cases you forgot, such as __all__ gaps or re-exports hiding behind import *.

Limitations, and who should skip this

The scanner only sees module-level definitions and static import nodes. It will miss dynamically built module names, runtime getattr plugins, C extensions, and public behavior that exists only as CLI flags or HTTP routes. Teams whose real contract is a network schema need a different freeze, built from response fixtures rather than AST names.

The change-budget test can be bypassed by editing many symbols inside the two allowed files. Pair it with a human grep for deleted def names when a “small” file is actually a god module. The method also assumes a single package root. A monorepo with several messy packages needs one freeze per package, or the budget becomes meaningless.

Skip this protocol if you are deleting a prototype that has no external callers, or if you already have an explicit public API package with compatibility tests. Skip it if the goal is a semantic rewrite rather than a move: characterization of names will not catch a tax-rounding change inside a function that kept the same import path. In that case pin outputs, not boundaries.

The useful result is modest. After a freeze exists, a messy-repo extract becomes a sequence of one-symbol moves with shims, each proven against a committed surface. That is slower than a glamorous tree rewrite, and it is the pace at which plugins, operators, and later reviewers can still explain what changed.

Golden-File a Messy Reporter Before You Extract a Status Mapper

Dakota Wu — Fri, 18 Sep 2026 07:27:41 +0000

A weekly order reporter had grown into one 400-line script with three formats and a process-wide cache. Finance still parsed a side CSV by column position, while operators grepped a stdout banner that embedded a local timestamp. Cleanup pull requests kept failing for reasons that looked cosmetic until someone compared the files byte for byte. The useful move was not a repo-wide restyle, but a characterization suite that locked today's messy output and then one extracted mapping function.

This article treats that sequence as a worked example, not as a war story from a named production system. The reporter below is compact on purpose, and the tests are meant to be copied. If an agent later proposes a broader cleanup, the goldens decide whether the diff is still the smallest safe change.

Why inherited reporters punish large cleanups

Messy modules rarely hide a single defect that a rewrite would isolate cleanly. They hide several implicit contracts that nobody wrote down and that downstream tools still depend on.

Typical contracts in a reporter like this include the following load-bearing quirks:

Status aliases that treat "shipped " and "SHIPPED" as the same operator-facing label
A banner line that on-call grep recipes assume will appear on stdout
A sidecar CSV whose column order is positional for a spreadsheet import
A module-level cache that skips rereads when a batch wrapper calls the reporter twice

An agent that tidies the whole file will often normalize whitespace, rename helpers, and rewrite loops inside one diff. Each edit can look locally reasonable in isolation during review. Together they change more contracts than a reviewer can hold in working memory without a snapshot.

What counts as the smallest safe change

Treat the smallest safe change as a diff that satisfies all four of the checks below. If any check fails, split the work rather than arguing that the extra cleanup is obviously harmless.

It alters one named unit, such as a status mapper, and does not touch an I/O path.
Characterization goldens for stdout and sidecar files remain byte-identical after the diff.
Non-determinism is already pinned, including clock, working directory, locale, and cache seed.
Reviewers can explain the diff without opening a second production file for context.

If a proposed extract also trims trailing spaces inside labels, it is no longer the smallest safe change. Record that behavior change as a later ticket with its own expected output. Ship the extract only after the existing goldens stay green without edits.

Phase 1: freeze the messy reporter

The following module is a compact stand-in for an inherited reporter and is intentionally awkward. Do not clean aliases, caching, or banner formatting until characterization is committed and passing.

# order_reporter.py
from __future__ import annotations

import csv
from datetime import datetime
from pathlib import Path

STATUS_ALIASES = {
    "shipped": "Shipped",
    "SHIPPED": "Shipped",
    "shipped ": "Shipped",
    "queued": "Queued",
    "hold": "On Hold",
    "on-hold": "On Hold",
}

_CACHE: dict[str, list[dict[str, str]]] = {}


def load_rows(path: str) -> list[dict[str, str]]:
    if path in _CACHE:
        return _CACHE[path]
    with open(path, newline="") as handle:
        rows = list(csv.DictReader(handle))
    _CACHE[path] = rows
    return rows


def report(path: str, side_csv: str) -> None:
    rows = load_rows(path)
    stamp = datetime.now().strftime("%Y-%m-%d %H:%M")
    print(f"ORDER REPORT @ {stamp}")
    print(f"source={path}")
    written: list[dict[str, str]] = []
    for row in rows:
        raw = row.get("status") or ""
        stripped = raw.strip()
        label = STATUS_ALIASES.get(raw, STATUS_ALIASES.get(stripped, stripped or "Unknown"))
        sku = row.get("sku") or ""
        print(f"{sku}:{label}")
        written.append({"sku": sku, "status": label})
    with open(side_csv, "w", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=["sku", "status"])
        writer.writeheader()
        writer.writerows(written)

The mapper already contains a trap that tidy-up diffs love to erase. Lookup uses the unstripped status first and only then the stripped value, so a key with a trailing space can win. Characterization must keep that order, because a later cleanup of .strip() calls will change labels without touching any call site that looks important.

A characterization harness you can run

Pin time, clear the module cache, and capture both stdout and the sidecar file. Compare each captured document against a committed golden rather than against a rewritten expectation in the same commit.

# test_characterize_reporter.py
from __future__ import annotations

import io
from datetime import datetime as real_datetime
from pathlib import Path
from unittest.mock import patch

import order_reporter

FIXTURES = Path(__file__).parent / "fixtures"
GOLDENS = Path(__file__).parent / "goldens"


def _run_report(tmp_path: Path) -> tuple[str, str]:
    source = FIXTURES / "orders.csv"
    side = tmp_path / "side.csv"
    order_reporter._CACHE.clear()
    buf = io.StringIO()
    with patch("order_reporter.datetime") as mock_dt:
        mock_dt.now.return_value = real_datetime(2026, 9, 18, 9, 0)
        with patch("sys.stdout", buf):
            order_reporter.report(str(source), str(side))
    return buf.getvalue(), side.read_text(encoding="utf-8")


def test_stdout_matches_golden(tmp_path: Path) -> None:
    stdout, _ = _run_report(tmp_path)
    expected = (GOLDENS / "stdout.txt").read_text(encoding="utf-8")
    assert stdout == expected


def test_sidecar_matches_golden(tmp_path: Path) -> None:
    _, sidecar = _run_report(tmp_path)
    expected = (GOLDENS / "side.csv").read_text(encoding="utf-8")
    assert sidecar == expected

Use a fixture that exercises aliases, unknown labels, and the trailing-space key:

sku,status
A-100,shipped
B-200,shipped 
C-300,hold
D-400,unknown

Generate goldens once from current behavior, then commit those files as the contract. Do not edit goldens in the same change that extracts a helper, even when the new output looks cleaner to a reviewer.

python - <<'PY'
from pathlib import Path
import tempfile
from test_characterize_reporter import _run_report, GOLDENS

GOLDENS.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory() as tmp:
    stdout, sidecar = _run_report(Path(tmp))
    (GOLDENS / "stdout.txt").write_text(stdout, encoding="utf-8")
    (GOLDENS / "side.csv").write_text(sidecar, encoding="utf-8")
print("goldens written")
PY
pytest test_characterize_reporter.py -q

If pytest is not already a project dependency, run the same two equality checks as a short script. The important property is byte equality, not a snapshot library that normalizes whitespace or line endings by default. Platform newline drift will otherwise get promoted into a fake product contract.

Phase 2: extract one mapping function

After both goldens are green on the messy module, extract only the status lookup. Leave load_rows, the banner, the cache, and CSV writing untouched in that commit.

def map_status(raw: str | None) -> str:
    value = raw or ""
    stripped = value.strip()
    return STATUS_ALIASES.get(value, STATUS_ALIASES.get(stripped, stripped or "Unknown"))

Wire the helper into the existing loop with a one-line replacement and re-run the characterization tests immediately. If stdout or the sidecar drifts, revert the extract instead of repairing goldens. Do not fix forward by rewriting snapshots while the helper is still under review.

That extract is the entire first refactor on purpose. Resist adding file-wide type hints, renaming _CACHE, or sorting SKUs, because those edits expand the blast radius. Each of those ideas can become a later ticket with its own golden review once this seam is isolated.

Decision table for later cuts

Keep a short table next to the harness so humans and agents do not bundle obvious fixes once the file is open.

Candidate change	Safe as the next diff?	Why
Extract `map_status` only	Yes	Pure function; existing goldens stay identical
Strip keys inside `STATUS_ALIASES`	No	Changes how `"shipped "` is labeled
Remove the module cache	No	Alters repeated-read behavior in batch wrappers
Drop the banner timestamp	No	Operators grep the line; pin time instead
Reorder sidecar columns	No	Spreadsheet import is positional
Normalize aliases in a later ticket	Yes, after new goldens	That is an explicit behavior change

The table is the review checklist, not decoration. A diff that touches two rows in that table is already larger than the smallest safe change.

Where a disposable agent workspace helps

Generating characterization fixtures by hand is slow when the messy module has many branches and alias keys. A coding agent can propose extra rows for the input CSV after it reads STATUS_ALIASES and the golden-generation script. The reviewer still decides which rows encode real contracts and which rows are synthetic noise.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option, and those two properties matter here only as a workspace constraint. The free server is a place to generate fixtures and run the harness without writing goldens into a laptop checkout that already has dirty files. Copy the resulting fixtures/ and goldens/ directories back into review, then discard the session rather than merging an in-place restyle.

The method does not depend on any particular product, and a second local clone with a clean virtualenv works the same way. The agent is optional scaffolding around fixture discovery. The goldens remain the contract that makes the later extract auditable.

Limitations and who should skip this

This approach is a poor fit when behavior is undefined on purpose, such as a prototype that must change daily and has no downstream parser. It is also the wrong tool for cryptographic code, concurrent writers, or modules whose useful output is a GUI rather than text. Golden files freeze encoding and newline details, so they will flake if one machine writes \r\n and another writes \n.

Pin newline="" and UTF-8 at every file boundary, or the harness will encode platform noise as a product contract. Do not use characterization-then-extract as cover for leaving secrets or customer rows in fixtures. Sample CSVs should be synthetic, and if the messy reporter talks to a network, fake the transport before you snapshot anything.

Teams that already have fast unit tests around pure functions do not need this ceremony for those functions. Save it for inherited scripts where tests never existed and the I/O itself is the product that finance and operators still consume.

Close the loop on the first cut

A messy repo becomes safer when the first cleanup diff is boring and easy to revert. Lock stdout and sidecar bytes, extract one mapper, and stop before the file looks tidy. The next implicit contract can wait for its own goldens cycle, including cache behavior and banner format, because those are separate seams. That sequence is slower than a generated restyle, and it is the sequence that keeps a spreadsheet import aligned with what operators still grep in logs.

Build an Environment Ledger Before Cleaning a Tangled CLI

Dakota Wu — Thu, 17 Sep 2026 05:35:13 +0000

A cleanup pull request looked harmless until the nightly invoice job wrote files into the wrong directory. The refactor had replaced os.getcwd() with a packaged resource path, and every relative open() followed it. Tests still passed because they ran from the repository root, which matched the old accidental contract. The failure appeared only on a scheduler whose working directory was /var/lib/batch rather than the checkout.

Messy repositories accumulate these environmental contracts for years without documenting them as interfaces. Coding agents and human cleanups both treat cwd, locale, timezone, and temp directories as leftover implementation details. They are not details when a batch job, a cron entry, or a container WORKDIR already depends on the old accident. The practical sequence is to characterize those surfaces first, then apply the smallest safe change.

Hidden environment contracts in tangled checkouts

Cleanup diffs usually chase names, helpers, and folder structure rather than observable input and output. That bias is reasonable when the module is already a pure function of explicit arguments. Most long-lived scripts are not pure, because they read os.environ, format dates, and write beside the process working directory. An agent that merely modernizes paths can preserve unit tests while changing production file placement.

Three failure modes show up repeatedly in invoice, ETL, and report jobs:

Relative file I/O that silently follows cwd instead of an explicit base directory.
Locale-sensitive decimal separators, month names, and CSV headers that differ under C versus en_US.UTF-8.
Temporary files that land in $TMPDIR on a laptop and in /tmp inside a scheduler image.

None of these outcomes require a malicious model or an unusually large rewrite. They require an unpinned environment plus a cleanup that looks locally smaller than its production effect. Current public debate about agents outgrowing tests is adjacent here: the tests did not shrink, they never observed cwd, locale, or temp placement at all.

What to record before anyone rewrites a module

Treat the process environment as a public API even when the original authors never named it that way. A side-effect ledger is a JSON document that captures the surfaces a later diff is not allowed to change without an explicit decision. The ledger is not a substitute for domain tests; it is a fence around accidental behavior that unit tests rarely exercise.

Record at least the following fields for each entry-point script:

process cwd and the resolved path of every relative open, mkdir, and glob
TZ, LC_ALL, LANG, LC_TIME, LC_NUMERIC, and the locale.getlocale() tuple
tempfile.gettempdir(), TMPDIR, TMP, TEMP, and a sample NamedTemporaryFile prefix
umask, file-creation mode of one artifact, and whether output paths are absolute
a hash of stdout, stderr, and the sorted list of written file paths

Keep secrets out of the ledger on purpose. Redact tokens, connection strings, and customer payloads before the file is committed or copied onto any shared machine.

A proposed characterization harness

The following Python example is a proposed, unexecuted sketch for a small CLI module. It is meant to run against a frozen fixture directory, not against production data. Adjust the command list and fixture layout for the messy repository under review.

#!/usr/bin/env python3
"""Proposed characterization harness for environment-sensitive CLIs."""
from __future__ import annotations

import hashlib
import json
import locale
import os
import subprocess
import sys
import tempfile
from pathlib import Path

LEDGER_VERSION = 1


def read_umask() -> str:
    current = os.umask(0)
    os.umask(current)
    return oct(current)


def snapshot_env(cwd: Path) -> dict:
    return {
        "cwd": str(cwd.resolve()),
        "tz": os.environ.get("TZ"),
        "lang": os.environ.get("LANG"),
        "lc_all": os.environ.get("LC_ALL"),
        "lc_time": os.environ.get("LC_TIME"),
        "lc_numeric": os.environ.get("LC_NUMERIC"),
        "locale": list(locale.getlocale()),
        "tmpdir": tempfile.gettempdir(),
        "tmp_env": {
            "TMPDIR": os.environ.get("TMPDIR"),
            "TMP": os.environ.get("TMP"),
            "TEMP": os.environ.get("TEMP"),
        },
        "umask": read_umask(),
        "python": sys.version.split()[0],
    }


def run_subject(command: list[str], cwd: Path, env: dict) -> dict:
    proc = subprocess.run(
        command,
        cwd=cwd,
        env=env,
        capture_output=True,
        text=True,
        check=False,
    )
    written = sorted(
        str(p.relative_to(cwd)) for p in cwd.rglob("*") if p.is_file()
    )
    stdout_hash = hashlib.sha256(proc.stdout.encode("utf-8")).hexdigest()
    stderr_hash = hashlib.sha256(proc.stderr.encode("utf-8")).hexdigest()
    tree_src = "\n".join([str(proc.returncode), stdout_hash, stderr_hash, *written])
    return {
        "returncode": proc.returncode,
        "stdout_sha256": stdout_hash,
        "stderr_sha256": stderr_hash,
        "tree_sha256": hashlib.sha256(tree_src.encode("utf-8")).hexdigest(),
        "written": written,
    }


def main() -> None:
    fixture = Path("fixtures/invoice_batch").resolve()
    cwd = fixture / "work"
    out_dir = Path("ledgers")
    out_dir.mkdir(exist_ok=True)
    env = os.environ.copy()
    env.update(
        {
            "TZ": "UTC",
            "LC_ALL": "C",
            "LANG": "C",
            "TMPDIR": str(fixture / "tmp"),
        }
    )
    ledger = {
        "version": LEDGER_VERSION,
        "env": snapshot_env(cwd),
        "run": run_subject(
            [sys.executable, "export_invoices.py", "--date", "2026-09-16"],
            cwd=cwd,
            env=env,
        ),
    }
    path = out_dir / "export_invoices.v1.json"
    generated = out_dir / "export_invoices.generated.json"
    generated.write_text(json.dumps(ledger, indent=2, sort_keys=True))
    if path.exists():
        previous = json.loads(path.read_text())
        if previous != ledger:
            raise SystemExit("characterization drift: environment contract changed")
        print("ledger unchanged")
        return
    path.write_text(json.dumps(ledger, indent=2, sort_keys=True))
    print(f"wrote {path}")


if __name__ == "__main__":
    main()

Pin the process environment from the shell as well, because a developer laptop often leaks LANG and TMPDIR into otherwise careful Python. The command below is a proposed isolation wrapper, not a production scheduler definition.

mkdir -p fixtures/invoice_batch/work fixtures/invoice_batch/tmp ledgers

env -i \
  PATH="$PATH" \
  HOME="$HOME" \
  TZ=UTC \
  LC_ALL=C \
  LANG=C \
  TMPDIR="$(pwd)/fixtures/invoice_batch/tmp" \
  python3 tools/env_ledger.py

python3 tools/env_ledger.py
git diff --stat -- ledgers export_invoices.py

A tiny proposed pytest check can fail the build when a cleanup regenerates a different ledger. Keep that test on the committed JSON, and do not overwrite the committed file from CI.

# proposed: tools/test_env_ledger.py
import json
from pathlib import Path


def test_ledger_matches_committed_snapshot():
    committed = json.loads(Path("ledgers/export_invoices.v1.json").read_text())
    generated = json.loads(Path("ledgers/export_invoices.generated.json").read_text())
    assert generated == committed

If the ledger is unchanged and git diff --stat stays inside the intended files, the cleanup remains a candidate. If either check moves, stop and decide whether the new behavior is an intended contract change. Do not refresh the JSON to make a large rewrite look green.

Decision table for the smallest safe change

Use the table below before accepting an agent diff or a human cleanup on a tangled script.

Observation	Allowed next step	Disallowed next step
Ledger matches; diff touches only path constants in one file	Replace cwd-relative opens with an explicit `--base-dir`	Rename modules, extract helpers, or reformat adjacent jobs
Ledger matches; diff also reorders imports and comments	Keep the path change; revert the noise	Bundle style and behavior in one review
Ledger drifts on tmp paths only	Pin `TMPDIR` in the job spec, then retry	Accept "we use tempfile correctly now" without a pin
Ledger drifts on stdout hashes because dates shifted	Freeze `TZ=UTC` and `LC_TIME=C`, then retry	Patch the golden hash to match the new local clock
Ledger drifts on written relative paths	Treat cwd as an API; add a test that `chdir()`s	Continue extracting functions on top of the drift
Diff spans more than one job or shared util	Split the change; re-run the ledger per entry point	"While we are here" refactors across the batch folder

The smallest safe change is the diff that preserves the ledger and touches one contract at a time. Extracting a helper is a second change, even when the helper looks obviously right in isolation. That split is the entire method: characterize, change one surface, re-read the ledger, then stop.

Where a free coding agent belongs in this loop

An agent is useful after the ledger exists, not before the first characterization run. It can propose additional relative-open sites, draft a --base-dir flag, or generate a second fixture that starts in a different cwd. It should not be the process that decides the ledger is stale or that a golden hash deserves a rewrite.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to run the harness in a throwaway workspace instead of on a laptop that already holds production exports. That separation matters because characterization fixtures can include path layouts that resemble real jobs even after payloads are redacted.

A practical loop on a free server looks like the following numbered sequence.

Copy a redacted fixture tree, not the production working directory.
Run the ledger command until it writes a stable JSON document.
Ask the model only for the smallest path-explicit change in one file.
Re-run the ledger and inspect git diff --stat before any further prompt.

If the model returns a multi-file rewrite, reject the whole patch and restate the file budget in the next request. Free capacity does not change the review rule; it only keeps the experiment off the machine that mounts customer disks. A throwaway workspace is useful here when the alternative is running fixture trees beside real export directories.

Limitations

This harness does not prove functional correctness of invoice math, tax rounding, or SQL results. It only detects environmental drift that unit tests often miss because they share the developer's cwd and locale. Hashing stdout will fail on intentional logging changes, which then requires a split between semantic output and diagnostic streams. Parallel jobs that race on temporary names need extra pinning, such as a deterministic prefix, or the ledger will flap without a real regression. Container images that reset LANG at entry can disagree with a laptop run even when the Python file itself is unchanged.

The umask field is easy to mis-snapshot if another thread mutates the mask during the run. Treat that field as advisory unless the job creates files whose mode is part of an operational agreement. Do not upload ledgers that still contain customer identifiers in written path names or in captured standard output.

Who should not use this approach

Skip this workflow when the program is already a pure function with injected clocks, paths, and locales. Skip it when the repository cannot provide a redacted fixture and the alternative is copying live data onto a shared server. Skip it for cryptographic code, access-control changes, or anything where a matching stdout hash is the wrong safety property. Teams that already maintain formal acceptance tests should keep those tests, because the ledger is a fence rather than a specification.

Operators who cannot inspect git diff --stat before merge should not delegate cleanup to any model, paid or free. The method assumes a human still owns the environmental contract after the agent returns a patch.

Cleanup diffs fail in production because messy repositories encode cwd, locale, and temp paths as silent APIs. Pin those surfaces with a ledger, allow one contract-preserving change, and only then consider a larger rewrite. A free remote workspace can host the fixture run, but it cannot replace the decision table that keeps the first diff small.

Pin a Characterization Suite Before You Accept the First Refactor Diff

Dakota Wu — Wed, 16 Sep 2026 03:51:57 +0000

A four-year-old invoice aggregator often lives in one nine-hundred-line module with almost no tests. A teammate pastes that file into a coding agent and asks the model to clean the structure before Friday's export. The agent returns a confident diff that extracts helpers, reorders branches, and quietly changes a rounding path near tax-exempt rows. Reviewers see smaller functions and miss that weekly totals no longer match last quarter's frozen finance CSV.

That failure is not primarily an intelligence problem in the model or the reviewer. It is a missing characterization problem sitting in front of the first refactor commit. Observed behavior was never pinned against real fixtures, so every rename looked like progress until payroll compared two files. This article walks through a workflow that records current behavior first, then accepts only the smallest safe change that still satisfies those recordings.

Current developer discussion keeps returning to agents that write most of a diff. The engineering question underneath those threads is narrower and older. Which observable contracts must remain identical after someone, human or model, touches a tangled module?

Why "clean it up" destroys accidental contracts

Legacy modules accumulate three kinds of knowledge that comments almost never capture in a reviewable form.

Branch order that accidentally encodes precedence between discounts, credits, and tax flags.
Implicit rounding, string coercion, and locale formatting at the CSV and money edges.
Side effects that fire only for certain customer flags, backfill dates, or retry counts.

An agent trained to reduce complexity will happily collapse those accidents into a tidier control flow. Collapse can be the correct long-term design when the accidents were never business rules. Collapse can also move a one-cent discrepancy into a different fiscal week without failing a linter.

Vibe-led refactoring is popular because the resulting file is easier to read in a pull request. The missing artifact is a characterization suite that fails when observed outputs move, even if cyclomatic complexity improved. Until that suite exists, a smaller function is not evidence of a safer change.

Step 1 — Wrap the current entry point before anyone edits it

Do not start by asking a model to rewrite the file in place. Start by wrapping the current public entry point and recording outputs from fixtures you already possess. Treat the recordings as a proposal until they run twice with identical bytes on the same revision.

# characterize_invoice.py
# Labeled example: unexecuted against your ledger; adapt paths and types.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any, Callable

GOLDEN_DIR = Path("tests/golden/invoice_aggregator")


def dump_canonical(value: Any) -> str:
    return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)


def record_case(name: str, entry: Callable[..., Any], **kwargs: Any) -> Path:
    GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
    payload = entry(**kwargs)
    target = GOLDEN_DIR / f"{name}.json"
    target.write_text(dump_canonical(payload) + "\n", encoding="utf-8")
    return target


def assert_unchanged(name: str, entry: Callable[..., Any], **kwargs: Any) -> None:
    target = GOLDEN_DIR / f"{name}.json"
    if not target.exists():
        raise FileNotFoundError(f"missing golden file for {name}; record it first")
    observed = dump_canonical(entry(**kwargs)) + "\n"
    expected = target.read_text(encoding="utf-8")
    if observed != expected:
        raise AssertionError(f"characterization drift in {name}")

Record from production-shaped fixtures, not from synthetic rows invented to make the function look pure. Include at least one month-end file, one tax-exempt customer, one zero-total refund, and one retry duplicate. If the module writes a file or a queue message, capture that artifact in the same canonical dump instead of asserting only the return value.

# Record once against the untouched revision.
python -c "from characterize_invoice import record_case; from invoice_agg import run; record_case('fy25_w36', run, path='fixtures/fy25_w36.csv')"

# Confirm the recording is stable before any refactor branch exists.
python -c "from characterize_invoice import assert_unchanged; from invoice_agg import run; assert_unchanged('fy25_w36', run, path='fixtures/fy25_w36.csv')"

If the second command fails on an untouched revision, you do not yet have a characterization suite. You have a flaky exporter, a clock dependency, or an unordered dict that still needs pinning. Fix recording stability before you discuss helpers, types, or agent-generated patches.

Step 2 — Promote recordings into tests that own the merge gate

A golden file that nobody runs is a souvenir. Promote each recording into a test that CI, a laptop, and a free remote shell can execute with the same command. Keep the test body boring so failures point at behavior, not at framework cleverness.

# tests/test_characterize_invoice.py
# Labeled example: add cases only from fixtures you can legally store.
import pytest
from characterize_invoice import assert_unchanged
from invoice_agg import run

CASES = [
    "fy25_w36",
    "tax_exempt_retail",
    "zero_total_refund",
    "duplicate_retry_same_day",
]

@pytest.mark.parametrize("name", CASES)
def test_aggregator_matches_pinned_behavior(name: str) -> None:
    fixture = f"fixtures/{name}.csv"
    assert_unchanged(name, run, path=fixture)

python -m pytest tests/test_characterize_invoice.py -q
git add tests/golden tests/test_characterize_invoice.py characterize_invoice.py
git commit -m "test: pin invoice aggregator characterization before refactor"

That commit should contain no production edit. Reviewers then have a baseline that is older than the cleanup story. Any later diff that claims to be behavior-preserving must keep this command green without rewriting golden files in the same change.

Step 3 — Decide what counts as the smallest safe change

Characterization makes large rewrites detectable. It does not by itself stop a model from proposing a two-hundred-line aesthetic restyle. Define a budget before anyone opens the agent chat, and reject patches that exceed the budget even when tests stay green.

Change class	Allowed in the first refactor PR	Characterization must stay green	Notes
Rename a private helper without moving logic	Yes	Yes	Keep the public entry signature frozen.
Extract one already-straight-line block	Yes	Yes	One new function, one call site, no new branches.
Reorder independent statements	No	Would hide precedence bugs	Split into a later PR with a new fixture.
Change rounding, sorting, or CSV dialect	No	Would lock a product change	That is a behavior change, not a refactor.
Rewrite control flow to "simplify" nested flags	No	High false-green risk	Requires new intent tests, not only goldens.
Update golden files in the same PR	No	Gate would be meaningless	Record a new baseline only after product sign-off.

The table is a process artifact, not a style preference. Teams that skip the budget still get green characterization tests after a rewrite that no single reviewer can hold in working memory. Smallest-safe-change means one behavior-preserving mechanical edit, not one pull request that happens to compile.

A useful local check is a diff budget measured after the characterization commit.

# Proposal: fail the branch if the production hunk is larger than one function extract.
git diff main --stat -- invoice_agg.py
git diff main -- invoice_agg.py | wc -l

If invoice_agg.py grew a new abstraction layer, a new config object, and a new error hierarchy, the change is no longer the smallest safe one. Put the extra design on a backlog that starts from green goldens, not from a single agent transcript.

Step 4 — Ask a model only for the budgeted hunk

Once the suite is green on the baseline revision, a coding model becomes a patch proposer instead of an unsupervised editor. The prompt should include the characterization command, the diff budget, and the forbidden change classes from the table. It should not include a vague request to make the module clean, modern, or more functional.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is useful here because the task is a constrained hunk, not a full rewrite contest. The free server option is useful because the characterization suite should run on a second machine that does not share your laptop's hidden environment variables, working directory dirt, or cached bytecode.

A practical loop looks like the following sequence, and it stays useful if you swap the editor for any other assistant.

Check out the characterization commit and confirm pytest is green twice.
Ask the model for one extract or rename that does not alter golden bytes.
Apply the hunk on a branch that cannot rewrite files under tests/golden/.
Run the same pytest command locally and on the free server checkout.
Reject the branch if either environment drifts or the production diff exceeds the budget.

# On the remote checkout, use a clean tree and the same fixtures.
git status --porcelain
python -m pytest tests/test_characterize_invoice.py -q --maxfail=1

Do not paste secrets into that remote shell, and do not treat a green remote run as product approval. Treat it as independent evidence that the golden files are not an accident of one developer's machine. If local passes and remote fails, you likely recorded time zones, absolute paths, or hash seed order instead of business output.

Step 5 — Keep golden updates on a separate intent track

Some teams discover during characterization that the current module is wrong. That discovery is valuable and it is also a different change. A refactor PR that "fixes" rounding while extracting a helper mixes intent until nobody can say which assertion still matters.

Split the work:

Track A freezes today's bytes, including known one-cent oddities, under characterization tests.
Track B writes an explicit intent test for the corrected rounding, with a product owner named in the ticket.
Track C, only after B merges, regenerates golden files in a commit that contains no structural rewrite.

Agents collapse A, B, and C because the combined diff looks decisive. Reviewers should uncollapse them. Characterization is a lock on the past. Intent tests are a claim about the future. Refactor diffs should not carry both stories at once.

Limitations of this workflow

Characterization tests freeze accidents as well as rules. If the module formats money with banker's rounding on Tuesdays and truncation on retries, the suite will protect that contradiction. That is correct for a first refactor and dangerous if the team later confuses green goldens with a specified domain model.

The suite is only as honest as the fixtures. Internal CSV samples miss the partner file that uses a BOM, a blank trailer line, or a duplicated invoice id. If legal or privacy rules block production-shaped fixtures, this method under-approximates and should not be sold as full coverage.

Canonical JSON dumps also hide some faults. They will not catch performance regressions, lock order, or extra network calls unless those effects are included in the recorded payload. Teams that need those properties should add a side-channel log as a separate, later pin, not as scope creep in the first PR.

Free model drafts can still smuggle behavior changes inside an extract that looks mechanical. The budget and the golden gate reduce that risk. They do not remove the need for a human to read the hunk against the decision table.

Who should not use this approach

Do not use characterization-first refactoring when the current behavior is an incident in progress and the correct output is already known. Write the intent test and the fix; do not spend a day gilding the broken baseline. Do not use it on modules whose outputs cannot be stored, even redacted, without leaking customer data.

Do not use a free remote server as the only copy of fixtures that contain production extracts. Do not use the smallest-safe-change budget as a reason to postpone deleting dead code that has no callers and no goldens. Dead code with no characterization is a deletion candidate, not a refactor candidate.

Skip this workflow if the file is already covered by stable, high-signal unit tests that encode rounding, order, and side effects. In that case the characterization layer is duplicate evidence. Add it only when those tests do not exist or have been asserting mocks instead of bytes.

The smallest safe change after a green characterization suite is usually unglamorous. It is also the first refactor a later agent can extend without inventing a new weekly total. If you want a constrained place to practice that loop with free model access and a free server checkout, MonkeyCode is one option to try on a non-secret module that already has fixtures you can pin.

Retry Amplification: A Circuit Breaker for Batch Model Jobs

Dakota Wu — Mon, 14 Sep 2026 21:36:42 +0000

A nightly job walked a folder of 4,000 JSON files and called a model once per file. At 02:14 the upstream started returning 429s. By 02:40 the job had issued roughly 61,000 calls and written zero results. The queue was not slow. It was multiplying itself.

That shape of incident is common in batch model work, and it rarely comes from a broken model. It comes from a retry loop that treats every error as temporary, including errors that mean "stop". The fix is not a bigger retry budget. It is a breaker that refuses to make the next call.

Retry amplification, in three numbers

Amplification is easy to measure after the fact and easy to predict before it. Three numbers tell you whether your job can melt itself:

Calls per item. One call per item is healthy. Eight calls per item means your loop is the load generator.
Wall-clock per item. If p95 time per item doubles while p50 stays flat, a subset of items is stuck retrying.
Identical error signatures. Twenty different errors are noise. One error repeated 4,000 times is a hard stop signal.

The third number is the one most teams never collect, because logs store messages, not normalized signatures.

Why free access changes your math but not your failure modes

Using a free tier, like the free model access and free server option MonkeyCode describes, removes the invoice from the loop. It does not remove wall-clock limits, rate limits, or the cost of your own wasted runs. A runaway loop against free capacity still burns an afternoon.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The breaker below is provider-neutral on purpose. It counts failures, normalizes them, and stops. Whether the call behind it is billed or free only changes how loudly you notice a bad night.

The state machine, in one table

A batch breaker has three states. Write them down before you write code, because half-open logic is where most hand-rolled breakers go wrong.

State	`allow()`	On success	On failure
CLOSED	always true	reset counters, stay CLOSED	increment; trip at threshold
OPEN	false until cooldown expires	n/a	extend cooldown, capped
HALF-OPEN	true exactly once	close and reset cooldown	reopen immediately

The subtle rule is the last row: a half-open breaker grants one probe, not unlimited probes with a timer attached. If that one probe fails, the breaker reopens without waiting for the threshold again.

A breaker with an injectable clock

Persist state to a file so a restart cannot silently reset a tripped breaker. Inject the clock so tests never sleep.

# batch/breaker.py
"""Stop making calls before the retry loop stops you."""
from __future__ import annotations

import hashlib
import json
import pathlib
import re
import time

_DIGITS = re.compile(r"\d+")


def signature(exc: BaseException) -> str:
    """Normalize an error so identical failures share one fingerprint."""
    text = f"{type(exc).__name__}: {exc}".lower()
    return hashlib.sha256(_DIGITS.sub("n", text).encode()).hexdigest()[:12]


class Breaker:
    def __init__(self, path, *, threshold=3, base_cooldown=30.0,
                 max_cooldown=600.0, repeat_trip=2, clock=time.monotonic):
        self.path = pathlib.Path(path)
        self.threshold = threshold
        self.base_cooldown = base_cooldown
        self.max_cooldown = max_cooldown
        self.repeat_trip = repeat_trip
        self.clock = clock
        self.state = self._load()

    def _default(self) -> dict:
        return {"failures": 0, "opened_at": None, "cooldown": self.base_cooldown,
                "half_open": False, "last_signature": None,
                "signature_repeats": 0, "trips": 0}

    def _load(self) -> dict:
        if self.path.exists():
            return json.loads(self.path.read_text())
        return self._default()

    def _save(self) -> None:
        self.path.write_text(json.dumps(self.state, indent=2, sort_keys=True))

    def allow(self) -> bool:
        s = self.state
        if s["opened_at"] is None:
            return True
        if s["half_open"]:
            return False                      # one probe only
        if self.clock() - s["opened_at"] < s["cooldown"]:
            return False
        s["half_open"] = True
        self._save()
        return True

    def record_success(self) -> None:
        self.state = self._default()
        self._save()

    def record_failure(self, exc: BaseException) -> None:
        s = self.state
        sig = signature(exc)
        s["signature_repeats"] = (
            s["signature_repeats"] + 1 if sig == s["last_signature"] else 1
        )
        s["last_signature"] = sig
        s["failures"] += 1

        storm = s["signature_repeats"] >= self.repeat_trip
        if s["half_open"] or storm or s["failures"] >= self.threshold:
            if s["trips"] > 0:
                s["cooldown"] = min(s["cooldown"] * 2, self.max_cooldown)
            s["opened_at"] = self.clock()
            s["trips"] += 1
            s["half_open"] = False
        self._save()

Signatures, not messages

signature() strips digits before hashing, so attempt 7 failed and attempt 8 failed collapse into one fingerprint. That is what lets repeat_trip catch an error storm long before the plain failure count reaches its threshold. Pick the normalization you can defend: identifiers and UUIDs usually belong in the stripped set, status codes usually do not.

Tests that never sleep

A fake clock keeps the suite fast and makes cooldown behavior exact.

# tests/test_breaker.py
from batch.breaker import Breaker


class Clock:
    def __init__(self): self.now = 0.0
    def __call__(self): return self.now
    def advance(self, seconds): self.now += seconds


def make(tmp_path, clock, **kw):
    return Breaker(tmp_path / "breaker.json", clock=clock, **kw)


def test_trips_after_threshold(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=3)
    for _ in range(3):
        assert b.allow()
        b.record_failure(RuntimeError("upstream 429"))
    assert b.allow() is False


def test_half_open_grants_exactly_one_probe(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=1, base_cooldown=10.0)
    assert b.allow()
    b.record_failure(RuntimeError("boom"))
    clock.advance(9)
    assert b.allow() is False
    clock.advance(1)
    assert b.allow() is True
    assert b.allow() is False


def test_success_closes_and_resets(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=2)
    b.allow(); b.record_failure(RuntimeError("boom"))
    b.allow(); b.record_failure(RuntimeError("boom"))
    clock.advance(30)
    assert b.allow()
    b.record_success()
    assert b.state["trips"] == 0


def test_error_storm_trips_before_threshold(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=50, repeat_trip=2)
    b.allow(); b.record_failure(TimeoutError("attempt 7 failed"))
    b.allow(); b.record_failure(TimeoutError("attempt 8 failed"))
    assert b.allow() is False

python -m pytest -q tests/test_breaker.py

Wiring it into a batch runner

The runner holds one rule: check allow() before every call, and never let an exception escape into the next item.

# batch/run.py
import pathlib
from batch.breaker import Breaker

breaker = Breaker(".breaker.json", threshold=3, repeat_trip=2)


def handle(item: pathlib.Path) -> None:
    ...  # your model call, then write the result atomically


def main() -> int:
    done = calls = 0
    for item in sorted(pathlib.Path("inbox").glob("*.json")):
        if not breaker.allow():
            print(f"open: {calls} calls, {done} items done")
            return 2
        calls += 1
        try:
            handle(item)
        except Exception as exc:
            breaker.record_failure(exc)
            continue
        done += 1
        breaker.record_success()
    print(f"closed: {calls} calls, {done} items done")
    return 0

Exit code 2 and the calls/done pair are the whole postmortem. A cron wrapper can page on that code and archive .breaker.json with the run logs.

Two operational details

Delete .breaker.json deliberately. A job that resumes six hours later should not inherit a stale cooldown, so make resetting it an explicit step in your runbook.
Probe with the cheapest item. When half-open fires, feed it a tiny input first. A probe that OOMs tells you nothing about the upstream.

Health metrics worth paging on

calls_per_completed_item above 1.2 over 15 minutes.
trips_total increasing after the job was previously stable for a week.
signature_repeats for the current run reaching repeat_trip - 1.

If a breaker never trips across months of runs, the threshold is decoration. Lower it in a staging job and confirm the open path still works before you trust it in production.

Decide in ten seconds

Situation	Use a breaker?	Why
Batch job, idempotent items, one upstream	Yes	Safe to stop and resume later
Interactive request path	No	The user needs an answer, not a stopped service
Items that mutate shared state mid-run	Not yet	Make them idempotent first
One-shot script, under a minute	No	Nothing to protect
Multiple upstreams with different limits	Per upstream	One breaker per dependency

Limits and who should skip this

A breaker suppresses the symptom, not the cause. If the upstream is genuinely down for an hour, you get a quiet job instead of a loud one, and quiet jobs get forgotten. Pair the breaker with an alert, or you have built a silencer.

It also hides partial progress. A run that stops at 900 of 4,000 items looks successful unless you report done explicitly, which is why the runner prints both numbers.

Skip this approach if your items are not idempotent, if retries are the only way you recover from a known flaky dependency, or if a human is waiting on each result. In those cases a per-request timeout and a retry budget are the simpler tools.

What to run this week

Add breaker.py and the four tests above to your smallest batch job, then point a staging run at an upstream you can break on purpose. Run the job on disposable infrastructure if you do not want a tripped breaker in your working tree; the free server option in MonkeyCode is one place to try that without touching your laptop's state.

Confirm one thing before you walk away: when the breaker opens mid-run, does your job resume cleanly from the items it already finished? If the answer is no, fix that first. A breaker on top of non-idempotent work just makes the outage smaller and harder to see.

Pin Sort Keys and Rounding Before You Extract a Helper From a Totals Loop

Dakota Wu — Sun, 13 Sep 2026 19:56:50 +0000

Public developer threads this week keep asking whether models already write production code better than working developers. The more local question for a messy totals module is narrower, and it is testable. Agents rewrite nested loops because the result looks cleaner in a side-by-side diff. Downstream jobs then break on sort order, cent rounding, and silent defaults that never appeared in the pull-request story.

A payments operations team inherited a 900-line report builder that folded mixed CSV rows into weekly merchant totals. Rounding rules, locale-aware sorting, and missing-field defaults lived inside one nested loop without tests. An agent patch removed duplication and renamed helpers, which looked like real progress during code review. Finance jobs then failed because group order shifted, and half-up cents quietly became banker's rounding.

That failure pattern now shows up regularly during agent-assisted refactors of legacy Python report services. The model optimizes for local cleanliness rather than for downstream consumers of order and money. Characterization goldens that freeze those observables let a team extract one helper without accepting a taste-driven rewrite.

What the first tests should actually pin

The first tests are not a proof that the loop is correct under a finance spec. They are a replayable record of the answers the loop already emits for a frozen corpus. If those answers move, the extract is not small, even when the new names look nicer.

Pin these observables before anyone touches helpers:

Group order, including stable ties across period labels and merchant identifiers
Money rounding, especially half-up versus banker's rounding at the cent
Missing fields, such as a default tax rate, a skipped row, or a zero-filled amount
String identity, including currency codes, trimmed names, and period labels

A one-helper extract is reviewable when those four stay byte-identical. Anything that also “fixes” sort keys is a second change and needs its own product note.

Worked example: a tangled weekly totals loop

The module below is a compact stand-in, labeled as an unexecuted example rather than a harvested production file. It still mixes parsing, tax math, rounding context, and a locale switch that only changes sort direction. That mix is enough to demonstrate the trap.

# weekly_totals.py — compact stand-in for a tangled report module
from collections import defaultdict
from decimal import Decimal, ROUND_HALF_UP, localcontext


def build_weekly_totals(rows, locale="en_US"):
    buckets = defaultdict(lambda: {"amount": Decimal("0.00"), "tax": Decimal("0.00")})
    for raw in rows:
        if not raw.get("merchant_id"):
            continue
        period = (raw.get("period") or "unknown").strip()
        key = (str(raw["merchant_id"]), period)
        amount = Decimal(str(raw.get("amount") or "0"))
        raw_rate = raw.get("tax_rate")
        rate = Decimal(str(raw_rate if raw_rate not in (None, "") else "0.0875"))
        buckets[key]["amount"] += amount
        buckets[key]["tax"] += amount * rate

    items = []
    for (merchant_id, period), acc in buckets.items():
        with localcontext() as ctx:
            ctx.rounding = ROUND_HALF_UP
            amount = acc["amount"].quantize(Decimal("0.01"))
            tax = acc["tax"].quantize(Decimal("0.01"))
        items.append({
            "merchant_id": merchant_id,
            "period": period,
            "amount": format(amount, "f"),
            "tax": format(tax, "f"),
            "currency": "USD",
        })

    # Locale is ignored except as a hidden sort switch — the kind of trap agents "fix".
    reverse = locale.startswith("de")
    items.sort(key=lambda row: (row["period"], row["merchant_id"]), reverse=reverse)
    return items

A cleanup that sorts by merchant first, or that swaps Decimal for round(float(...), 2), will still read as a refactor. Canonical goldens make that cleanup visible as a data change instead of a naming change.

Step 1: freeze a fixture corpus before the edit

Check the inputs into git before the loop is edited, and keep each case small enough to read in review. The useful cases are not happy-path totals; they are ties, missing rates, rounding edges, and the locale flag.

fixtures/weekly/
  empty.json
  single_row.json
  tie_same_period.json
  missing_tax_rate.json
  half_up_edge.json
  german_locale_flag.json

Example half_up_edge.json:

{
  "locale": "en_US",
  "rows": [
    {"merchant_id": "m-9", "period": "2026-W37", "amount": "1.005", "tax_rate": "0.10"},
    {"merchant_id": "m-9", "period": "2026-W37", "amount": "2.015", "tax_rate": "0.10"}
  ]
}

Store locale inside the fixture document rather than in the process environment. Hidden runner locale is a common way for goldens to rot after a host change.

Step 2: emit canonical golden JSON

Canonical JSON avoids false diffs from key order, spacing, and Unicode escapes. Readable goldens are the review surface; a later hash of those bytes is optional insurance. Emit once from the untouched module, then treat later diffs as either a real behavior change or a bad extract.

# tools/emit_goldens.py
import json
from pathlib import Path

from weekly_totals import build_weekly_totals

ROOT = Path("fixtures/weekly")


def canonical(obj):
    return json.dumps(obj, sort_keys=True, indent=2, ensure_ascii=True) + "\n"


def main():
    for src in sorted(ROOT.glob("*.json")):
        if src.name.endswith(".golden.json"):
            continue
        payload = json.loads(src.read_text())
        result = build_weekly_totals(
            payload["rows"], locale=payload.get("locale", "en_US")
        )
        out = src.with_suffix(".golden.json")
        out.write_text(canonical(result))
        print(f"wrote {out}")


if __name__ == "__main__":
    main()

python tools/emit_goldens.py
git add fixtures/weekly/*.golden.json
git diff --stat

Commit the goldens beside the fixtures so the lock travels with the module. Reviewers should see merchant, period, and cent fields, not a boolean from a loosely named unit test.

Step 3: assert today against yesterday, not against an ideal spec

The characterization test does not decide whether a default tax rate of 0.0875 is fair. It only decides whether the current loop still matches the committed answers. Empty corpora should fail loudly, because a missing fixture directory is not a green suite.

# tests/test_weekly_totals_characterization.py
import json
from pathlib import Path

from weekly_totals import build_weekly_totals

ROOT = Path("fixtures/weekly")


def canonical(obj):
    return json.dumps(obj, sort_keys=True, indent=2, ensure_ascii=True) + "\n"


def test_weekly_totals_match_goldens():
    sources = [
        path for path in sorted(ROOT.glob("*.json"))
        if not path.name.endswith(".golden.json")
    ]
    assert sources, "fixture corpus is empty"
    for src in sources:
        payload = json.loads(src.read_text())
        result = build_weekly_totals(
            payload["rows"], locale=payload.get("locale", "en_US")
        )
        golden = src.with_suffix(".golden.json").read_text()
        assert canonical(result) == golden, src.name

python -m pytest tests/test_weekly_totals_characterization.py -q

Run that command on the untouched module until it is green, and only then open an editor. If it later fails, print a unified diff of the two JSON strings so the changed cent is obvious.

Step 4: extract one pure helper, then stop

Do not split the loop, rename public keys, or introduce a money library in the same patch. Extract one helper that the goldens already constrain, and leave sort keys and skip rules untouched. The smallest honest change in this stand-in is the repeated quantize block.

def quantize_cents(value):
    with localcontext() as ctx:
        ctx.rounding = ROUND_HALF_UP
        return value.quantize(Decimal("0.01"))

Replace the two inlined quantize calls with quantize_cents, and keep the Decimal values in their original accumulation order. Re-run the characterization test before reading the rest of the file for more ideas. A green result means order and money held; a red result means the helper quantized too early or lost rounding context.

Review checklist after the extract

The public function signature is unchanged, including the locale argument.
The golden JSON bytes are unchanged across every fixture file.
The diff is the helper plus the two call sites, and nothing else.
No locale library swap, float conversion, or default-rate move landed in the same patch.

If an agent also reordered items.sort, reject that hunk even when informal “business logic” checks still pass. Sort order is an observable for any consumer that writes CSV and then diffs weeks.

Use an agent only after the goldens exist

Coding agents are useful once the goldens exist, because the test file is the contract the model cannot narrate away. Without that contract, the agent will treat sort stability and rounding mode as style. The safe sequence is emit, test, allow a narrow edit, test again, then read the diff as data.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option, which is enough to run this emit-and-test loop on a throwaway machine. That availability is an operator-supplied product fact, not a benchmark of extract quality. A green agent session still needs a human reading the sort key and the ROUND_HALF_UP context.

python tools/emit_goldens.py
python -m pytest tests/test_weekly_totals_characterization.py -q
# permit edits only in weekly_totals.py
python -m pytest tests/test_weekly_totals_characterization.py -q
git diff -- weekly_totals.py

Keep fixture files out of the writable set. If the agent rewrites goldens so the new loop passes, the lock has been replaced with the change under test, and the review is no longer a characterization review.

Decision table for the incoming diff

Signal in the diff	Keep?	Why
New `quantize_cents` helper that still uses `ROUND_HALF_UP`	Yes	Pure extract; goldens should hold
Sort key changed from `(period, merchant_id)`	No	Order is an observable for finance exports
`Decimal` replaced with `round(float(...), 2)`	No	Half-up edges drift on values like `1.005`
Default tax rate moved into a config file	Not in this patch	That is a second behavior change
Fixture or golden files rewritten to match new output	No	The corpus is the lock, not the implementation
`ensure_ascii` dropped during JSON emission	No	Golden bytes must stay comparable across hosts

Limitations

This method records current behavior, including bugs that finance already absorbed. If the loop double-taxes a row, the golden will protect that bug until a product owner adds a new fixture and an explicit decision. Characterization is a stability tool, not a correctness oracle.

The function must be deterministic given the fixture document. Wall-clock timestamps, process locale, network calls, and unordered iteration on very old Python runtimes will make goldens flicker. Freeze time and isolate I/O before treating a red test as a real extract failure.

The corpus will not catch a change that appears only for a merchant identifier nobody recorded. Grow fixtures from production samples when that is legal, and scrub account numbers before commit. A golden that contains live identifiers is not a test asset; it is an exposure.

Who should skip this approach

Teams that cannot build a replayable input corpus from logs, dumps, or synthetic rows
Modules whose public contract is a GUI session rather than emitted data
Patches that must change rounding or sort order on purpose, which need new goldens and a written product note
Anyone treating a green characterization suite as proof that the tax math matches a statute

Pin the answers the downstream job already depends on, then extract one helper. Stop there so the next diff remains small enough to review against JSON, not against the agent's explanation of cleanliness.

A Hashed Output Corpus Makes a Transformer Refactor Auditable

Dakota Wu — Sat, 12 Sep 2026 17:55:05 +0000

A checkout team inherited a four-hundred-line transform_row helper that still mutated its input dictionary in place. The helper mixed tax rounding, currency labels, coupon stacking, and CSV escaping without a single assertion. An agent then proposed a cleanup that renamed locals, extracted helpers, and quietly changed discounted-tax rounding. Reviewers almost merged the tidy diff because the file looked smaller, not because behavior had been pinned.

That near-miss is the usual shape of a messy-repo refactor in 2026. Coding agents now emit fluent extracts on demand, and public threads argue they already outwrite most humans. Fluency still does not tell a reviewer whether the transformer emits the same bytes. This article treats the agent as an optional proposer and treats a hashed output corpus as the actual gate.

Why a tidy extract is not evidence

A transformer is a bad target for unconstrained rewrite because its contract lives in combined output, not in helper names. Rounding mode, blank-versus-missing keys, stable key order, and timezone defaults rarely show up in a linter. An extract that “only moves code” can still change 1.005 versus 1.00, or swap "" for a dropped field.

Human characterization tests help, but they usually encode the cases a person remembered. Batch files contain the cases the team actually ships. A hashed corpus records what the current mess already produces for a frozen sample set. The corpus is not a claim of correctness. It is a freeze of present behavior, including the ugly parts you are not ready to change.

Use this loop when all of the following are true:

The module is a pure-ish function from fixtures to bytes, JSON, or CSV.
You can collect a few dozen real-looking inputs without production secrets.
You want one structural change, not a redesign of rates or tax law.
Reviewers will reject the change unless old outputs replay bit-for-bit.

Skip the loop when the transformer talks to live clocks, networks, or unseeded randomness that you cannot seam. Skip it when the business already wants different numbers, because a golden master will fight a deliberate behavior change.

A freeze-then-extract loop

The working rule is narrow. Mine fixtures first, pin output hashes, allow one extract, then replay the same corpus. Anything larger is a second change and needs a second pin.

1. Inventory samples, not opinions

Collect files the batch already understands: truncated production-like rows, empty carts, stacked coupons, and one hostile Unicode name. Store them as fixtures, not as comments in a chat transcript. Label every file with a stable identifier so later diffs name the failing sample instead of a line number inside a blob.

fixtures/transform/
  001_empty_cart.json
  002_coupon_stack.json
  003_half_up_tax.json
  004_unicode_name.json
  005_missing_currency.json

Do not redact by rewriting numbers by hand if you can strip secrets with a dedicated scrubber. Hand-edited fixtures silently become a second, undocumented transformer.

2. Render through today’s mess and hash the bytes

Call the current function once per fixture and hash the exact bytes it already emits. Prefer canonical JSON or explicit CSV settings so key order and newline policy are part of the pin. The script below is an illustrative harness, not a measured production run.

# illustrative harness — unexecuted in this article
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from transform_row import transform_row  # the messy module under freeze

FIXTURES = Path("fixtures/transform")
LOCK = Path("corpus.lock.json")

def render(path: Path) -> bytes:
    payload = json.loads(path.read_text(encoding="utf-8"))
    result = transform_row(payload["row"], payload["rates"], payload["flags"])
    text = json.dumps(result, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return (text + "\n").encode("utf-8")

def digest(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def freeze() -> None:
    entries = []
    for path in sorted(FIXTURES.glob("*.json")):
        blob = render(path)
        entries.append({"id": path.name, "sha256": digest(blob), "nbytes": len(blob)})
    LOCK.write_text(json.dumps({"entries": entries}, indent=2) + "\n", encoding="utf-8")

def replay() -> int:
    lock = json.loads(LOCK.read_text(encoding="utf-8"))
    failures = []
    for entry in lock["entries"]:
        path = FIXTURES / entry["id"]
        actual = digest(render(path))
        if actual != entry["sha256"]:
            failures.append(entry["id"])
            print(f"MISMATCH {entry['id']}: {actual} != {entry['sha256']}")
    print(f"replayed={len(lock['entries'])} mismatches={len(failures)}")
    return 1 if failures else 0

if __name__ == "__main__":
    import sys
    sys.exit(freeze() or 0) if sys.argv[1:] == ["freeze"] else sys.exit(replay())

Run the freeze only against the unpatched tree. After that, treat corpus.lock.json as reviewable source, the same way you treat a lockfile for dependencies.

python harness.py freeze
git add fixtures/transform corpus.lock.json harness.py
git commit -m "Pin transformer golden-master corpus before extract"

3. Name the smallest safe change

Write the intended extract in one sentence that names a seam and forbids extra work. Example: extract _format_money(amount, currency) and keep rounding inside transform_row. If the sentence needs an “and”, you are proposing two changes. Two changes need two corpus pins or an explicit behavior waiver.

A useful seam is a function that turns already-computed values into bytes or labels. A dangerous seam is a function that reorders discounts, talks to today’s date, or reads environment variables. Keep I/O and policy on the original side of the cut until the corpus stays green.

4. Replay before you argue about style

After the extract, replay the lockfile on the same fixtures. A green replay means the public bytes did not move. A red replay means the extract was not an extract. Do not “fix” a mismatch by rewriting the lockfile unless the mismatch is the product change you meant to ship, and that change has its own review.

python harness.py replay
# expected: replayed=5 mismatches=0

If a single fixture fails, print both JSON documents with sorted keys and diff them as text. Hash mismatches without a document diff train people to shrug and re-freeze. The corpus only works when a mismatch is cheaper to inspect than to ignore.

Decision table for the extract itself

Use the table as a review checklist, not as a scoring rubric. Any “no” in the reject column stops the patch, even when the agent’s names look nicer than yours.

Question	Accept the extract	Reject and restore
Does `corpus.lock.json` still match every fixture?	Yes, bit-for-bit	Any hash moved
Did public function signatures stay in place?	Yes	Callers now need adapters
Did rounding, sorting, or defaulting stay in the old function?	Yes	Policy moved with the helper
Is there a single new function or module?	One seam	Drive-by renames elsewhere
Are tests, fixtures, and lockfile in the same patch?	Yes	“Cleanup” without replay
Did clocks, locales, or network calls gain a new read?	No new reads	Hidden non-determinism

The table is deliberately boring. Boring gates are the point, because agent diffs optimize for local readability, not for byte stability.

An illustrative extract that should survive replay

The original function below is labeled illustrative. It is a compressed stand-in for the usual mess: mutation, string building, and rounding in one place.

# illustrative messy baseline — not production code
from decimal import Decimal, ROUND_HALF_UP

def transform_row(row, rates, flags):
    row = dict(row)  # later agents often drop this copy
    subtotal = Decimal(str(row.get("subtotal") or 0))
    rate = Decimal(str(rates.get(row.get("region"), "0")))
    if flags.get("coupon"):
        subtotal -= Decimal(str(flags["coupon"]))
    tax = (subtotal * rate).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
    total = subtotal + tax
    currency = row.get("currency") or "USD"
    row["tax_label"] = f"{currency} {tax}"
    row["total_label"] = f"{currency} {total}"
    row["tax"] = str(tax)
    row["total"] = str(total)
    return row

The smallest safe change moves only label formatting. Rounding stays put, because rounding is policy, not presentation.

# illustrative extract — still unexecuted here
def _format_money(currency, amount):
    return f"{currency} {amount}"

def transform_row(row, rates, flags):
    row = dict(row)
    subtotal = Decimal(str(row.get("subtotal") or 0))
    rate = Decimal(str(rates.get(row.get("region"), "0")))
    if flags.get("coupon"):
        subtotal -= Decimal(str(flags["coupon"]))
    tax = (subtotal * rate).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
    total = subtotal + tax
    currency = row.get("currency") or "USD"
    row["tax_label"] = _format_money(currency, tax)
    row["total_label"] = _format_money(currency, total)
    row["tax"] = str(tax)
    row["total"] = str(total)
    return row

If replay fails after this cut, the extract is lying about being presentational. Common liars are str(Decimal) format changes, implicit float, and dropping the defensive dict(row) copy. Restore the tree, tighten the sentence that named the seam, and try a smaller cut.

Where a constrained agent runtime belongs

The corpus and the replay belong on your machine or in CI, next to the fixtures. The agent, if you use one at all, belongs after corpus.lock.json exists, and it should receive the one-sentence seam as a constraint, not a vague “please clean this up”. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode’s free model access and free server option are relevant only for that second half of the loop: proposing the extract inside a disposable runtime while the golden master stays under your version control. They do not replace the lockfile, and they do not prove the extract is correct. If you try that split, keep secrets out of the fixtures and treat the remote session as untrusted compute, the same way you would treat any other shared box.

A practical prompt to paste after the freeze looks like the block below. It is a constraint list, not a request for a rewrite of tax policy.

Propose one extract in transform_row.py.
Do not change rounding, defaults, key names, or return shapes.
Do not edit fixtures/ or corpus.lock.json.
After the edit, I will run: python harness.py replay
If you cannot keep the replay green, return no diff.

Reject patches that expand the prompt into extra files, extra helpers, or “while we are here” import sorting across unrelated modules. Those extras are how byte-stable extracts become unreviewable novellas.

Limitations, cost, and who should not bother

Golden masters freeze bugs as faithfully as they freeze intended behavior. If 003_half_up_tax.json already encodes a rounding error, the corpus will punish the fix until you replace that fixture in a dedicated behavior-change commit. That is a feature when you wanted a structural extract. It is a hazard when the team thought it was fixing math.

The method also fails closed on non-determinism. Unsorted dicts on older runtimes, local timezone stamps, and iterating set objects will thrash hashes. If you cannot seam those reads, you do not have a transformer you can golden-master yet. Build a clock or locale seam first, then freeze, then extract.

Do not use this approach for:

Cryptographic or financial code that needs proof, not historical bytes.
Modules whose output is streaming UI, not a finite document.
Refactors whose whole point is changing rates, copy, or legal rounding.
Samples you cannot store without customer secrets or credentials.

There is no benchmark in this article because the claim is not speed. The claim is auditability: a reviewer can say what stayed the same. If you cannot name the fixtures, you are still negotiating with an agent’s confidence instead of with a lockfile.

The next time a fluent diff offers to rescue transform_row, pin the bytes it already prints. Extract one presentation seam. Replay the corpus. Only then discuss whether the names are nicer.

Freeze the Export Manifest Before You Touch a Tangled Batch Job

Dakota Wu — Fri, 11 Sep 2026 15:37:47 +0000

A weekly finance export that lived in one file for years invites a sudden, agent-driven split into modules. The usual failure is not a syntax error but a renamed column or a reordered CSV. Finance already wired those artifacts into spreadsheets, so a tidy diff can still break the week. Characterization for this mess is a frozen output manifest, not a unit test of a helper.

This article treats that exporter as a teaching fixture and walks a merge gate you can run locally. The gate pins clock, locale, filenames, headers, row counts, and content hashes before anyone extracts a formatter. Coding agents can propose the extract later. They should not own the first merge if the output tree can drift.

The Monday job nobody wants to open

Picture a payments repo where scripts/weekly_export.py still owns SQL, rounding, CSV writing, and a JSON summary. The file is about fourteen hundred lines, and the last six commits only appended helpers. Operators run it from cron with WEEKLY_EXPORT_DIR pointed at a shared folder. Downstream tools then glob payments_*.csv and parse header row zero as a contract.

A coding agent, asked to clean the file, often extracts three classes and renames amount to amount_usd. The Python tests still pass because they mocked the database and never opened the directory. Finance notices on Monday, when a VLOOKUP misses a column that used to sit in position four. The missing piece was not coverage of helpers. It was a byte-level picture of the tree the job emits.

What to freeze besides return values

Unit tests of pure functions are useful after the mess is already sliced. They are a weak first harness when I/O, clocks, and filenames are the product. For a batch exporter, characterize the observable directory, then allow one internal change that cannot alter that directory.

Freeze at least these fields under a pinned clock and locale:

Relative paths of every file the job writes, including empty marker files
SHA-256 of file bytes, so silent reordering or newline changes show up
First line of each CSV, treated as an explicit header contract
Row counts excluding the header, so a filter change cannot hide
JSON keys in sorted order for the summary document, ignoring insignificant whitespace

If any field moves after a refactor, the change is not yet a cleanup. It is a behavior change and needs a product decision, not a style argument.

Teaching fixture: a tangled weekly export

The following module is a simplified stand-in, not production finance code. It mixes clock reads, locale-sensitive money formatting, and two output files on purpose. Label it as a fixture for the harness, not as advice about how to design exporters.

# weekly_export.py — teaching fixture, mixed I/O on purpose
from __future__ import annotations

import csv
import json
from datetime import datetime, timezone
from pathlib import Path


def load_rows() -> list[dict]:
    # Stand-in for a SQL dump. Keep this deterministic in tests.
    return [
        {"account": "A-100", "cents": 1999, "status": "paid"},
        {"account": "B-200", "cents": 500, "status": "paid"},
        {"account": "C-300", "cents": 0, "status": "void"},
    ]


def run_export(out_dir: Path) -> None:
    out_dir.mkdir(parents=True, exist_ok=True)
    stamp = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    rows = [r for r in load_rows() if r["status"] != "void"]
    csv_path = out_dir / f"payments_{stamp}.csv"
    with csv_path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["account", "amount", "status"])
        for row in rows:
            amount = f"${row['cents'] / 100:.2f}"
            writer.writerow([row["account"], amount, row["status"]])
    summary = {
        "date": stamp,
        "count": len(rows),
        "total_cents": sum(r["cents"] for r in rows),
    }
    (out_dir / "summary.json").write_text(
        json.dumps(summary, indent=2) + "\n",
        encoding="utf-8",
    )

The amount string is the landmine. An agent that extracts formatting and switches to locale.currency will change $19.99 into a different grouping or symbol. Tests that only check len(rows) will stay green. The manifest will not.

Harness: write a manifest, then fail on drift

Pin time and locale before the job runs. Then hash the tree. The script below is a proposal you can copy into tools/characterize_export.py and run against a temporary directory.

# tools/characterize_export.py
from __future__ import annotations

import hashlib
import json
import os
import sys
from pathlib import Path

GOLDEN = Path("tests/goldens/weekly_export_manifest.json")


def file_digest(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def csv_header_and_rows(path: Path) -> tuple[str, int]:
    lines = path.read_text(encoding="utf-8").splitlines()
    header = lines[0] if lines else ""
    return header, max(len(lines) - 1, 0)


def build_manifest(out_dir: Path) -> dict:
    entries = []
    for path in sorted(p for p in out_dir.rglob("*") if p.is_file()):
        rel = path.relative_to(out_dir).as_posix()
        header, rows = (csv_header_and_rows(path) if path.suffix == ".csv" else ("", 0))
        entries.append(
            {
                "path": rel,
                "sha256": file_digest(path),
                "bytes": path.stat().st_size,
                "header": header,
                "data_rows": rows,
            }
        )
    return {"files": entries}


def main(argv: list[str]) -> int:
    mode = argv[1] if len(argv) > 1 else "check"
    out_dir = Path(os.environ["WEEKLY_EXPORT_DIR"])
    current = build_manifest(out_dir)
    if mode == "record":
        GOLDEN.parent.mkdir(parents=True, exist_ok=True)
        GOLDEN.write_text(json.dumps(current, indent=2) + "\n", encoding="utf-8")
        print(f"recorded {GOLDEN}")
        return 0
    expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
    if current != expected:
        sys.stderr.write("export manifest drifted\n")
        sys.stderr.write(json.dumps({"expected": expected, "current": current}, indent=2))
        return 1
    print("export manifest matched")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Record once against a fixture clock, then check on every refactor attempt. A small wrapper keeps the job from reading the real wall clock during characterization.

export WEEKLY_EXPORT_DIR="$(mktemp -d)"
export TZ=UTC
export LC_ALL=C.UTF-8
export PYTHONPATH=.
python - <<'PY'
from datetime import datetime, timezone
from pathlib import Path
import weekly_export
from unittest.mock import patch

frozen = datetime(2026, 9, 7, tzinfo=timezone.utc)
with patch("weekly_export.datetime") as dt:
    dt.now.return_value = frozen
    dt.side_effect = lambda *args, **kwargs: datetime(*args, **kwargs)
    weekly_export.run_export(Path(__import__("os").environ["WEEKLY_EXPORT_DIR"]))
PY
python tools/characterize_export.py record   # first capture only
python tools/characterize_export.py check

The first record is a human action. After that, agents may run check as often as they want. They should not rewrite the golden file to make a refactor pass.

Decision table: snapshot, example, or mutation

Use the table when someone asks why a directory hash is the first test, not a pytest of format_amount.

Signal you care about	First characterization	Weak substitute	When to switch
Column names and order	CSV header line in the manifest	Asserting `dict.keys()` on an in-memory row	After the writer is a pure function
Money text and locale	SHA-256 of the CSV bytes	Rounding unit tests with floats	After formatting is extracted and frozen
Which accounts are dropped	`data_rows` plus hash	A single filtered fixture list	When filters become an explicit policy module
Summary JSON shape	Canonical object compare after load	Snapshot of `indent=2` text only	If whitespace is declared insignificant
Filename stamp	Path list under a patched clock	Regex on `payments_*.csv`	Never, if finance globs the stamp

Mutation testing of helpers is valuable later. It is the wrong first spend when the observable product is a folder of files. Example-based tests of format_amount become the second layer, once that function exists and the manifest still matches.

Smallest safe change after the tree is pinned

Do not extract a WeeklyExporter class on the first pass. Extract one pure formatter whose output is already visible in the golden CSV. For this fixture, that function is the amount string, and nothing else.

def format_amount(cents: int) -> str:
    return f"${cents / 100:.2f}"

Then replace the inline f"${row['cents'] / 100:.2f}" with format_amount(row["cents"]). Re-run the frozen clock, rebuild the directory, and execute python tools/characterize_export.py check. If the manifest matches, the extract is a cleanup. If it fails, stop and treat the formatter as a behavior change.

A useful review rule is numerical, not aesthetic. Count files touched, functions extracted, and manifest fields at risk:

One new pure function, with no filesystem access and no clock read
One call-site replacement inside the existing writer loop
Zero filename, header, or JSON-key edits in the same diff
Manifest check green before the branch is eligible for review

Anything larger is a second change. Agents collapse those steps because the resulting code looks more like a tutorial. The manifest exists to reject that collapse.

Where a free agent run helps, and where it does not

Generating the first format_amount extract is cheap model work. Generating a stable clock wrapper and a manifest checker is also cheap, if a human reviews the golden file. Running those commands on a disposable machine keeps local locale settings from leaking into hashes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that disposable run: the agent drafts the one-function extract, executes the pinned-clock export, and reports the manifest diff without writing into a laptop working tree. The merge gate stays the same JSON file in tests/goldens/. The product does not replace the record step, and this article does not claim model names, quotas, hardware, or lasting availability.

If you already isolate agent jobs this way, keep the golden manifest in git and treat any record rewrite as a product review. The interesting output is the diff of weekly_export_manifest.json, not a score the model assigns itself.

Limitations

Directory hashing is brittle when exporters embed timestamps inside rows, not only in filenames. You must stub those clocks, or the manifest will flap every run and teams will start ignoring it. Binary artifacts such as PDFs and XLSX files also hash-unstable if a library embeds write times; prefer structured exports or a parsed logical snapshot for those formats.

The harness also assumes a hermetic fixture for load_rows(). A characterization run that hits shared staging data will snapshot whoever happened to pay that afternoon. Do not record goldens from production folders, and do not let an agent refresh goldens as a way to green a refactor.

Who should not use this first

Skip the manifest approach when the batch job has no stable consumers of filenames or headers. An internal debug dump that nobody parses can be covered with ordinary unit tests. Skip it when the legitimate change is a schema migration that finance already scheduled; in that case update the golden under review, then extract. Skip it when you cannot pin time, locale, and input rows, because a flapping hash trains people to delete the gate.

Teams that need many coordinated extracts should still start here, then add helper tests. The first merge remains one formatter, one call site, and an unchanged export tree. That sequencing is slower than an overnight rewrite, and it is the reason Monday spreadsheets keep resolving.

Map Every Call Site Before You Split a God Function

Dakota Wu — Wed, 09 Sep 2026 15:51:51 +0000

The following walkthrough uses a synthetic order-processing module, not a production incident attributed to any employer. A coding agent proposed splitting process_order into retail, wholesale, and refund helpers after one overnight prompt. The diff looked tidy until a nightly CSV importer failed because it still passed a loosely typed dictionary. Mapping callers first would have shown that importer, an admin retry job, and a forgotten fax channel.

God functions hide disagreement among callers

A four-hundred-line callee is rarely one abstraction wearing an unfortunate name. Its callers usually disagree about required keys, default channels, and whether missing prices should raise or skip. An agent that reads only the function body will invent a narrower signature that matches the comments. The inventory below does not claim the module is well designed. It only records who still depends on the current shape.

You should treat a proposed split as a behavior change until every observed call site is listed. Characterization tests then pin those shapes so the first extract cannot quietly drop a branch. The extract itself should move one predicate, not three business lines, in a single commit.

Typical damage from an unmapped split looks like the list below.

A required channel argument appears, and the fax command still calls with one positional dict.
None for status becomes the string "open", which changes retry jobs that branch on identity.
Refunds move to a new module, while the importer still inspects the old tuple layout.
Dynamic getattr(orders, name) calls vanish from the agent's file-level rewrite.

A six-step workflow you can rerun

Freeze the public function name and refuse renames until the caller map is green.
Collect static call sites with an AST walk and a fallback ripgrep pass.
Record argument shapes from fixtures, logs, or a short tracing wrapper.
Turn each distinct shape into a characterization test that asserts today's result.
Extract one branch whose predicate is already covered by those tests.
Regenerate the caller map and fail the build if a site appears or disappears.

The loop is deliberately boring. Agents optimize for a clean file tree, while this loop optimizes for unchanged callers. Re-run the map after every agent session, not only after the extract you intended to accept.

Artifact: regenerate a caller map from AST

Save the following example script as tools/caller_map.py. It is labeled example code for a local inventory, not a measured production scanner.

"""Labeled example: inventory call sites for a function name."""
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

TARGET = "process_order"
SKIP_PARTS = {".venv", "node_modules", "__pycache__", "migrations"}


class CallCollector(ast.NodeVisitor):
    def __init__(self, path: str) -> None:
        self.path = path
        self.hits: list[dict] = []

    def visit_Call(self, node: ast.Call) -> None:
        name = None
        if isinstance(node.func, ast.Name) and node.func.id == TARGET:
            name = node.func.id
        elif isinstance(node.func, ast.Attribute) and node.func.attr == TARGET:
            name = node.func.attr
        if name:
            self.hits.append(
                {
                    "file": self.path,
                    "line": node.lineno,
                    "kwargs": [k.arg for k in node.keywords if k.arg],
                    "positional": len(node.args),
                    "starargs": any(isinstance(a, ast.Starred) for a in node.args),
                }
            )
        self.generic_visit(node)


def iter_python_files(root: Path):
    for path in root.rglob("*.py"):
        if any(part in SKIP_PARTS for part in path.parts):
            continue
        yield path


def main() -> int:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    rows: list[dict] = []
    for path in iter_python_files(root):
        try:
            tree = ast.parse(path.read_text(encoding="utf-8"))
        except SyntaxError as exc:
            rows.append({"file": str(path), "error": str(exc)})
            continue
        visitor = CallCollector(str(path))
        visitor.visit(tree)
        rows.extend(visitor.hits)
    Path("caller_map.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
    print(f"wrote {len(rows)} rows to caller_map.json")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it beside a second, dumber net so dynamic calls are not silently dropped.

python tools/caller_map.py ./app
rg -n "process_order\s*\(" --glob "*.py" --glob "!tools/caller_map.py"
python -c "import json; print(len(json.load(open('caller_map.json'))))"
diff -u <(jq -S . caller_map.json) <(jq -S . caller_map.committed.json)

The AST pass records keyword names and positional arity for each hit. The ripgrep pass catches comments, strings, and getattr patterns the tree walker will miss. Store both outputs in version control next to the characterization tests. A later agent diff cannot delete a caller without a failing check if that pair is committed.

What to keep in `caller_map.json`

Keep the committed map small enough to read in review, and reject extra fields that churn for no reason.

Keep file, line, kwargs, positional, and starargs.
Drop absolute timestamps, usernames, and generated tool versions.
Sort rows by file then line before you commit the file.
Treat a missing fax_retry.py row as a failed build, not a cleanup win.

Record argument shapes, not the agent's preferred types

Static hits are not enough, because two call sites can share a name and disagree about payload shape. Add a tracing wrapper in example tests or a staging importer, then dump distinct shapes to shapes.json.

# Labeled example: record observed kwargs without claiming production coverage.
from collections import Counter
import json
from functools import wraps

_shapes = Counter()


def trace_process_order(fn):
    @wraps(fn)
    def wrapper(*args, **kwargs):
        key = json.dumps(
            {
                "arity": len(args),
                "keys": sorted(kwargs),
                "channel": kwargs.get("channel"),
                "has_price": "price" in kwargs,
                "status": None
                if kwargs.get("status") is None
                else type(kwargs.get("status")).__name__,
            },
            sort_keys=True,
        )
        _shapes[key] += 1
        return fn(*args, **kwargs)

    return wrapper

After a representative fixture run, persist the counter. A compact table for this synthetic module might look like the following, which you should replace with shapes you actually observed.

source	arity	keys	channel	status type	count
csv_importer	1	order, channel	retail	str	412
admin_retry	1	order, channel, price	wholesale	NoneType	18
fax_command	1	order	fax	missing	2
refund_job	1	order, channel, refund	retail	str	77

The fax row is the one an agent will drop while renaming parameters to a closed set of channels. The inventory exists to make that row expensive to ignore. Counts in the table are synthetic labels for the example, not measurements from a live system.

Characterization tests built from the map

Each distinct shape becomes a test that calls today's process_order and asserts a frozen tuple of (status, total_cents, events). Do not assert internal helper names. Do not assert log wording unless a caller already parses those logs.

# Labeled example tests. Replace fixtures with recordings from your repo.
import json
from pathlib import Path

import pytest

from app.orders import process_order

SHAPES = json.loads(Path("shapes.json").read_text(encoding="utf-8"))


def _freeze(result):
    status, total_cents, events = result
    return status, total_cents, tuple(events)


@pytest.mark.parametrize("shape_id", sorted(SHAPES))
def test_observed_shape_still_returns_today(shape_id):
    payload = SHAPES[shape_id]["payload"]
    expected = tuple(SHAPES[shape_id]["frozen_result"])
    assert _freeze(process_order(**payload)) == expected


def test_fax_channel_is_still_a_caller():
    result = process_order(order={"id": "fax-1", "lines": []}, channel="fax")
    assert result[0] in {"skipped", "queued"}


def test_caller_map_has_not_shrunk():
    recorded = json.loads(Path("caller_map.json").read_text(encoding="utf-8"))
    files = {row["file"] for row in recorded if "line" in row}
    assert "app/importers/csv_orders.py" in files
    assert "app/management/fax_retry.py" in files

If you lack a snapshot helper, write the frozen tuple next to the payload in shapes.json on the first green run, then commit the file. Later runs must not rewrite that file during an agent session. A one-line CI check that shapes.json is unchanged is cheaper than arguing about intent in review.

git diff --exit-code -- shapes.json caller_map.json
pytest -q tests/test_process_order_shapes.py

Extract one branch after the map is green

Only after the tests pass should you move a single predicate. In this synthetic module the refund path is the safest first extract, because the caller map shows one job and the shape table shows one extra key.

# Before: one function owns every channel and the refund mutation.
def process_order(order, channel="retail", price=None, refund=None, status="open"):
    if refund:
        return _refund_inline(order, refund, status)
    if channel == "fax":
        return "queued", 0, ("fax-deferred",)
    # hundreds of retail and wholesale lines follow
    raise NotImplementedError("example remainder")


# After: same signature, refund body lives next door.
def process_order(order, channel="retail", price=None, refund=None, status="open"):
    if refund:
        return process_refund(order, refund=refund, status=status)
    if channel == "fax":
        return "queued", 0, ("fax-deferred",)
    raise NotImplementedError("example remainder")

Keep the original function name, default values, and return tuple. The agent can rename internals only after this commit stays green for a full importer run. If the agent also wants to delete channel="fax", that is a product change and needs its own test and rollout, not a cleanup label.

Review checklist for the extract commit

The public signature is byte-for-byte unchanged, including defaults.
caller_map.json still lists every previously committed file.
shapes.json frozen tuples are unchanged for retail, wholesale, fax, and refund.
No rename of process_order landed in the same commit as the extract.
Ripgrep still finds the fax command, even if AST missed a dynamic helper.

Decision table for the next commit

Observation	Allowed next commit	Reject the agent diff
New helper, same signature, same frozen tuples	Extract one covered branch	Extra signature changes in the same PR
Caller map row count unchanged	Proceed	Silent deletion of `fax_command`
Shape keys unchanged	Proceed	`price` became required
AST miss but ripgrep hit on `getattr`	Add a test, then extract	Assume dynamic calls are dead
Tests pass, importer golden file drifts	Stop and inspect events	Explain it as formatting

Use the table in review comments instead of debating whether the agent meant well. The map either still describes production callers, or the cleanup is not a refactor.

Running the inventory off your laptop

The script and the tests above are ordinary Python, and they do not require any vendor. When the messy tree is too large for a local checkout, a disposable machine can still regenerate the map. A model may propose the one-branch extract against that copy after the characterization suite is green.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can run this inventory. Those options can host the caller-map script on a throwaway copy of the repository tree. Do not place production secrets, customer dumps, or live credentials on a shared free host. Treat model output as a draft extract that still has to pass the committed map.

If you already have CI, run caller_map.py in that pipeline and skip any hosted environment. The workflow remains useful if that hosted option is removed from the process entirely.

Limitations you should budget for

AST inventory misses getattr(module, name), exec, and calls assembled from strings.
Ripgrep over-counts comments, tests, and documentation that mention the function.
JSON shape keys collapse distinct values that happen to share a type name.
Tracing wrappers change timing and can hide race-only branches in messy modules.
Extracting one branch can still alter exception order if helpers start catching earlier.
Frozen tuples will not catch extra outbound HTTP unless you also stub those clients.
A free shared server is the wrong place for proprietary order data or signing keys.

None of these limitations are reasons to skip the map. They are reasons to keep the first extract small and to read the ripgrep misses by hand. If a call site cannot be expressed as JSON, write a dedicated test rather than widening the tracer until it lies.

Who should not use this approach

Skip this workflow if the module already has an honest public contract and a complete caller list in your type checker. Skip it if your goal is a ground-up rewrite rather than a behavior-preserving cleanup. Skip it if the only available compute is a shared free host and the repository contains secrets, because the inventory is not worth a credential leak. Skip it if you cannot freeze shapes.json against mutation during an agent session.

Teams that should use it are the ones receiving confident cleanup pull requests against functions that still serve importers, cron jobs, and one forgotten channel. The caller map will not make the god function elegant. It will make the first extract boring enough to merge.

Hash the Side-Effect Ledger Before You Accept a Cleanup Refactor

Dakota Wu — Tue, 08 Sep 2026 03:31:41 +0000

Messy modules rarely break because a pure helper returns the wrong integer on a tidy fixture. They break because three functions share a temporary CSV path, an environment flag, and a cache nobody named. A coding agent then proposes a cleanup that deletes dead branches, renames locals, and still satisfies every existing assertion. The next production export fails because the implicit file layout moved while the return payload stayed identical.

That failure mode is the reason this workflow exists, and it is not a style problem. The first commit should freeze a ledger of hidden couplings and store a hash beside it. Only after that hash is in source control should you allow one structural change. The cleanup is legitimate only when the recorded hash remains identical.

Cleanup diffs fail differently than feature diffs

Feature work usually changes an observable on purpose, so reviewers know which assertions must move. Cleanup work is sold as behavior-preserving, which trains people to trust deletions and rename-only hunks. Coding agents amplify that bias because they optimize for shorter files, conventional names, and green unit tests. Reviewers then accept large deletions that would look suspicious inside a feature pull request.

Return-value tests are the wrong gate for that class of change. The public function can still return {"ok": true, "rows": 12} while the working directory quietly shifts. Downstream jobs that glob files or catch a named exception will fail after merge. Those hidden couplings remain part of the contract even when no unit test mentions them.

Build a side-effect ledger instead of another unit test

Treat the messy module as a black box that emits more than a return value. A ledger is a canonical JSONL file with one record per fixture and fully sorted keys. Side-effect entries need stable ordering so the serialized bytes stay deterministic across reruns. The SHA-256 digest of that file is the only number that must remain constant.

Each record should capture the following fields and omit anything that varies by machine:

public callable name and a short digest of the arguments
result kind: return, raise, or timeout
exception class name when the result kind is raise
environment keys the module actually read during the call
files created, removed, or appended, relative to a sandbox root
working directory relative to that same sandbox root

Canonicalize every path against the sandbox root before you serialize the record. Wall-clock timestamps and absolute home-directory prefixes make the hash flaky on contact. A flaky gate teaches the team to skip the protocol, which is worse than having no gate.

Keep the first change inside a six-step loop

Choose one public entrypoint rather than the entire package, and list fixtures that replay without network access.
Wrap the process in a sandbox with a temporary root, a copied fixture tree, and an environment allowlist.
Run the recorder and commit ledger.jsonl plus ledger.sha256 with no production code changes.
Declare a file-touch budget: the cleanup commit may edit one implementation file and one test file.
Permit a single structural change, such as extracting a path helper or passing the cache as an argument.
Re-run the recorder and accept the diff only when the hash matches and the touch budget holds.

If the hash changes, the cleanup is not a cleanup and should not keep that label. Treat the diff as a behavior change, add an intentional test, and restart the protocol. Agents that continue after a mismatch are doing product work under a refactor heading.

Example recorder, labeled as a proposal

The script below is an unexecuted example you can adapt to one entrypoint. It does not claim production coverage numbers, and it will miss native writes outside the sandbox. Read it as a starting template rather than as a library you vendor unchanged.

# ledger_recorder.py — proposal: pin hidden couplings for one entrypoint
from __future__ import annotations

import hashlib
import json
import os
import sys
import traceback
from pathlib import Path
from typing import Any, Callable

SANDBOX = Path(os.environ["LEDGER_SANDBOX"]).resolve()
LEDGER_PATH = Path(os.environ.get("LEDGER_PATH", "ledger.jsonl"))


def canonicalize(path: Path) -> str:
    try:
        return str(path.resolve().relative_to(SANDBOX))
    except ValueError:
        return f"<outside>/{path.name}"


def digest_args(args: tuple[Any, ...]) -> str:
    blob = json.dumps(args, default=str, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:12]


def record_call(name: str, args: tuple[Any, ...], fn: Callable[..., Any]) -> dict[str, Any]:
    env_reads: set[str] = set()
    before = {
        canonicalize(p): p.stat().st_mtime_ns
        for p in SANDBOX.rglob("*")
        if p.is_file()
    }
    real_getenv = os.getenv

    def wrapped_getenv(key: str, default: Any = None) -> Any:
        env_reads.add(key)
        return real_getenv(key, default)

    os.getenv = wrapped_getenv  # type: ignore[assignment]
    try:
        value = fn(*args)
        result = {"kind": "return", "value": value, "error": None}
    except Exception as exc:
        result = {
            "kind": "raise",
            "value": None,
            "error": type(exc).__name__,
            "trace_tail": traceback.format_exc().splitlines()[-1],
        }
    finally:
        os.getenv = real_getenv  # type: ignore[assignment]

    after = {
        canonicalize(p): p.stat().st_mtime_ns
        for p in SANDBOX.rglob("*")
        if p.is_file()
    }
    files: set[tuple[str, str]] = set()
    for rel in sorted(set(before) | set(after)):
        if rel not in before:
            files.add(("create", rel))
        elif rel not in after:
            files.add(("remove", rel))
        elif before[rel] != after[rel]:
            files.add(("append", rel))

    return {
        "callable": name,
        "args_digest": digest_args(args),
        "kind": result["kind"],
        "error": result["error"],
        "value": result["value"],
        "env_reads": sorted(env_reads),
        "files": sorted(files),
        "cwd": canonicalize(Path.cwd()),
        "sys_path_heads": [
            canonicalize(Path(p)) if Path(p).exists() else p for p in sys.path[:3]
        ],
    }


def write_ledger(rows: list[dict[str, Any]]) -> str:
    canonical = [json.dumps(row, sort_keys=True, default=str) for row in rows]
    canonical.sort()
    LEDGER_PATH.write_text("\n".join(canonical) + "\n", encoding="utf-8")
    digest = hashlib.sha256(LEDGER_PATH.read_bytes()).hexdigest()
    Path("ledger.sha256").write_text(digest + "\n", encoding="utf-8")
    return digest


def main() -> None:
    # Proposal: replace with the real entrypoint and replayable fixtures.
    from app.export import run_export  # labeled example import

    os.chdir(SANDBOX)
    fixtures = [
        ("run_export", ("2026-09-08",)),
        ("run_export", ("2026-09-09",)),
    ]
    rows = [record_call(name, args, run_export) for name, args in fixtures]
    print(write_ledger(rows))


if __name__ == "__main__":
    main()

A matching check belongs in continuous integration as a command, not as a comment on the pull request. The commands assume a clean sandbox and a hash file committed on the characterization branch.

export LEDGER_SANDBOX="$(pwd)/.ledger-sandbox"
rm -rf "$LEDGER_SANDBOX"
mkdir -p "$LEDGER_SANDBOX/fixtures"
cp -R tests/fixtures/. "$LEDGER_SANDBOX/fixtures/"
python ledger_recorder.py
test "$(tr -d '[:space:]' < ledger.sha256)" = "$(sha256sum ledger.jsonl | awk '{print $1}')"
git diff --name-only origin/main...HEAD | awk 'END { if (NR > 2) { print "touch budget exceeded"; exit 1 } }'

The final command is the file-touch budget for the cleanup commit itself. Diffs that rewrite six files while claiming no behavior change should fail even when the hash matches. Reviewers cannot audit accidental protocol shifts across that much surface in one sitting.

Negotiate the first change with a decision table

Use the table as the only negotiation surface with the agent session. Anything outside the selected row is a new task, not a continuation of the cleanup. Put the selected row in the prompt and omit future style goals that would invite extra edits.

Observed coupling	Safe first change	Hash must stay	Touch budget
Shared temp CSV path	Extract `export_path(sandbox, date)` and call it from one site	Yes	1 impl + 1 test
Process-global cache dict	Pass the cache into the entrypoint; do not rename keys	Yes	1 impl + 1 test
`os.getenv("EXPORT_MODE")`	Read the flag once at the edge; keep the default identical	Yes	1 impl + 1 test
Catches `ValueError` by name	Leave the type; do not switch to a custom hierarchy yet	Yes	0 impl if the type would change
Absolute `/tmp` writes	Relocate writes under the sandbox root only	Yes	1 impl + 1 test

If two rows look tempting, pick the coupling that already causes production incidents and defer the rest. Parallel cleanups destroy the meaning of the hash because a mismatch cannot be attributed to one seam. The agent should see the selected row and the current ledger hash, then stop.

Run the ledger on a pinned runner

Local laptops pollute characterization hashes through leftover files, extra environment variables, and home-directory prefixes. A dedicated runner with a known sandbox root removes that noise from the digest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft the single-seam patch from the decision table, and the free server option can execute the recorder so the hash is not a function of your laptop.

That split is more useful than another chat transcript about naming style. The model proposes one change inside the touch budget, and the server replays the fixtures. You accept the diff only when both gates pass on the same job log. Skip any run that cannot pin the Python version, working directory, and fixture tree.

Limitations

The ledger does not see network I/O you forgot to stub or threads that flush after the recorder returns. Native extensions that write outside Path.rglob will also slip past the file list. Hash stability still depends on canonical JSON and on redacting values that embed timestamps.

This protocol is slower than asking an agent to clean the module in one pass. It will reject useful renames that change an exception type or a filename pattern on purpose. Those edits are product changes and need an explicit test update, not a cleanup label.

Who should skip this protocol

Do not use a side-effect ledger on greenfield code where the public contract still moves every day. Do not use it as a substitute for a typed interface when you already have a stable API module. Do not point an agent at the whole repository and then widen the touch budget until the gate becomes theater.

If the module's only consumers are humans clicking a button, return-value checks may already be enough. The ledger earns its keep when hidden couplings are the real API for downstream jobs. Matching hashes plus a two-file budget is a boring gate, and boring gates are how messy modules survive a first structural change.

Replay the CLI Contract Before You Accept an Agent Diff

Dakota Wu — Sun, 06 Sep 2026 21:07:23 +0000

The billing export script printed invoices to stdout, mixed logs into stderr, and encoded tax rules as nested if statements. Nobody on the current team could still name the last person who understood the rounding path. A coding agent could rewrite the module in minutes, which is exactly why the rewrite was the wrong first move.

This walkthrough uses a constructed brownfield example rather than an unverified production incident from a private repo. The working goal is a strict replay gate that treats process output as the missing specification. After the pin file is locked, one internal edit is allowed only if replay still matches. Internals may move; the process boundary may not.

Cheap patches do not make behavior cheap to trust

Faster generation does not retire implicit contracts with cron, downstream parsers, and operators who grep log lines. An agent that cleans names can still change a trailing newline, a money format, or an exit code that a wrapper treats as success. Those are behavior changes, even when the pull request reads like a refactor.

Technical debt does not shrink just because patches arrive faster than reviews. Debt grows when unverified edits accumulate around undocumented edges that only production still exercises. The practical response is not a larger prompt and a broader cleanup. The practical response is a frozen I/O record that the next patch must satisfy line for line.

What a pin file actually records

A pin is one complete process execution, not a unit assertion about private helpers. Each line in the pin log is a JSON object with a closed field list. Humans should be able to read a pin and explain the scenario in one sentence.

id: stable name for the scenario, such as tax-exempt-q3
argv: argument vector after the interpreter name
env: only the keys the script actually reads
stdin: raw text used for the recorded run
exit_code: integer status observed from the process
stdout and stderr: exact Unicode text with a documented decode mode
cwd_rel: working directory relative to the repository root

The pin file is the specification the original authors never wrote down. If a field is not in that list, it is not part of the merge contract yet.

A messy module worth pinning

The example below is intentionally awkward on purpose. It mutates globals, prints progress on stderr, and formats money with ad hoc rounding. Treat it as labeled sample code, not as a measured production extract.

# export_invoices.py — constructed example, not production code
from __future__ import annotations

import json
import os
import sys
from decimal import Decimal, ROUND_HALF_EVEN

TAX = Decimal(os.environ.get("TAX_RATE", "0.0875"))
SEEN = []


def money(n: str) -> str:
    d = Decimal(n).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
    if os.environ.get("CURRENCY") == "USD":
        return f"${d}"
    return str(d)


def main(argv: list[str]) -> int:
    path = argv[1] if len(argv) > 1 else "-"
    raw = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read()
    rows = json.loads(raw) if raw.strip() else []
    exempt = os.environ.get("TAX_EXEMPT", "0") == "1"
    out = []
    for row in rows:
        SEEN.append(row["id"])
        base = Decimal(str(row["amount"]))
        tax = Decimal("0.00") if exempt or row.get("kind") == "internal" else (base * TAX)
        tax = tax.quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
        line = f"{row['id']},{money(str(base))},{money(str(tax))},{money(str(base + tax))}"
        out.append(line)
        print(f"wrote {row['id']}", file=sys.stderr)
    print("\n".join(out))
    print(f"count={len(SEEN)}", file=sys.stderr)
    return 0 if out else 2


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

That script is small enough to read and still unsafe to improve without a gate. Rounding, exemption flags, empty-input exit codes, and stderr chatter are all load-bearing. A rename-only patch can still break a downstream cut or a cron mail filter.

Record pins before any production edit

The recorder runs the current script as the oracle and does not interpret business rules. Label this harness as a proposed workflow, not as a benchmark with timings or pass rates. Commit the pins while the code is still ugly.

# record_pins.py — proposed recorder
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
PIN_PATH = ROOT / "pins.jsonl"


def run_case(case: dict) -> dict:
    env = os.environ.copy()
    for key in ("TAX_RATE", "TAX_EXEMPT", "CURRENCY"):
        env.pop(key, None)
    env.update(case.get("env") or {})
    stdin = case.get("stdin", "")
    proc = subprocess.run(
        [sys.executable, str(ROOT / "export_invoices.py"), *case.get("argv", ["-"])],
        input=stdin,
        text=True,
        capture_output=True,
        env=env,
        cwd=ROOT,
        check=False,
    )
    return {
        "id": case["id"],
        "argv": case.get("argv", ["-"]),
        "env": case.get("env") or {},
        "stdin_sha256": hashlib.sha256(stdin.encode()).hexdigest(),
        "stdin": stdin,
        "exit_code": proc.returncode,
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "cwd_rel": ".",
    }


CASES = [
    {
        "id": "empty-stdin",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD"},
        "stdin": "",
    },
    {
        "id": "mixed-kinds",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD", "TAX_EXEMPT": "0"},
        "stdin": json.dumps(
            [
                {"id": "A-1", "amount": "10.00", "kind": "external"},
                {"id": "A-2", "amount": "10.005", "kind": "internal"},
                {"id": "A-3", "amount": "0.015", "kind": "external"},
            ]
        ),
    },
    {
        "id": "org-exempt",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD", "TAX_EXEMPT": "1"},
        "stdin": json.dumps([{"id": "B-9", "amount": "99.99", "kind": "external"}]),
    },
]


def main() -> None:
    lines = [json.dumps(run_case(c), ensure_ascii=False) for c in CASES]
    PIN_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")
    print(f"wrote {len(lines)} pins to {PIN_PATH}")


if __name__ == "__main__":
    main()

Run the recorder once against the unedited script, then commit pins.jsonl in a change that contains no refactor. After that commit, the agent is not allowed to regenerate pins to make a patch pass.

python record_pins.py
git add export_invoices.py record_pins.py pins.jsonl
git commit -m "Pin invoice exporter I/O before any refactor"

Replay is the only merge gate

The test below rehydrates each pin and compares exit code, stdout, and stderr as strings. It is deliberately strict on whitespace and log wording. If you later need normalization, document that function in the test file instead of quietly editing pins.

# test_replay_pins.py
from __future__ import annotations

import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent


def load_pins():
    text = (ROOT / "pins.jsonl").read_text(encoding="utf-8")
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def replay(pin: dict) -> subprocess.CompletedProcess:
    env = os.environ.copy()
    for key in ("TAX_RATE", "TAX_EXEMPT", "CURRENCY"):
        env.pop(key, None)
    env.update(pin["env"])
    return subprocess.run(
        [sys.executable, str(ROOT / "export_invoices.py"), *pin["argv"]],
        input=pin["stdin"],
        text=True,
        capture_output=True,
        env=env,
        cwd=ROOT / pin["cwd_rel"],
        check=False,
    )


def test_every_pin_replays():
    failures = []
    for pin in load_pins():
        proc = replay(pin)
        if proc.returncode != pin["exit_code"]:
            failures.append(f"{pin['id']}: exit {proc.returncode} != {pin['exit_code']}")
        if proc.stdout != pin["stdout"]:
            failures.append(
                f"{pin['id']}: stdout mismatch\n---\n{proc.stdout!r}\n===\n{pin['stdout']!r}"
            )
        if proc.stderr != pin["stderr"]:
            failures.append(
                f"{pin['id']}: stderr mismatch\n---\n{proc.stderr!r}\n===\n{pin['stderr']!r}"
            )
    assert not failures, "\n\n".join(failures)

python -m pytest test_replay_pins.py -q

A tiny hook keeps the agent honest when it tries to refresh the oracle. The hook is a proposal you can drop into .git/hooks/pre-commit after reviewing it.

#!/bin/sh
# proposed pre-commit: fail if pins.jsonl changes beside production code
if git diff --cached --name-only | grep -q '^pins\.jsonl$'; then
  if git diff --cached --name-only | grep -q '^export_invoices\.py$'; then
    echo "refuse: pins.jsonl and export_invoices.py in the same commit"
    exit 1
  fi
fi

The first internal edit after green replay

Only after replay is green may the next patch touch internals, and only for one concern. A reasonable first edit in this example extracts rounding without altering prints. Keep main as the only process entry so argv and env stay in one place.

Proposed rules, labeled as unexecuted guidance rather than a completed refactor:

Move rounding into a pure function that main already calls.
Do not silence stderr lines that operators may grep in cron mail.
Do not change the empty-input exit code from 2 to 0.
Reject any patch that updates pins.jsonl in the same commit as production code.

If replay fails, revert the patch instead of editing the pin. The pin is older than the patch, which is the entire point of the gate. Green replay with a logging rewrite is still a contract break.

Decision table: keep the diff or stop the session

The table below is the workflow. Prompts around it are optional commentary and never override a red replay.

Observation after the patch	Action
Replay green, diff limited to one function	Keep the patch and end the session
Replay green, but the diff rewrites logging	Revert; logging is part of the CLI contract
Replay red on stdout only	Revert; treat it as a behavior change
Replay red on stderr only	Revert unless an operator signs off in review
Agent rewrites `pins.jsonl`	Reject the commit in hook and in CI
Agent adds features while cleaning	Reject; new behavior needs human-authored pins

Coverage holes remain possible. A hidden branch that no pin exercises will replay green and still be wrong. Add pins when you discover those branches in production logs, not when an agent asks for more freedom.

Where a disposable agent session belongs

An agent is useful after pins exist, because failed attempts then show up as red replay instead of plausible prose. It is not useful as a substitute for recording the current oracle on a dirty tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option fit this loop as a disposable place to attempt that single internal edit. Record pins on your own checkout, commit them, and copy only the repository snapshot onto that server. The merge decision stays with local replay, not with the session transcript. If you remove the product from the method, the pin-and-replay gate still works on a laptop.

Limitations, and who should skip this

Process-level pins are weak when the script talks to clocks, networks, or unordered maps. They also overfit to log wording, which is intended here and harmful if you already planned a logging redesign. Empty coverage of a hidden branch will look like success.

Skip this approach when any of the following are true.

The program is not a batch boundary, so pins cannot see GUI or long-lived server state.
Output includes timestamps, random identifiers, or unstable iteration order.
You need to specify new behavior rather than preserve old behavior.
The agent would receive production credentials or write access to real data.
A human cannot read pins.jsonl and explain each case in one sentence.

Clock and network seams should be injected before recording, not after the agent starts editing. If you cannot freeze time, you do not yet have a pin worth merging against. Do not treat green replay as proof that the tax rules are correct; it only proves they did not change.

A pull-request checklist instead of a prompt dump

Use this list as review notes. It is slower than letting the agent sweep the file, and that slowness is the control.

A human records pins against the current default branch with no code edits in that commit.
CI runs pytest test_replay_pins.py and fails the build if pins.jsonl is missing.
The agent receives one sentence of intent, the pin path, and a diff budget of one function.
A human reruns replay locally and inspects git diff --stat before review.
If the diff crosses logging, environment parsing, or exit codes, the patch is discarded.

Generation is no longer the scarce resource on a messy script. Agreement with the last known process boundary is. Lock that boundary in JSONL, then accept an agent diff only when replay still prints the same contract.

Characterize the Mess First, Then Make One Safe Change

Dakota Wu — Sat, 05 Sep 2026 19:45:29 +0000

A messy pricing helper is a common place where coding agents overreach, because the file looks small and the tests look green. Consider a labeled brownfield example: quoteTotal.js mixes tax flags, coupon fallbacks, and a loyalty multiplier that three checkout paths already call differently. An agent that sees sparse tests often proposes a clean rewrite and a renamed public API in one diff. That rewrite can stay green while two regional carts drift by a few cents, because nothing locked the old, slightly wrong behavior.

Characterization tests exist for that gap between a green suite and the behavior callers already shipped. They do not claim the current helper is correct, and they do not bless rounding quirks as a product strategy. They pin what the mess already does, including silent fallbacks, so the next edit has a contract.

Why a Green Suite Still Lies

Sparse unit tests usually encode the author's intention, not the production contract that three call sites already rely on. A single happy-path fixture cannot represent coupon casing, missing tax flags, or the zero-discount fallback operations already depend on. When an agent rewrites the helper, those tests still pass if the new code matches the fixture, even if every other branch moved. The recorded contract is the fixture, and the fixture is almost never the full matrix of flags.

The practical failure is not that agents generate large diffs on first contact with a messy file. The failure is that the repository never recorded the behavior the callers already shipped to production. Characterization tests reverse that order by recording first, then changing the smallest thing that still reduces risk.

A Compact Mess Worth Pinning

The module below is labeled sample code, not a dump from a production employer. It is intentionally awkward so the test strategy stays visible, including shared mutable state. Three callers already depend on fail-open coupons, default tax, and a NaN-to-zero collapse that a rewrite would “fix” without asking.

// quoteTotal.js — labeled messy example, not production source
let LOYALTY = 1.0;

function money(n) {
  return Math.round(n * 100) / 100;
}

function quoteTotal(subtotal, coupon, flags) {
  flags = flags || {};
  let total = Number(subtotal);
  if (coupon) {
    const code = String(coupon).trim().toUpperCase();
    if (code === "SAVE10") total = total * 0.9;
    else if (code === "SAVE5") total = total - 5;
    // unknown coupons silently do nothing
  }
  if (flags.tax === true) {
    total = total * (flags.rate != null ? flags.rate : 1.08);
  }
  if (flags.loyalty) {
    total = total * LOYALTY;
  }
  if (isNaN(total)) return 0;
  return money(total);
}

module.exports = {
  quoteTotal,
  setLoyalty: (v) => {
    LOYALTY = v;
  },
};

Global loyalty state leaks across tests when a file forgets to reset it after a run. Unknown coupons fail open, which reorder flows already treat as a valid no-op rather than an error. NaN becomes a zero total instead of an exception, and a batch invoicing job already swallows that zero.

Step 1: Inventory Callers Before Any Generated Tests

Do not start by asking a model to improve the module, because that prompt skips the contract you already shipped. Start by listing every call site and the flags it actually passes in the tree. The inventory is the specification you have, not the specification anyone wished they had written.

Search the repo for quoteTotal( and record file, coupon usage, and tax flags.
Note hidden coupling, especially the setLoyalty export and tests that never reset it.
Capture one production-shaped input per call site, including empty coupons and missing flags.
Write those inputs as a table before any test file exists, and refuse patches until the table is filled.

rg -n "quoteTotal\(" -g '!node_modules' -g '!dist'
rg -n "setLoyalty\(" -g '!node_modules' -g '!dist'

A short inventory table keeps later tests honest and stops an agent from inventing a fourth flags combination.

Call site	Coupon	`flags.tax`	`flags.rate`	Notes
`cart.js`	user input	`true`	`1.08` or `1.0`	trims whitespace on codes
`reorder.js`	stored code	omitted	omitted	relies on fail-open unknown coupons
`invoiceJob.js`	`null`	`true`	`1.0875`	needs two-decimal rounding

Step 2: Characterization Tests That Pin Quirks

A characterization test asserts the current result, even when that result is ugly or inconvenient. Name the tests after observed behavior, not after a future money type or a hoped-for API. Expected numbers below were computed from this implementation; they are a lock, not a finance sign-off.

// quoteTotal.characterization.test.js
const { quoteTotal, setLoyalty } = require("./quoteTotal");

afterEach(() => setLoyalty(1.0));

test("SAVE10 applies before default tax and rounds to cents", () => {
  expect(quoteTotal(100, "save10", { tax: true })).toBe(97.2);
});

test("SAVE5 subtracts a flat amount then applies an explicit rate", () => {
  expect(quoteTotal(40, "SAVE5", { tax: true, rate: 1.0875 })).toBe(38.06);
});

test("unknown coupons fail open instead of rejecting the quote", () => {
  expect(quoteTotal(50, "WELCOME", { tax: true })).toBe(54);
});

test("missing flags object is treated as no tax and no loyalty", () => {
  expect(quoteTotal(19.99, null)).toBe(19.99);
});

test("NaN subtotal collapses to zero rather than throwing", () => {
  expect(quoteTotal("n/a", "SAVE10", { tax: true })).toBe(0);
});

test("loyalty multiplies after tax when the global is not 1", () => {
  setLoyalty(0.95);
  expect(quoteTotal(100, null, { tax: true, loyalty: true })).toBe(102.6);
});

Run the file before changing production code, and treat a red fixture as a measurement error until a caller proves the implementation wrong. If the test disagrees with quoteTotal.js, keep the implementation and fix the expected value. The suite is a seismograph, not a style guide.

npx --yes jest quoteTotal.characterization.test.js --runInBand

--runInBand keeps the global loyalty mutation from racing other files in the same process. Characterization suites that share process state should stay serial until that global is removed on purpose. Commit the green suite as its own change, with no production edit in the same diff.

A tiny oracle script for new rows

When the inventory table grows, do not hand-compute rounding again if the lock is “what the file does today.” Print the current result, paste it into a test name that describes the quirk, and only then decide whether the quirk is allowed to survive.

// printQuote.js — labeled helper for capturing current results
const { quoteTotal } = require("./quoteTotal");
const rows = [
  [100, "save10", { tax: true }],
  [40, "SAVE5", { tax: true, rate: 1.0875 }],
  [50, "WELCOME", { tax: true }],
];
for (const [subtotal, coupon, flags] of rows) {
  console.log(JSON.stringify({ subtotal, coupon, flags, out: quoteTotal(subtotal, coupon, flags) }));
}

node printQuote.js

Step 3: The Smallest Safe Change

After the suite is green and committed, pick one mechanical edit that a reviewer can hold in working memory. Do not rename the public function, do not introduce a money class, and do not close the unknown-coupon path in the same patch. Smallest means one observable behavior change, or one structural extract with zero behavior change.

Useful first moves, ordered by risk rather than by how impressive the diff looks:

Reset or wrap the LOYALTY global so tests cannot leak, with no quote formula change.
Extract applyCoupon(total, coupon) without changing any branch result the suite already pinned.
Replace flags = flags || {} with a default parameter, then re-run the same file.
Only then consider failing unknown coupons, and only behind an explicit opt-in flag that old callers do not set.

Move 2 should leave every characterization assertion identical. If any assertion flips, the extract was not behavior-preserving and the patch should stop immediately.

function applyCoupon(total, coupon) {
  if (!coupon) return total;
  const code = String(coupon).trim().toUpperCase();
  if (code === "SAVE10") return total * 0.9;
  if (code === "SAVE5") return total - 5;
  return total;
}

function quoteTotal(subtotal, coupon, flags) {
  flags = flags || {};
  let total = applyCoupon(Number(subtotal), coupon);
  if (flags.tax === true) {
    total = total * (flags.rate != null ? flags.rate : 1.08);
  }
  if (flags.loyalty) {
    total = total * LOYALTY;
  }
  if (isNaN(total)) return 0;
  return money(total);
}

Agents fail this step when they clean up the fail-open coupon path during the extract, because the cleanup looks locally correct. Re-run the committed characterization file on that extract before any reviewer reads the new helper. A flipped assertion is cheaper than a regional cart that drifted after merge.

Decision Table: Characterize, Extract, or Stop

Print this table next to the pull request template so review comments point at a cell instead of at taste. The table is the workflow’s only scoring system: missing locks block work, mixed behavior changes reject the patch, and identical assertions accept structure-only edits.

Signal	Action	Reason
Tests miss a call-site flag combination	Add a characterization case first	The lock is incomplete
Diff rewrites rounding and coupon policy together	Reject the patch	Two behaviors moved at once
Extract keeps every assertion identical	Accept	Structure changed, contract did not
Agent wants a new public API	Defer until callers have adapters	Renames hide contract drift
`NaN`-to-zero is load-bearing for a batch job	Keep it, and name the test after the collapse	Correctness is a product decision

Where Free Model Access and a Free Server Fit

Generating the first characterization cases from a call-site inventory is repetitive work, and it does not require a paid evaluation cluster. MonkeyCode offers free model access and a free server option that can draft those tests and execute them in an isolated workspace.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The useful loop is narrow and should stay narrow. Paste the inventory table and the messy module, ask for characterization tests that pin current results, then run the suite on the free server before any refactor patch is proposed. Treat the model output as untrusted draft tests: every expected value must be re-executed against the real file, including rounding. Do not ask the model to fix rounding, coupons, or NaN handling until that suite is green and committed on its own.

This article does not claim specific model names, token quotas, hardware profiles, or how long a free server remains available. Those details change, and a workflow that depends on an unpublished quota is not a workflow a team can schedule. If a local Node install already runs Jest, the same inventory-to-lock loop works there; the free server is optional isolation, not a requirement for characterization tests. One isolated run of that loop is enough to see whether the sequence fits the repo.

Limitations and Who Should Skip This

Characterization tests freeze today’s behavior, including bugs that finance may already want gone next sprint. If a regulator already forbids the fail-open coupon path, pinning it without a removal ticket creates false confidence and a harder later change. Do not use this sequence on cryptographic code, authorization checks, or anything where preserving “what it does today” is itself unsafe.

The approach also wastes time on greenfield modules that already have an explicit contract and a complete unit suite. If the public behavior is already documented, write a normal regression test and make the small change without the extra ceremony. Skip it when the helper is ten lines, has one caller, and the agent diff is already smaller than the test file you would add.

A characterization suite that never lands in CI is theater, because the lock disappears as soon as the next rewrite arrives. If the team will not run Jest on that file in the default pipeline, stop at the inventory table and fix the pipeline first. Shared mutable state, like the loyalty global above, also limits parallel test runners until the extract removes it.

Repeatable Order, With Stopping Rules

The repeatable order is inventory, pin, extract, then one behavior change, and each step has a hard stop. Missing call sites block tests, a red characterization file blocks extracts, and a flipped assertion blocks the pull request. That order is slower than rewriting the module in one prompt, and it is the reason regional carts stop drifting by cents after a tidy-looking diff.

Keep the characterization file beside the messy module until each quirk is either removed on purpose or documented as product behavior in the test name. The smallest safe change is the one existing callers can survive without a silent change to money.