Freeze a Side-Effect Ledger Before One Formatter Extract

A model can rewrite a messy module in seconds. That extra speed is not a merge signal. Freeze stdout, sidecar files, and mutable accumulators first.

Extract one formatter after those observables are pinned. Leave the rest of the untested tree untouched.

This workflow is a gate, not a vibe check. It fits untested Python that mixes I/O with string building. It does not fit greenfield design or broad API redesigns.

Why "just clean it up" fails on messy trees

Untested trees hide order, encoding, and mutation bugs. A formatter change can alter JSON key order. A helper extract can skip a required sidecar write.

A sort cleanup can reorder stable invoice identifiers. AI-assisted diffs make this failure cheaper to ignore.

The generated patch often looks fluent and complete. The characterization tests usually do not exist yet. Reviewers then approve tone instead of exact bytes.

The countermeasure is a recorded side-effect ledger. Record every observable the current code already emits. Assert that ledger after exactly one small change.

Current AI coding threads reward confident looking output. They rarely reward frozen bytes over fluent prose. This article treats that confidence as untrusted input.

The messy module under test

Treat the following module as a fixture, not production history. It mixes parsing, mutation, printing, and a sidecar file. That mix is the actual bug surface here.

# invoice_summary.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

TOTALS: dict[str, float] = {}


def run(data_dir: str, out_dir: str) -> dict[str, Any]:
    root = Path(data_dir)
    dest = Path(out_dir)
    dest.mkdir(parents=True, exist_ok=True)
    rows: list[dict[str, Any]] = []
    for path in root.glob("*.json"):
        payload = json.loads(path.read_text(encoding="utf-8"))
        code = str(payload.get("code") or path.stem)
        amount = float(payload.get("amount") or 0)
        TOTALS[code] = TOTALS.get(code, 0.0) + amount
        rows.append({"code": code, "amount": amount, "file": path.name})
    rows.sort(key=lambda r: (r["code"], r["file"]))
    lines = []
    for row in rows:
        lines.append(f"{row['code']}:{row['amount']:.2f}")
    text = "\n".join(lines) + "\n"
    print(text, end="")
    (dest / "summary.txt").write_text(text, encoding="utf-8")
    (dest / "summary.ok").write_bytes(b"ok\n")
    return {"count": len(rows), "codes": [r["code"] for r in rows]}

Note the process-wide TOTALS map in module scope. Note glob order feeding a later explicit sort. Note print output plus two on-disk artifacts.

A model will want to simplify all of that at once. Refuse any patch that wide on the first pass. Keep the next diff to one pure helper.

Artifact: the side-effect ledger test

Build a ledger that freezes four independent observables. Do not mock Path and do not stub print. Use a temp directory and capture stdout instead.

# test_invoice_summary_ledger.py
from __future__ import annotations

import io
import json
import sys
from pathlib import Path

import invoice_summary as mod


def write_fixture(root: Path) -> None:
    (root / "b.json").write_text(
        json.dumps({"code": "B-9", "amount": 10}), encoding="utf-8"
    )
    (root / "a.json").write_text(
        json.dumps({"code": "A-1", "amount": 2.5}), encoding="utf-8"
    )
    (root / "a2.json").write_text(
        json.dumps({"code": "A-1", "amount": 0.5}), encoding="utf-8"
    )


def capture_run(data: Path, out: Path) -> tuple[dict, str]:
    buf = io.StringIO()
    old = sys.stdout
    sys.stdout = buf
    try:
        result = mod.run(str(data), str(out))
    finally:
        sys.stdout = old
    return result, buf.getvalue()


def test_side_effect_ledger(tmp_path: Path) -> None:
    mod.TOTALS.clear()
    data = tmp_path / "in"
    out = tmp_path / "out"
    data.mkdir()
    write_fixture(data)

    result, stdout = capture_run(data, out)

    ledger = {
        "stdout": stdout,
        "summary": (out / "summary.txt").read_text(encoding="utf-8"),
        "ok_bytes": (out / "summary.ok").read_bytes(),
        "totals": dict(mod.TOTALS),
        "result": result,
    }
    assert ledger["stdout"] == "A-1:2.50\nA-1:0.50\nB-9:10.00\n"
    assert ledger["summary"] == ledger["stdout"]
    assert ledger["ok_bytes"] == b"ok\n"
    assert ledger["totals"] == {"A-1": 3.0, "B-9": 10.0}
    assert ledger["result"] == {
        "count": 3,
        "codes": ["A-1", "A-1", "B-9"],
    }

Run the ledger before any extract attempt starts.

python -m pytest test_invoice_summary_ledger.py -q

The ledger is the merge contract for this file. Duplicate codes stay duplicated in the line list. Totals still merge while sidecar bytes stay ASCII.

A formatter extract must not "fix" those behaviors. Duplicate invoice lines are the current product behavior. Silent cleanup would be an unreviewed semantic change.

Decision table: what the next diff may touch

Use this table as the change budget. Read each row before sending a model prompt.

Observable	Frozen	Allowed in the next diff
stdout text	yes	no
summary.txt bytes	yes	no
summary.ok bytes	yes	no
TOTALS keys and values	yes	no
return codes list	yes	no
sort key (code, file)	yes	no
glob walk internals	no	yes, if sort still holds
helper function names	no	yes, one extract only
comments beside the helper	no	yes

One allowed-change row is the entire budget. Two helper extracts belong in a later pull request. Comments are allowed only beside the new function.

Numbered workflow

Follow these seven steps in this strict order.

1. Copy the messy file onto a dedicated branch first.
2. Add the ledger test with tiny explicit fixtures only.
3. Run pytest and confirm a green baseline on current bytes.
4. Write a one-extract prompt that bans extra edits.
5. Apply the model diff onto a second working tree.
6. Re-run the ledger and reject stdout or sidecar drift.
7. Merge only the helper plus the ledger test.

Step four is where a free coding model can help. It does not replace step three or step six. Local pytest remains the only merge oracle.

Prompt that keeps the change small

Label this prompt as unexecuted guidance for now. Paste it only after the ledger is green.

Extract a pure function format_rows(rows) -> str from run().
Do not rename run.
Do not clear TOTALS.
Do not change print, glob, or sidecar writes.
Do not dedupe codes.
Keep the sort key (code, file).
Keep two-decimal amounts and a trailing newline.
Return a unified diff for invoice_summary.py only.

The prompt is a fence around change scope. Models expand cleanup without an explicit fence. Discard the patch if test files also change.

Where free model access belongs

A characterization loop needs a throwaway proposal environment. Local pytest should stay on your own machine. The model only proposes the formatter extract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Use that pair for the proposal step, not ledger execution. Keep secrets off that server and keep fixtures synthetic.

Drop the patch if the diff edits more than format_rows. Re-prompt with the same fence and no extra files. Do not accept a wide cleanup because inference was free.

If you try free model access, keep the ledger as merge gate. Do not treat a free server as a substitute test runner.

Expected extract shape

The smallest safe change looks like the snippet below. Anything larger is a different task and a different review.

def format_rows(rows: list[dict[str, Any]]) -> str:
    lines = [f"{row['code']}:{row['amount']:.2f}" for row in rows]
    return "\n".join(lines) + "\n"


def run(data_dir: str, out_dir: str) -> dict[str, Any]:
    root = Path(data_dir)
    dest = Path(out_dir)
    dest.mkdir(parents=True, exist_ok=True)
    rows: list[dict[str, Any]] = []
    for path in root.glob("*.json"):
        payload = json.loads(path.read_text(encoding="utf-8"))
        code = str(payload.get("code") or path.stem)
        amount = float(payload.get("amount") or 0)
        TOTALS[code] = TOTALS.get(code, 0.0) + amount
        rows.append({"code": code, "amount": amount, "file": path.name})
    rows.sort(key=lambda r: (r["code"], r["file"]))
    text = format_rows(rows)
    print(text, end="")
    (dest / "summary.txt").write_text(text, encoding="utf-8")
    (dest / "summary.ok").write_bytes(b"ok\n")
    return {"count": len(rows), "codes": [r["code"] for r in rows]}

Check three facts after the patch applies cleanly. format_rows must stay a pure function without I/O. run must still mutate TOTALS and write both files.

Re-run the ledger on the patched tree immediately. Matching stdout is not optional after the extract. Matching sidecar bytes is also not optional here.

Failure analysis

These failures show up often in model diffs. Map each one to a ledger assertion before retrying.

1. Deduped codes: both A-1 lines must remain in stdout.
2. Float drift: amounts stay two decimals, including 10.00.
3. Newline drift: the joined text keeps a trailing newline.
4. Sidecar drift: summary.ok stays exactly ok plus newline.
5. Global cleanup: TOTALS must not reset inside the extract.
6. Sort drift: ordering stays by (code, file), not glob walk.

Each failure is a byte mismatch, not a style debate. Retry with a tighter prompt after updating the fence. Do not relax the ledger to make the model pass.

Limitations

This method does not prove functional correctness at all. It only proves the current bytes stayed put. Wrong totals in the fixture remain wrong after the extract.

The ledger does not pin wall-clock time. It does not pin glob order before the sort. It does not pin JSON key order inside source fixtures.

Do not use this as a security review. Do not use this for concurrent writers on TOTALS. Do not treat a green ledger as a license to rewrite the package.

Networked invoice feeds are out of scope here. So are database transactions and multi-process accumulators. Those need different oracles than a sidecar file.

Who should skip this approach

Skip this if the module already has real contract tests. Skip this if the change is a semantic fix, not an extract. Skip this if the sidecar is a live lock file.

Skip this if you cannot run pytest on a local tree. A remote model is not your test runner. Skip this if production secrets cannot be reduced to fixtures.

Skip this for binary formats you cannot snapshot cheaply. Skip this when legal output must change in the same patch. Combine a semantic fix with an extract only under new tests.

Close

Start with the ledger, then extract one pure function. Re-run the four observables before any merge. That sequence is the whole method for this file.

A free model can draft the helper text. The ledger still decides whether the diff lands. Keep proposal and oracle on separate machines when you can.