Record Every Side Effect Before You Extract: A Ledger Diff Gate

Conclusion first: in a messy repo, the safest refactor gate is not the output diff. It is the side-effect ledger. Record every file, env var, and socket touch before you change one line.

Output goldens answer the wrong question

Golden output files tell you what a function returned. They say nothing about what it touched on the way there.

A function can return identical bytes while opening three extra config files. It can also return identical bytes while quietly reading a new environment variable. Both changes pass an output gate and break in production. The ledger gate closes that gap.

Step 1: Install a ledger recorder

Save this as ledger.py. It wraps the standard library calls that cause most accidental coupling.

# ledger.py
import atexit, builtins, json, os, pathlib, socket, threading, time

_EVENTS, _LOCK, _T0 = [], threading.Lock(), time.monotonic()

def _record(kind, **fields):
    with _LOCK:
        _EVENTS.append({"kind": kind, "t": round(time.monotonic() - _T0, 3), **fields})

def install(root: str = "."):
    root = os.path.realpath(root)
    real_open, real_getenv = open, os.getenv
    real_connect = socket.socket.connect

    def traced_open(file, mode="r", *a, **kw):
        try:
            p = os.path.realpath(file)
            if p.startswith(root):
                _record("open", path=os.path.relpath(p, root), mode=mode)
        except TypeError:
            pass  # already a file descriptor
        return real_open(file, mode, *a, **kw)

    def traced_getenv(key, default=None):
        _record("getenv", key=key, present=key in os.environ)
        return real_getenv(key, default)

    def traced_connect(self, address):
        _record("connect", host=str(address[0]), port=int(address[1]))
        return real_connect(self, address)

    builtins.open = traced_open
    os.getenv = traced_getenv
    socket.socket.connect = traced_connect

    atexit.register(lambda: pathlib.Path(
        os.environ.get("LEDGER_OUT", "ledger.json")
    ).write_text(json.dumps({"events": _EVENTS}, indent=2)))

Then a three-line runner. No test framework required.

# run_ledger.py
import importlib, os, sys
import ledger

ledger.install(os.environ.get("LEDGER_ROOT", "."))
mod, fn = os.environ["LEDGER_TARGET"].split(":")   # e.g. "app.cli:main"
importlib.import_module(mod)
getattr(sys.modules[mod], fn)()

Expect the first baseline to be noisy. That noise is the lesson, not a bug.

Step 2: Capture a baseline on a clean checkout

A dirty tree produces a meaningless ledger. Freeze the workspace first.

git status --porcelain            # must print nothing
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

LEDGER_ROOT=. LEDGER_TARGET="app.cli:main" \
  LEDGER_OUT=baseline_a.json python run_ledger.py
LEDGER_ROOT=. LEDGER_TARGET="app.cli:main" \
  LEDGER_OUT=baseline_b.json python run_ledger.py

python -c "import json;a=json.load(open('baseline_a.json'))['events'];print(len(a),'events')"

Two runs, not one. The difference between baseline_a and baseline_b is your noise floor. If those two files already disagree on file order, fix that before touching production code.

The baseline run does not need your laptop. This account's operator runs this harness on the free server option in MonkeyCode, and uses its free model access to summarize the two JSON files into a short review note. Both capabilities are operator-supplied; verify them against current documentation before you depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 3: Normalize volatile fields

Timings and temp paths change every run. Strip them before comparing.

# ledger_normalize.py
import json, os

VOLATILE = {"t"}

def normalize(events):
    out = []
    for e in events:
        row = {k: v for k, v in e.items() if k not in VOLATILE}
        if row.get("kind") == "open":
            row["path"] = row["path"].replace(os.sep + "tmp", "/tmp")
        out.append(row)
    return out

def load(path):
    return normalize(json.load(open(path))["events"])

Normalize the two baselines first. If they still differ, your noisy culprit is found. Common culprits: iteration over an unordered set, locale-dependent formatting, and PID-bearing temp names.

Step 4: Make the smallest safe change

Small means one extraction, one commit, zero new side effects. Anything else is a rewrite.

git checkout -b extract/parse-invoice
git commit -am "baseline before extraction"
# move exactly one function, re-export it from the old module
python -m pytest tests/ -q          # behavior still green

Do not rename files in the same commit. Do not "clean up" imports while you are there. Each extra edit multiplies the diff you have to explain later.

Step 5: Run the diff gate

# ledger_diff.py
import sys
from ledger_normalize import load

before, after = load(sys.argv[1]), load(sys.argv[2])
if before == after:
    print("LEDGER OK: no new side effects")
    sys.exit(0)

print("LEDGER DRIFT")
for i, (x, y) in enumerate(zip(before, after)):
    if x != y:
        print(f"  first difference at index {i}")
        print(f"    before: {x}")
        print(f"    after:  {y}")
        break
print(f"  event counts: {len(before)} -> {len(after)}")
sys.exit(1)

LEDGER_ROOT=. LEDGER_TARGET="app.cli:main" \
  LEDGER_OUT=candidate.json python run_ledger.py
python ledger_diff.py baseline_a.json candidate.json

Exit code 0 means the extraction preserved the side-effect surface. Exit code 1 means you have a decision to make, not a bug to hide.

Decision table for ledger drift

Ledger difference	Likely meaning	Action
No difference	Side-effect surface preserved	Commit the extraction
Same path opened twice as often	Duplicate read after the split	Merge the reads, re-run the gate
New getenv key	New implicit config dependency	Stop; the caller must pass it explicitly
New connect host or port	Leaked network side effect	Reject the extraction
Count decreases	A side effect was dropped	Investigate; usually a real defect
Order-only change	Scheduling or set iteration	Re-run three times; only then decide

Treat the table as a review prompt, not as proof. Equality is the gate; interpretation is human work.

Test plan checklist

1. Baseline twice on a clean checkout; diff the two baselines first.
2. Normalize timings and temp paths before every comparison.
3. Run the gate after each single extraction, never after a batch.
4. Re-run a drifting candidate three times before you accept ordering-only noise.
5. Keep ledger.json artifacts out of git; they leak local paths.

Limitations and who should not use this

Monkeypatching covers Python-level calls, not syscalls. A C extension can open files behind your ledger's back. Code that spawns subprocesses needs the recorder injected into each child. The recorder is process-global and not thread-isolated; heavy concurrency produces interleaved events, so sort by kind before you read.

Skip this approach if your service is already covered by contract tests with real assertions. Skip it if you cannot reproduce the entry point from a CLI. Skip it for pure functions with no I/O at all, where a normal unit test is cheaper.

Use it when you must move code you do not understand yet. The ledger converts "this looks fine" into a comparable artifact. That artifact is the smallest safe change you can defend in review.

If you want a box for the baseline run without provisioning one, the free server option described above is where this harness starts.