Pin the Order of Side Effects Before the Refactor: An Effect Ledger Gate

Return values pass. Side effects drift. That is the refactor bug your tests never catch.

A characterization suite asserts outputs. It does not assert that the invoice file is written once, then the email, then the log line. Reorder those three and every assertion stays green.

This post adds one gate to the loop you already run: characterization tests first, smallest safe change second. The gate is an effect ledger. Record the ordered list of boundary crossings before the change. Re-record after. Diff the two lists.

The reference implementation below is about sixty lines of Python. It stores digests, not payloads, so secrets stay out of the fixture.

Why the return value is a weak gate

A function can return an identical dict and still behave differently. It may call the payment API twice, or write the export before validating it. Return-value tests see one snapshot at the end. The boundary sees a sequence.

Sequences are cheap to pin. Each event is a small tuple: kind, target, digest. For most single-threaded code the list is fully deterministic.

That determinism is the whole trick. You are not testing correctness. You are testing that nothing moved.

Step 1 — Inventory the effect boundaries

Before writing any test, list where the code touches the outside world. Grep the module for six families of calls.

rg -n "open\(|\.write\(|requests\.|httpx\.|session\.|logging\.|logger\.|datetime\.now|time\.time|random\.|uuid4|os\.environ" app/billing/

Then classify every hit. Separate these kinds: fs.write, http.out, log, clock, random, env.read, db.exec. Seven buckets cover most backend code.

This labeling pass is where a free model is genuinely useful. Paste the grep output and ask for a table: file, line, candidate kind, confidence. Free model access is enough for that job, because it is classification, not reasoning.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not accept the table as final. Dynamic dispatch hides boundaries: getattr, DI containers, decorators that wrap clients. Read the call sites yourself and delete wrong rows.

Step 2 — Install the recorder

Reference implementation. Python 3.10+, standard library only.

# ledger.py
import hashlib, json

KINDS = ("fs.write", "http.out", "log", "clock", "random", "env.read", "db.exec")


class Ledger:
    def __init__(self):
        self.events = []

    def record(self, kind, target, payload):
        assert kind in KINDS, kind
        blob = json.dumps(payload, sort_keys=True, default=str)
        self.events.append({
            "seq": len(self.events) + 1,
            "kind": kind,
            "target": str(target),
            "digest": hashlib.sha256(blob.encode()).hexdigest()[:16],
        })

    def trace(self):
        return [f'{e["kind"]}|{e["target"]}|{e["digest"]}' for e in self.events]

    def dump(self, path):
        with open(path, "w", encoding="utf-8") as f:
            json.dump(self.trace(), f, indent=2)

Note what is missing: payloads. Only digest and target survive. That keeps tokens, emails, and account numbers out of the fixture file.

Now the patch set. One context manager, one patch per boundary you confirmed in step 1.

# seams.py
import builtins, contextlib, hashlib
from datetime import datetime, timezone
from unittest import mock

FROZEN = datetime(2026, 1, 1, tzinfo=timezone.utc)


@contextlib.contextmanager
def effects(ledger, pay_client, clock):
    real_open = builtins.open

    def traced_open(file, mode="r", *a, **kw):
        if any(flag in mode for flag in ("w", "a", "x", "+")):
            ledger.record("fs.write", file, {"mode": mode})
        return real_open(file, mode, *a, **kw)

    def traced_send(url, **kw):
        payload = kw.get("json") or kw.get("data") or {}
        ledger.record("http.out", url, payload)
        return mock.Mock(status_code=202, json=lambda: {"ok": True})

    def traced_now():
        ledger.record("clock", "utcnow", {})
        return FROZEN

    with mock.patch("builtins.open", traced_open), \
         mock.patch.object(pay_client, "post", traced_send), \
         mock.patch.object(clock, "now", traced_now):
        yield

The frozen clock does two jobs at once. It records the read. It also removes wall-clock noise from payload digests.

Step 3 — Capture the baseline on a clean box

Run the capture where the machine cannot leak state. Your laptop has leftover files, a dirty environment, and a real clock.

The free server option fits this step. It gives you a disposable box for the capture run; nothing about the box needs to be fast. Copy the repo, capture, keep the JSON.

python -m tests.capture --scenario invoice_reissue --out tests/ledger/invoice_reissue.json
python -m tests.capture --scenario invoice_reissue --out /tmp/second.json
cmp tests/ledger/invoice_reissue.json /tmp/second.json && echo deterministic

Short capture driver:

# tests/capture.py
import argparse, importlib
from ledger import Ledger
from seams import effects

SCENARIOS = {
    "invoice_reissue": ("app.billing", "reissue_open_invoices", {"account": "acct_1"}),
}


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--scenario", required=True)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()

    mod_name, fn_name, kwargs = SCENARIOS[args.scenario]
    mod = importlib.import_module(mod_name)
    ledger = Ledger()
    with effects(ledger, mod.pay_client, mod.clock):
        getattr(mod, fn_name)(**kwargs)
    ledger.dump(args.out)

Run it twice before trusting it. If the two files differ, stop. Non-determinism means the gate is not ready. Fix the source first: sort dict iteration, inject the clock, seed randomness.

Step 4 — Gate the change in pytest

List equality is the check. The useful part is the first divergence index, not a wall of diff text.

# tests/test_effect_ledger.py
import json, pathlib, pytest
from ledger import Ledger
from seams import effects
from app import billing

FIXTURES = pathlib.Path("tests/ledger")


@pytest.mark.parametrize("scenario", ["invoice_reissue"])
def test_effect_order_unchanged(scenario):
    baseline = json.loads((FIXTURES / f"{scenario}.json").read_text())
    ledger = Ledger()
    with effects(ledger, billing.pay_client, billing.clock):
        billing.reissue_open_invoices(account="acct_1")
    got = ledger.trace()

    if got != baseline:
        first = next(
            (n for n, (a, b) in enumerate(zip(got, baseline)) if a != b),
            min(len(got), len(baseline)),
        )
        rows = []
        for n in range(max(0, first - 2), first + 3):
            old = baseline[n] if n < len(baseline) else "-"
            new = got[n] if n < len(got) else "-"
            rows.append(f"  {n:>3} baseline={old}\n      got     ={new}")
        pytest.fail(f"effect order diverged at index {first}\n" + "\n".join(rows))

Now make the smallest safe change. Extract one function. Move one block. Do not edit behavior.

Run the gate again. Green means the effects did not move. That is the only claim you get to make.

Step 5 — Read the divergence, decide, repeat

A diverged index is information, not automatic failure. Three cases cover almost everything.

1. Index shifted, digest identical. You added a read or reordered a log line. Usually harmless; refresh the baseline in its own commit.
2. Digest changed, count identical. The payload shape moved. Inspect before accepting anything.
3. Count changed. A boundary crossing appeared or vanished. This is the double-charge class, and it deserves a real review.

Never refresh the baseline inside the refactor commit. Keep the refresh separate so reviewers see intent.

Decision table — which gate to add first

Symptom in the messy repo	Gate	Typical cost
Function has no tests at all	Characterization tests on return values	hours
Return values pinned, I/O order unproven	Effect ledger (this post)	about an hour
Tests exist but may be unable to fail	Kill-rate or mutation gate	about two hours
Change must be mechanical only	AST diff gate	about an hour
Public surface must not move	Export manifest gate	about thirty minutes

Cheap gates run first. Each one closes a different failure class, so they compose instead of overlapping.

Limitations

• Concurrency breaks sequence determinism. Record per thread or serialize the scenario.
• Digest-only ledgers hide the why. Keep redacted payloads in a private artifact when you truly need them.
• Cross-process effects escape the patch set: queues, workers, cron jobs, subprocess calls.
• Model-classified boundaries miss getattr dispatch and DI wiring. Human review is mandatory.
• If the refactor is supposed to change effects, freeze the target list first. This gate cannot separate intended from accidental.

Who should skip this

• No callable entry point yet. Capture inputs first, then come back.
• Heavy metaprogramming where seams are not nameable. Patch at the adapter layer instead.
• Effects already covered by contract tests at the integration boundary.

What you end up with

One JSON file per scenario. One pytest gate. A first-divergence index whenever the refactor moves something it should not.

That file is the cheapest insurance in the loop. Capture it before the change, never after. If you want the boundary-classification pass without a paid key, free model access handles it; the free server option handles the clean capture run.