Freeze the Call Tape, Not Just the Output: A Characterization Test for Order-Dependent Refactors

A cleanup diff hoists a rates.rate() call out of a loop, the suite stays green, and the totals still match to the cent. Three days later the invoice numbers drift, because that lookup was never a pure getter — it was a cache with a TTL and a metric counter attached. Output-equality tests cannot see this class of change, and that is exactly the class of change that messy-repo refactors produce most often.

This post builds a small, reproducible artifact for that problem: a call tape that records which collaborators ran, in what order, and how many times. You pin the tape on the pre-refactor commit, then let the tape argue about every diff that follows. The example is stdlib Python and runs in about a second.

Why output equality is the wrong sole oracle

Most characterization suites freeze return values only, which is cheap and catches arithmetic regressions well. They miss three families of change that survive identical outputs:

• Call count: memoizing or hoisting a lookup changes how many times a collaborator executes.
• Call order: reordering a filter before a fetch changes what gets fetched, and when.
• Evaluation timing: converting an eager list into a generator moves work outside the try/except that used to guard it.

These are not exotic. They are the usual residue of "extract a helper" and "avoid repeated work" cleanup, which is precisely what an AI-assisted refactor proposes first.

Step 1: A tape recorder in twenty lines

The artifact is deliberately dumb: it wraps a function, appends a JSON-safe event, and delegates.

# tape.py
import json
from dataclasses import dataclass, field

@dataclass
class Tape:
    events: list = field(default_factory=list)

    def spy(self, name, fn):
        def recorded(*args, **kwargs):
            self.events.append([name, [repr(a) for a in args]])
            return fn(*args, **kwargs)
        return recorded

    def assert_matches(self, golden_path, update=False):
        actual = json.loads(json.dumps(self.events))
        if update:
            with open(golden_path, "w") as fh:
                json.dump(actual, fh, indent=2)
            return
        with open(golden_path) as fh:
            expected = json.load(fh)
        assert actual == expected, "tape drift:\n" + json.dumps(actual, indent=2)

repr() keeps the tape JSON-safe without teaching the recorder about your domain types. If a collaborator returns a timestamp or a UUID, normalize it before it enters the tape, or the golden file will flap forever.

Step 2: Pin the tape on the pre-refactor commit

Here is the function under test, in its original shape. It is messy enough to invite a cleanup and small enough to reason about.

# billing.py  (pre-refactor)
def summarize(orders, rates, fetcher):
    lines = []
    for order in orders:
        lines.append(order["qty"] * rates.rate(order["currency"]))
    if any(o.get("late_fee") for o in orders):
        lines.append(fetcher.late_fee())
    return round(sum(lines), 2)

The collaborators are fakes that route through the tape:

# fakes.py
class Rates:
    def __init__(self, tape, table):
        self.table = table
        self.rate = tape.spy("rates.rate", self._rate)

    def _rate(self, currency):
        return self.table[currency]

class Fetcher:
    def __init__(self, tape, late_fee):
        self._late_fee = late_fee
        self.late_fee = tape.spy("fetcher.late_fee", lambda: self._late_fee)

And the test asserts both the number and the tape:

# tests/test_tape.py
import os, pathlib
from tape import Tape
from billing import summarize
import fakes

GOLDEN = pathlib.Path("golden/summarize_tape.json")

def test_summarize_output_and_tape():
    tape = Tape()
    rates = fakes.Rates(tape, {"USD": 12.5})
    fetcher = fakes.Fetcher(tape, late_fee=7.0)
    orders = [
        {"qty": 4, "currency": "USD"},
        {"qty": 2, "currency": "USD", "late_fee": True},
    ]
    out = summarize(orders, rates, fetcher)
    assert out == 82.0
    tape.assert_matches(GOLDEN, update=bool(os.environ.get("RECORD")))

Record the golden exactly once, on the commit you trust, and then never record on a branch:

RECORD=1 python -m pytest -q tests/test_tape.py   # only on the pre-refactor commit
git add golden/summarize_tape.json && git commit -m "pin call tape"
python -m pytest -q tests/test_tape.py            # replay mode from here on

The pinned tape for this input is three events: rates.rate('USD'), rates.rate('USD'), fetcher.late_fee(). That duplication is now a written contract rather than an accident.

Step 3: Make the smallest safe change and let the tape argue

Now the plausible cleanup: hoist the lookup and memoize per currency. Output is still 82.0, so an output-only suite signs off.

# billing.py  (candidate refactor)
def summarize(orders, rates, fetcher):
    cache, lines = {}, []
    for order in orders:
        cur = order["currency"]
        if cur not in cache:
            cache[cur] = rates.rate(cur)
        lines.append(order["qty"] * cache[cur])
    if any(o.get("late_fee") for o in orders):
        lines.append(fetcher.late_fee())
    return round(sum(lines), 2)

The tape now reports one rates.rate event instead of two, and the assertion fails with a readable diff. That failure is the useful part of the workflow: it converts an invisible semantic decision into a reviewable one. You either revert, or you accept the memoization as a behavior change and attach a new test that asserts the new call count deliberately.

I apply the same gate to the laziness variant. Turning the line builder into a generator is a two-character-looking change that moves rates.rate execution outside any surrounding try/except. The tape does not catch that alone, so pair it with an explicit test that a failing collaborator still surfaces as the expected exception type at the expected boundary.

The three-signal rule for accepting tape drift

A golden change is never auto-accepted. Require three signals in the diff description, in this order:

1. Output identical: the return-value assertion passes without modification.
2. One-sentence delta: the call-count or order change is explained in a single sentence a reviewer can falsify.
3. New intentional test: the new behavior has its own test, so the tape is not the only witness.

If any signal is missing, revert and split the commit. One seam per commit keeps the tape diff small enough to read, which is the only reason this gate stays cheap.

What to freeze, and what to leave alone

Signal	Catches	Cost	Skip when
Return values	arithmetic drift	low	never
Call tape (order + count)	hoisting, memoization, laziness	medium	functions with no collaborators
Exception type and timing	try/except scope drift	medium	nothing in the seam can raise
Wall-clock latency	performance regressions	high	any refactor with a flaky CI box

Keep the table honest: the tape is a behavioral contract, latency is not. Freezing timings produces reruns, not evidence.

Running the loop without new infrastructure

Two places in this workflow need tooling rather than judgment, and both are cheap to outsource.

First, generating adversarial inputs for the seam. I paste the seam function and its fake collaborators into a model and ask for an input table that maximizes distinguishable tapes: empty orders, mixed currencies, a late fee without a matching order, a currency missing from the table. Model suggestions are proposals, not evidence — each case earns its place only if it produces a distinct tape or a distinct exception, which you verify locally.

Second, replaying the harness somewhere other than a laptop. The free model access and the free server option in MonkeyCode cover exactly those two steps; I use the server to run the replay suite on a branch with synthetic fixtures and no production credentials. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability, quotas, and limits are operator-supplied and do change, so confirm the current terms on the project page before you build a workflow around them — and check the license if you intend to vendor anything.

Limitations and who should skip this

• No seam, no tape. If domain logic calls the database inline, wrap the call site first; a recorder cannot spy on what it cannot be injected into.
• Non-deterministic collaborators. Timestamps, random IDs, and retries must be normalized or the golden file is noise.
• Intentionally unordered work. If call order is genuinely unspecified, freeze the set of calls rather than the sequence.
• Performance refactors. Here the tape will fight you by design; use a benchmark, not a golden file.

If you only ever adopt one habit from this post, make it the pinned tape on the trusted commit before the first cleanup diff lands. If you want to try the loop end to end without provisioning anything, the free model access and free server option are the shortest path to one green replay — and keep the golden files in your own repository, where the next reviewer can read them.