The Call Order Is the Contract: Trace Fixtures Before a Messy-Repo Extraction

Call order is part of a module's contract. Most extraction refactors keep the output identical and silently change the sequence. The workflow below records an ordered call trace before the move, then diffs it after. You shrink the change until every difference has a name.

Output fixtures answer "same result?". They never answer "same path?". A refactor can call the discount function twice and still return the correct total. It can skip a validation call that only fires on error paths. Byte pinning catches neither of those.

So pin the path. Use the standard library tracer. No plugin, no service, no dashboard.

1. Record on the Commit You Have Not Touched

The "before" trace must come from the current code. Record it first, commit the fixture, then refactor. Recording after the change only proves the new code agrees with itself.

# tools/trace_record.py
import json, os, sys

PKG_ROOT = os.path.abspath("src")
SECRET_KEYS = {"token", "secret", "password", "api_key"}


def _scalar(name, value):
    if name.lower() in SECRET_KEYS:
        return "<redacted>"
    if isinstance(value, (bool, int, float)):
        return repr(value)
    if isinstance(value, str) and len(value) <= 40:
        return repr(value)
    return f"<{type(value).__name__}>"


def record(target, *args, **kwargs):
    events = []

    def tracer(frame, event, arg):
        if event == "call":
            code = frame.f_code
            if os.path.abspath(code.co_filename).startswith(PKG_ROOT):
                n = code.co_argcount + code.co_kwonlyargcount
                names = code.co_varnames[:n]
                fn = getattr(code, "co_qualname", code.co_name)
                events.append({
                    "fn": f'{frame.f_globals.get("__name__", "?")}::{fn}',
                    "args": {k: _scalar(k, frame.f_locals.get(k)) for k in names},
                })
        return tracer

    previous = sys.gettrace()
    sys.settrace(tracer)
    try:
        target(*args, **kwargs)
    finally:
        sys.settrace(previous)
    return {"events": events}

sys.settrace fires for every new frame in the current thread. Filter to your package root or the fixture fills with library noise. Arguments are read from frame.f_locals at call time, which is exactly when they are bound.

Capture scalars only. Objects collapse to <ClassName>. That keeps fixtures diffable and stops repr of large models from leaking data into a committed file.

2. Pick a Corpus You Can Argue About

One operation. Three to six inputs. Include the boring case, one branch case, one failure case.

# tools/record_corpus.py
import json
from pathlib import Path
from trace_record import record

from legacy.billing import build_invoice

CASES = {
    "single_item": lambda: build_invoice({"sku": "A1", "qty": 1}),
    "discount": lambda: build_invoice({"sku": "A1", "qty": 3, "coupon": "TEN"}),
    "empty_qty": lambda: build_invoice({"sku": "A1", "qty": 0}),
}

out = Path("tests/fixtures/trace")
out.mkdir(parents=True, exist_ok=True)
for name, call in CASES.items():
    trace = record(call)
    (out / f"{name}.json").write_text(json.dumps(trace, indent=1, sort_keys=True))
    print(f"{name}: {len(trace['events'])} events")

Commit tests/fixtures/trace/*.json. Size beats coverage here. A fixture over a few thousand events is slow to diff and slow to review.

3. Normalize Before You Compare

Raw traces are noisy. Four rules tame them.

1. Drop frames outside src/.
2. Drop dunders and loggers listed in NOISE (__repr__, __eq__, __hash__, log, debug).
3. Map moved modules through MODULE_MOVES.
4. Redact argument names in SECRET_KEYS during recording.

Rule 3 is the honest one. Moving a function to a new module changes key. Declare that move once in the diff tool, and the reviewer sees it as data, not as a surprise.

4. Diff Into Three Buckets

# tools/trace_diff.py
import json, sys
from pathlib import Path

MODULE_MOVES = {}  # {"legacy.billing": "invoicing.core"}
NOISE = {"__repr__", "__eq__", "__hash__", "log", "debug"}
ALLOW = Path("tests/fixtures/trace/allow.txt")


def load(path):
    keys = []
    for e in json.loads(Path(path).read_text())["events"]:
        module, _, fn = e["fn"].partition("::")
        if fn.split(".")[-1] in NOISE:
            continue
        module = MODULE_MOVES.get(module, module)
        args = ",".join(f"{k}={v}" for k, v in sorted(e["args"].items()))
        keys.append(f"{module}::{fn}({args})")
    return keys


def main(before_path, after_path):
    before, after = load(before_path), load(after_path)
    b, a = set(before), set(after)
    allowed = {
        line.strip()
        for line in ALLOW.read_text().splitlines()
        if line.strip() and not line.startswith("#")
    }
    missing = sorted(k for k in b - a if k not in allowed)
    new = sorted(a - b)
    moved = [k for k in b & a if before.index(k) != after.index(k)]
    print(f"events before={len(before)} after={len(after)}")
    print(f"missing={len(missing)} new={len(new)} moved={len(moved)}")
    for k in (missing + moved)[:20]:
        print("  !", k)
    return 1 if missing or moved else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Three outcomes carry different weight.

Bucket	Meaning	Policy
missing	a call disappeared	block, unless a reason is written in allow.txt
new	a call appeared	review; expected for a declared helper
moved	same key, new position	block by default

Missing calls block the merge. New calls on a pure extraction are a smell worth reading. Moved calls mean you changed sequencing, which is a behaviour change wearing a cleanup costume.

5. Shrink the Change Until the Diff Is Boring

Use a ladder, not a leap.

Step	Change	Expected diff
1	rename a local or parameter	zero
2	move one function, same module level	zero, after MODULE_MOVES
3	extract one helper, called from the same site	new only, declared
4	reorder calls or split a loop	moved, needs a written reason

Stop at the first step that produces a moved bucket you cannot justify in one sentence. That is your smallest safe change. The rest becomes a second commit with its own trace diff.

This is where the fixture pays for itself. Step 3 usually exposes one hidden coupling: a shared cache, a mutated default argument, a retry that fires twice. Output tests never showed it.

6. Where Free Model Access and a Free Server Fit

Drafting normalization rules is mechanical work. Filtering trace noise by hand costs an hour per repo. MonkeyCode's free model access and free server option are relevant there, and only there.

The free model proposes NOISE entries and MODULE_MOVES rows from a sample trace. A human accepts or rejects each line; nothing enters allow.txt unreviewed. The free server option runs the record-and-diff job in a stable environment, so both traces come from the same machine. Laptop-versus-CI comparisons fail for boring reasons.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The method stands on its own. Two small scripts and one committed JSON file are the whole mechanism.

Limitations

• settrace conflicts with anything that owns the tracer: coverage tools, some debuggers, some profilers. Disable them while recording.
• C extensions do not appear as Python frames. A parser written in Rust is invisible to this diff.
• Async and threaded code produces unstable ordering across await points. Keep the corpus single-threaded, or diff per task.
• Tracing slows call-heavy loops enough to break timeout-sensitive tests. Measure it before you wire it into CI.
• co_qualname needs Python 3.11+. On older versions the helper falls back to co_name, which loses class context.
• A clean trace diff proves structure held. It does not prove the refactor is correct.

Who Should Skip This

Skip it when the module fits on one screen. Skip it when the hot path is native code. Skip it when nobody will read the diff, because an unreviewed fixture is decoration with a commit message.

Use it when a repo has thin tests, real traffic, and one function everyone is afraid to move. Record once. Then move exactly one function.

If you have a candidate in mind, start with the smallest case in the corpus and a single operation. The first trace usually reveals a call you did not know existed.

Comments

[Launch Gate]

The redaction list deserves more attention than it gets - trace fixtures leak secrets into the repo faster than any other fixture kind, because nobody reviews a golden file for credentials. The limitation worth naming: this pins a total order, which holds only while the code under trace is single-threaded. The moment an extraction touches async or a worker pool, the trace becomes schedule-dependent and the diff fails on interleavings that are not regressions. The concurrent version of the same idea pins edges instead of sequences: record (caller, callee, correlation id) and assert the happens-before relationships survive - validation still precedes charge, lookup still precedes transform - while independent calls interleave freely. Same fixture philosophy, far less brittle. The other discipline that keeps these fixtures alive is budgeting the diff: a trace fixture that fails on every legitimate change trains everyone to regenerate it blindly, and a blindly regenerated fixture proves nothing. Every diff line should have to earn a name in the commit message, or the regeneration gets reverted.