Replay Captured Inputs, Then Move One Function: A Flake-Trial Gate for Messy Refactors

Characterization tests usually fail for one reason. You invent the inputs.

A messy repo does not care about your invented inputs. It reads a config file at import time, sorts by dict insertion order, and calls now() inside a loop. You write twelve tests against guessed arguments. They pass. You move one function. They still pass. Production breaks anyway, because production never sent your guessed arguments.

This post describes a different order of operations. Capture real inputs first. Prove the outputs are stable across repeated runs. Only then make the smallest safe change: one function, one new module, one alias left behind.

The artifact is a cassette file plus a flake trial. Both are small enough to run locally in a few minutes.

Snippets are illustrative Python and bash. Run them against your own repo before trusting the gate.

Step 1: Record inputs at the seam, not outputs

Pick the one function you plan to move. Wrap it with a recorder. Write every real invocation to JSONL.

import functools, json, os, pathlib

CASSETTE = pathlib.Path(os.environ.get("CASSETTE_PATH", "cassette.jsonl"))

def record(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        if os.environ.get("CASSETTE_RECORD") == "1":
            row = {"fn": fn.__name__, "args": args, "kwargs": kwargs}
            with CASSETTE.open("a") as fh:
                fh.write(json.dumps(row, default=str) + "\n")
        return fn(*args, **kwargs)
    return wrapper

Run your existing test suite, CLI smoke path, or one staging replay with CASSETTE_RECORD=1. Stop at fifty to two hundred rows. That is enough coverage for a single-function move.

Two constraints matter here.

1. Arguments must survive JSON. If they do not, capture a projection, not a repr blob.
2. You are recording inputs only. Outputs belong to the goldens you will freeze later.

Sanitize before committing. Strip tokens, emails, and customer identifiers from the cassette. A cassette is a fixture, and fixtures get pushed.

Step 2: Replay the cassette as parametrized cases

Convert each row into one test case. Compare against a golden.json keyed by row index.

import json, pytest
from app.parsing import parse_invoice

ROWS = [json.loads(l) for l in open("cassette.jsonl")]
GOLD = json.load(open("golden.json"))

@pytest.mark.parametrize("i", range(len(ROWS)))
def test_replay(i):
    row = ROWS[i]
    got = parse_invoice(*row["args"], **row["kwargs"])
    assert json.dumps(got, sort_keys=True, default=str) == GOLD[str(i)]

golden.json starts empty. Generate it on the first clean run. Do not hand-edit it.

This is the point where most teams freeze a bad golden. A single green run does not prove determinism.

Step 3: Run the flake trial before freezing anything

Run the same cassette five times. Collect the summary line from each run.

rm -f trials.txt
for i in 1 2 3 4 5; do
  python -m pytest tests/test_replay.py -q --tb=no | tail -1 >> trials.txt
done
sort -u trials.txt | wc -l

One unique line means the suite is stable enough to trust. More than one means you found nondeterminism before your refactor found it in production.

Use the trial result as a decision, not a mood.

Trial result	Likely cause	Action
5/5 identical	deterministic enough	freeze goldens, allow the move
3/5 or 4/5	set or dict ordering	pin PYTHONHASHSEED, sort before serialize
2/5 or worse	clock, timezone, locale, network	inject the dependency, do not freeze
never identical	unbounded external state	stop, add an adapter first

Rows three and four are the important ones. Freezing a golden over a live clock buys you a test that fails next month for no code reason.

Step 4: Freeze nondeterminism only where you own it

Pin the cheap sources globally. Inject the expensive ones.

import random, pytest

FIXED = "2026-01-01T00:00:00Z"

@pytest.fixture(autouse=True)
def freeze(monkeypatch):
    monkeypatch.setenv("TZ", "UTC")
    monkeypatch.setenv("LC_ALL", "C")
    monkeypatch.setattr("app.clock.now_iso", lambda: FIXED)
    random.seed(0)

Timezone, locale, hash seed, and RNG seed are cheap. Patch them and move on.

Network calls, database reads, and filesystem scans are not cheap. If the trial shows flakiness from those, the smallest safe change is not a move. It is an adapter at the seam, and the refactor waits.

Step 5: Move one function and keep the old name

Now make the smallest change that counts as progress.

mkdir -p app/parsing_v2
git mv app/parsing/invoice.py app/parsing_v2/invoice.py

Leave a re-export behind so callers do not change in the same commit.

# app/parsing/__init__.py
from app.parsing_v2.invoice import parse_invoice  # noqa: F401

One function moved. Zero call sites edited. The cassette does not know the difference, which is exactly the point.

Step 6: Gate the commit on a byte-stable verdict file

Capture the replay summary before and after the move. Diff them.

python -m pytest tests/test_replay.py -q > before.txt
git stash && python -m pytest tests/test_replay.py -q > after.txt && git stash pop
diff <(sort before.txt) <(sort after.txt) && echo SAFE-TO-REVIEW

A clean diff is a necessary condition, not a sufficient one. Still read the moved file for import-time side effects, which no cassette will catch.

Debugging a flaky golden: three classes, three fixes

When the trial returns two unique lines, classify before you patch.

1. Order drift. Output is a list or set whose order changes. Fix: sort at serialization, not at assertion time.
2. Clock and locale drift. Output contains a timestamp, currency, or %c format. Fix: pin in the fixture, or take the clock as a parameter.
3. Environment drift. Output depends on os.environ, cwd, or an on-disk cache. Fix: inject a path or mapping; do not patch globals per test.

Classify wrong and you get a suite that is green locally and red in CI. That is worse than no suite, because people start ignoring it.

Where a free model and a free server fit in

Two parts of this loop are tedious and parallelizable.

The model part is drafting. Given a cassette row and the real output, a free model can propose assertion labels, docstrings, and edge-case groupings. That is useful drafting and poor authority. A human still approves every golden. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode exposes free model access and a free server option; check current terms yourself rather than trusting a blog post for quotas.

The server part is the trial loop. Five sequential pytest runs on a repository with slow imports is a laptop heater. The free server option is a reasonable place to park the loop while you keep editing locally. Nothing exotic is required: a checkout, a Python runtime, and the same command you ran above.

One mention is enough. Verify plan limits and runtime images before you depend on either capability.

Limitations and who should not use this

Skip this workflow when the typed, fast test suite already covers the function. Skip it for a module with three callers you can read in ten minutes. The cassette is overhead there.

Do not start with this if the function touches customer data and you have no sanitizer. Recording real inputs is recording real inputs.

Do not use it as a substitute for design review. A green cassette proves behavior held across recorded inputs. It never proves the behavior was correct.

The order that matters

Capture inputs. Prove stability. Move one function. Gate on a stable verdict file.

The flake trial is the step people skip. It is also the cheapest place to discover that your "deterministic" parser depends on a timezone.