Run Your Characterization Recorder Three Times Before You Trust It

The recorder is the first thing to test

A characterization test proves nothing until the harness itself is stable. Capture the legacy behavior three times. Diff the passes byte for byte. Every byte that moves is an input you have not pinned.

Teams skip this step constantly. They capture one golden file and commit it. The next refactor produces a diff full of timestamps. Nobody can separate intent from noise, so the suite gets ignored.

The failure mode: a golden file that encodes the clock

A common shape looks like this. A CLI prints generated_at, a run UUID, and a set of plugin names. Set iteration order depends on PYTHONHASHSEED. The JSON writer uses the machine locale for floats.

The first capture pins all four. The next run on a different machine fails. That failure is not a behavior change. It is an environment fingerprint wearing a test's clothes.

Step 1 — Pick one seam and two observables

Choose exactly one entry point for the first pass. Record at most two observables. Exit code plus stdout is usually enough. Add written files only when stdout is thin.

Do not record everything. A wide recorder produces wide noise. You can widen it after the harness is deterministic.

Step 2 — One file per case, per pass

# record.py
import json, pathlib, subprocess, sys

GOLDEN = pathlib.Path("golden")

def capture(argv: list[str]) -> dict:
    proc = subprocess.run(argv, capture_output=True, text=True, timeout=60)
    return {"argv": argv, "code": proc.returncode,
            "stdout": proc.stdout, "stderr": proc.stderr}

def write(case_id: str, pass_no: int, payload: dict) -> None:
    out = GOLDEN / f"pass-{pass_no}"
    out.mkdir(parents=True, exist_ok=True)
    (out / f"{case_id}.json").write_text(
        json.dumps(payload, indent=2, sort_keys=True) + "\n")

if __name__ == "__main__":
    n = int(sys.argv[1])
    cases = {
        "empty": ["./legacy", "--input", "cases/empty.txt"],
        "one_row": ["./legacy", "--input", "cases/one.txt"],
    }
    for cid, args in cases.items():
        write(cid, n, capture(args))

sort_keys=True removes one pointless source of diff churn. It does not fix the payload.

Step 3 — Run three passes, not two

for i in 1 2 3; do python record.py "$i"; done
diff -rq golden/pass-1 golden/pass-2
diff -rq golden/pass-2 golden/pass-3

Two passes catch deterministic noise. Three passes catch intermittent noise. A value that differs in one pass out of three is a race or a network read. Treat it as a gap in your understanding, not a flake to retry.

Step 4 — Classify each differing byte

# volatility.py
import difflib, pathlib, sys

a, b = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
for old in sorted(a.glob("*.json")):
    new = b / old.name
    if not new.exists():
        print(f"MISSING {old.name}")
        continue
    if old.read_bytes() == new.read_bytes():
        continue
    print(f"DIFF {old.name}")
    for line in difflib.unified_diff(
            old.read_text().splitlines(),
            new.read_text().splitlines(), lineterm="", n=0):
        if line[:1] in "+-" and not line.startswith(("+++", "---")):
            print(f"  {line}")

Output looks like this:

DIFF one_row.json
  -  "generated_at": "2026-11-06T09:14:02Z",
  +  "generated_at": "2026-11-06T09:14:07Z",
  -  "plugins": ["beta", "alpha"],
  +  "plugins": ["alpha", "beta"],

Now you have names, not vibes. Each name goes into exactly one bucket.

Step 5 — Freeze, mask, or stop

Freeze when the value comes from process environment. Mask when the format is stable and the value is irrelevant to this seam. Stop when the value should be deterministic and is not.

# freeze.toml — export before every capture
[env]
PYTHONHASHSEED = "0"
TZ = "UTC"
LC_ALL = "C.UTF-8"
SOURCE_DATE_EPOCH = "1700000000"

Keep masks narrow. One pattern per token. A greedy regex hides the regression you built the harness to catch.

# normalize.py
import re

MASKS = [
    (re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z"), "<TS>"),
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}"
                r"-[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"),
]

def normalize(text: str) -> str:
    for pattern, token in MASKS:
        text = pattern.sub(token, text)
    return text

Differs in	Bucket	Action	Residual risk
timestamp, UUID, temp path	mask	one narrow regex per token	low
set or plugin ordering	freeze	pin hash seed, sort at boundary	medium
float formatting	freeze	pin locale and rounding	medium
a numeric value	stop	find hidden state or concurrency	high
one pass out of three	stop	treat as race or network read	high

Step 6 — Iterate until three passes match

Rerun the loop after every freeze or mask. Your exit condition is byte-identical output across three passes. Not "close enough". Not "only the timestamp".

Commit freeze.toml, normalize.py, and the goldens together. A teammate must reproduce the capture from a clean checkout without asking you questions.

Step 7 — Make the smallest safe change

Only now touch production code. One seam per commit. Replay the same cases against the new path and diff against frozen.

A clean refactor shows the intended diff and nothing else. If a timestamp reappears, your harness regressed, not your refactor.

Where free model access and a free server fit

The three-pass loop is repetitive and cheap. Capturing, diffing, and bucketing is mostly mechanical work.

Two availability claims here are operator-supplied. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A reasonable split: have a model propose mask regexes and freeze entries from the raw diff. You review every pattern and reject anything wider than the token it targets. Run the passes on the free server option so the loop does not occupy your laptop. Each pass is short and the case set is small.

That is a workflow note, not a performance claim. Validate the loop on your own cases before depending on it.

Limitations and who should not use this

• Output that is legitimately environment-dependent will never go byte-identical. Perf counters, nonce-bearing protocol traces, and wall-clock scheduling need a different harness.
• Masking is a small lie. Each mask widens the gap between goldens and reality. Too many masks and you are testing your normalizer.
• Interleaved multi-threaded logs rarely stabilize from seeding alone. Move to structured logs with per-thread fields first.
• If the legacy module already has honest assertions on its contract, a recorder adds little. Write the assertions instead.
• This does not replace property-based tests. It buys you one safe first move on code nobody can run.

Start with one seam, three passes, and a single diff. If you want a place to run that loop, the free server option is a reasonable place to begin.