Your Golden Master Is Flaky Because You Captured It Once

A golden master captured from a single run is a coin flip. Run the same command ten times and hash each output. If you get two hashes, your refactor gate is already broken.

Most characterization failures I review are not refactor damage. They are a harness comparing a frozen snapshot against a process that never produced identical bytes twice. Measuring that costs one script, and it runs before any assertion gets written.

Step 1: Measure nondeterminism instead of guessing

Write a probe. It runs the target command N times and counts distinct output signatures.

# tools/probe_nondeterminism.py
from __future__ import annotations

import argparse
import hashlib
import subprocess
from collections import Counter


def signature(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]


def run_once(cmd: list[str], timeout: int) -> tuple[int, str]:
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
    return proc.returncode, proc.stdout


def probe(cmd: list[str], runs: int, timeout: int) -> Counter:
    seen: Counter = Counter()
    for _ in range(runs):
        code, out = run_once(cmd, timeout)
        seen[(code, signature(out))] += 1
    return seen


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", type=int, default=10)
    ap.add_argument("--timeout", type=int, default=60)
    ap.add_argument("cmd", nargs=argparse.REMAINDER)
    args = ap.parse_args()

    seen = probe(args.cmd, args.runs, args.timeout)
    for (code, sig), n in seen.most_common():
        print(f"{n:>3}/{args.runs}  exit={code}  sha={sig}")
    return 0 if len(seen) == 1 else 1


if __name__ == "__main__":
    raise SystemExit(main())

Point it at the real entry point, not an imported helper.

python tools/probe_nondeterminism.py --runs 10 \
  python -m mycli report --account 42

Exit 0 means one distinct outcome across ten runs. Exit 1 means the output varies, and any golden file you commit today will flap tomorrow.

Step 2: Normalize volatile fields with named rules

Do not paper over this with a tolerance. Use substitutions a reviewer can read.

# tools/normalize.py
import re

RULES = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
                r"[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"),
    (re.compile(r"\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}"
                r"(\.\d+)?(Z|[+-]\d{2}:\d{2})?\b"), "<TS>"),
    (re.compile(r"(?:/tmp/|/private/var/folders/)\S+"), "<TMP>"),
    (re.compile(r"0x[0-9a-f]{6,}"), "<ADDR>"),
]


def normalize(text: str) -> str:
    for pattern, repl in RULES:
        text = pattern.sub(repl, text)
    return text

Call normalize inside the probe before hashing, then rerun with the same run count. Reaching one signature only after normalization is the correct result, not a defeat. Sort serialized structures too: json.dumps(payload, sort_keys=True, indent=2) removes ordering noise from set iteration and dict merges.

Leak	Replacement	Typical source
ISO-8601 timestamp	<TS>	log lines, report headers
UUID	<UUID>	generated run or request ids
temp path	<TMP>	tmp_path, mkdtemp, CI workspaces
0x... address	<ADDR>	default repr of objects
mapping order	sort_keys=True	set iteration, dict merge

Step 3: Freeze the golden file only after the probe exits 0

Now capture once, and commit the artifact to the repository.

python - <<'PY' > tests/characterization/report_account_42.txt
from tools.normalize import normalize
from tools.probe_nondeterminism import run_once
_, out = run_once(["python", "-m", "mycli", "report", "--account", "42"], 60)
print(normalize(out))
PY
git add tests/characterization/report_account_42.txt

The comparison happens after normalization, never before. The helper names below are illustrative; the ordering of normalize and === is the point.

def test_report_matches_golden():
    out = normalize(run_cli(["report", "--account", "42"]))
    golden = (GOLDEN / "report_account_42.txt").read_text()
    assert out == golden

Step 4: Convert fan-in into a change budget

"Smallest safe change" is a judgment call until a number is attached. Count the modules that import the target.

grep -rE "^\s*(from|import)\s+.*\breports\.build\b" --include="*.py" . | wc -l

Map that count to a hard limit on lines touched per commit.

Fan-in (importing modules)	Max changed lines	Gate before commit
1-3	20	golden on 2 probe runs
4-10	10	golden on 3 probe runs
11-25	5	probe exit 0 plus reviewer sign-off
26+	0	add an adapter seam first

Row four is the one teams skip. At that fan-in, introduce the seam with no behavior change and refactor in a separate commit. Enforce the budget mechanically so it does not depend on intent.

#!/usr/bin/env bash
# tools/refactor_step.sh
set -euo pipefail
MAX_CHANGED_LINES=10

python tools/probe_nondeterminism.py --runs 3 -- python -m mycli report --account 42
python -m pytest tests/characterization -q

CHANGED=$(git diff --numstat | awk '{s += $1 + $2} END {print s + 0}')
if [ "$CHANGED" -gt "$MAX_CHANGED_LINES" ]; then
  echo "budget exceeded: $CHANGED > $MAX_CHANGED_LINES" >&2
  exit 1
fi
echo "ok: $CHANGED lines changed"

Step 5: One transformation, then revert on any diff

Run the script. Read the failure output. Revert and split the change.

bash tools/refactor_step.sh || git checkout -- .

Repeat until the transformation fits the budget and the golden file still matches. A rename plus an inline plus a reorder in one commit is three guesses stacked. When a transformation cannot fit the row for its fan-in, that is the signal to add the seam instead.

Where the free tier fits this loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe is sequential, wall-clock heavy, and CPU light. MonkeyCode's free server option is a reasonable place to run it, so a ten-run probe does not occupy your laptop for the whole window. MonkeyCode's free model access is where I first asked for a candidate list of normalization patterns; every rule in the table above still had to be confirmed against real output from the target command.

Two limits matter here. Do not run the probe on a host whose OS paths or clock you cannot characterize, because the host itself becomes the flake. And do not treat generated normalization rules as correct until a second probe run proves them.

Limitations and who should skip this

Never normalize the field you plan to change. If the timestamp is the behavior under refactor, freezing it proves nothing.

Floating-point reductions shift with summation order. Fix the order, or compare with a documented tolerance. Silent tolerance plus aggressive normalization hides real regressions.

Concurrency probes only cover interleavings they happen to hit. Ten runs is a floor, not proof of safety.

Skip this workflow if the repository has no runnable entry point yet. Build that first. Skip it if you already run property-based tests over the same surface, because you have a stronger gate. And skip it if the output is deliberately random, unless the RNG is already seeded.

Start with the probe

Refactor nothing until the probe exits 0. Then change fewer lines than you wanted. The golden file stays green, and the budget script can show exactly why.