Twenty Identical Runs or No Refactor: A Determinism Gate for Characterization Tests

A characterization test that fails on the unchanged tree is worthless. You cannot separate a regression from noise.

So start every refactor with a determinism gate. Run the characterization suite many times on untouched code. If the output fingerprints differ, stop. Remove the noise before you move a single line.

This post gives a small harness for that gate, a triage table, and the smallest-safe-change ladder that comes after.

1. The problem is usually not your refactor

Most "the refactor broke the tests" reports are false. The suite was already unstable on main.

Common sources include timing strings in logs, temp directory paths, dict or set ordering, thread scheduling, locale, random seeds, and live network calls. Two runs produce two different outputs with zero code changes.

You need proof of stability, not a feeling. Twenty repeat runs is cheap proof.

2. The harness

Save this as determinism_gate.py. It has no third-party dependencies.

#!/usr/bin/env python3
"""Repeat-run fingerprint probe for a characterization suite."""
import hashlib
import json
import os
import subprocess
import sys
import time

RUNS = int(os.environ.get("GATE_RUNS", "20"))
CMD = os.environ.get("GATE_CMD", "pytest -q tests/characterization")
OUT = os.environ.get("GATE_OUT", "determinism_baseline.json")
TIMEOUT = int(os.environ.get("GATE_TIMEOUT", "300"))


def fingerprint():
    start = time.monotonic()
    proc = subprocess.run(CMD, shell=True, capture_output=True, timeout=TIMEOUT)
    blob = proc.stdout + b"\x00" + proc.stderr
    return {
        "exit": proc.returncode,
        "digest": hashlib.sha256(blob).hexdigest()[:16],
        "seconds": round(time.monotonic() - start, 2),
    }


def main():
    groups = {}
    for i in range(RUNS):
        fp = fingerprint()
        key = f'{fp["exit"]}:{fp["digest"]}'
        groups.setdefault(key, []).append(i)
        print(
            f'run {i + 1}/{RUNS} exit={fp["exit"]} '
            f'digest={fp["digest"]} {fp["seconds"]}s',
            flush=True,
        )

    report = {
        "cmd": CMD,
        "runs": RUNS,
        "distinct_fingerprints": len(groups),
        "stable": len(groups) == 1,
        "groups": {k: len(v) for k, v in sorted(groups.items())},
    }
    with open(OUT, "w", encoding="utf-8") as fh:
        json.dump(report, fh, indent=2, sort_keys=True)
    print(json.dumps(report, indent=2, sort_keys=True))
    return 0 if report["stable"] else 1


if __name__ == "__main__":
    sys.exit(main())

The fingerprint covers exit code plus stdout plus stderr. It ignores wall time, because slow runs are not failures.

Steps to run it

1. Pick one command. It must finish without a human typing anything.
2. Normalize the output. Add --tb=no -q to pytest so tracebacks stop varying.
3. Pin ambient inputs. Export PYTHONHASHSEED=0 and pin the worker count.
4. Run the gate on the untouched tree, before creating the refactor branch.

export GATE_CMD='pytest -q tests/characterization --tb=no -p no:randomly'
export GATE_RUNS=20
export GATE_TIMEOUT=300
export PYTHONHASHSEED=0
python3 determinism_gate.py

Exit code 0 means one distinct fingerprint across all runs. Exit code 1 means you have noise to remove first.

Commit determinism_baseline.json at the refactor branch point. It is your reference artifact.

3. Triage the unstable report

Read the groups map. The shape of the instability tells you the cause.

Symptom in report	Likely cause	First fix to try
Two or more digests, all exit 0	Volatile text: timings, ids, tmp paths	Filter output, freeze clock, uuids, tmpdir
Digests split across exit 0 and exit 1	Order dependency or real flake	Pin -p no:randomly, fix test isolation
Two digest groups, roughly even split	Worker or hash-seed ordering	Pin workers, set PYTHONHASHSEED=0
One digest, wildly varying seconds	Slow but stable	Keep it, raise GATE_TIMEOUT
Digest changes only on the first run	Cache or leftover state on disk	Wipe build dir, disable cache plugin

Do not "fix" the suite by widening assertions until everything passes. That deletes the signal you need later.

4. Choose the repeat count on purpose

Repeat runs cost CI minutes. Pick the count from risk, not habit.

Runs	Detects	Reasonable use
5	Frequent flakes	Local inner loop, suite under 30s
20	Intermittent ordering bugs	Default before a shared-branch refactor
50	Rare races	Concurrency or network-adjacent code

Those counts are a starting policy, not measured benchmarks.

5. Then make the smallest safe change

With a stable baseline, climb one rung at a time. Re-run the same gate after each rung.

Rung	Change	Files touched	Expected test diff
0	Rename a symbol only	1	Names in output only
1	Extract a function, same module	1	None
2	Move a function, keep an import shim	2	None
3	Change an interface or invert a dependency	3+	New tests required

Stop climbing when the diff stops matching the rung. A rename should never force a rewritten assertion.

Useful heuristic: if rung 0 or rung 1 fails the gate, the refactor is probably fine and the harness is wrong. Go back to section 3.

6. Where MonkeyCode fits in this workflow

Two tasks here are mechanical and safe to delegate. One is drafting the normalization filter list from a sample of unstable output. The other is babysitting the repeat-run loop while you keep editing.

The operator of this account supplied two availability claims about MonkeyCode: free model access, and a free server option. I have not benchmarked either, and availability may change, so verify current terms before you depend on them. Treat them as a way to keep the repeat loop off your laptop, not as a performance guarantee.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Either way, keep the gate itself in your repo. A harness you cannot run offline is not a gate.

7. Limitations and who should skip this

• If you cannot run the suite with one non-interactive command, fix that first.
• If the suite takes twenty minutes, use 5 runs or a trimmed characterization subset.
• If your failures are genuinely deterministic, skip the gate and go straight to rung 0.
• If the team will not commit the baseline JSON, the gate has no reference point.
• This detects instability. It does not prove correctness.

Do not treat the gate as a replacement for a real regression suite. It is a precondition for trusting one.

8. Pre-PR checklist

1. Gate exits 0 on the untouched tree, 20 runs.
2. determinism_baseline.json committed at the branch point.
3. Every change stayed inside a single rung.
4. Gate re-run after each rung, with the identical command.
5. Diff reviewed for rung violations before merge.

Twenty identical runs buy you a claim you can defend. Anything less is guesswork with extra steps.