Twenty Identical Runs, Then One Small Diff: Sealing Entropy Before a Messy Refactor

Your golden files rot because the code is not the only input. Clocks, random seeds, UUIDs, and hash order leak into output. Seal those four sources first. Then prove the suite is stable. Only after that do you record a baseline.

This post is step zero of a messy-repo refactor. It is not about the refactor itself. It is about earning the right to trust a baseline at all.

Why the baseline lies

A characterization test compares bytes. If the bytes change for reasons unrelated to your edit, the test is noise. Noisy tests teach you to re-record baselines. Re-recording is how regressions ship.

Most messy repos have four entropy sources in common:

1. Wall-clock reads (datetime.now, time.time, utcnow).
2. Unseeded RNG (random, NumPy generators, faker libraries).
3. uuid4, which reads os.urandom and cannot be seeded.
4. Set and dict iteration order under a random PYTHONHASHSEED.

Find them before you design anything. A grep is enough to start.

rg -n --glob '!tests/**' \
  'datetime\.now|utcnow|time\.time|random\.|uuid4|secrets\.' \
  src/ | tee entropy-inventory.txt
wc -l entropy-inventory.txt

Treat that count as your blast radius. Ten hits is a fixture problem. Two hundred hits is an injection problem, and you should budget for it.

1. Seal the four sources in one fixture

The harness below is illustrative. I have not run it against your repo. Read it as a template, then adapt the patch points to your call sites.

# tests/characterization/conftest.py
import datetime as dt
import random
import time
import uuid

import pytest

FROZEN = dt.datetime(2026, 1, 1, 12, 0, 0, tzinfo=dt.timezone.utc)


@pytest.fixture(autouse=True)
def seal_entropy(monkeypatch):
    # 1. Clock: freeze both naive and aware call sites.
    class FrozenDateTime(dt.datetime):
        @classmethod
        def now(cls, tz=None):
            return FROZEN if tz else FROZEN.replace(tzinfo=None)

        @classmethod
        def utcnow(cls):
            return FROZEN.replace(tzinfo=None)

    monkeypatch.setattr(dt, "datetime", FrozenDateTime)
    monkeypatch.setattr(time, "time", lambda: FROZEN.timestamp())

    # 2. RNG: reseed per test, not per session.
    random.seed(20260101)

    # 3. uuid4: deterministic counter, same shape as a real UUID.
    counter = iter(range(10_000))
    monkeypatch.setattr(uuid, "uuid4", lambda: uuid.UUID(int=next(counter)))

    yield

Two details matter more than the code. First, reseed per test. A session-level seed makes test order part of your baseline. Second, set PYTHONHASHSEED=0 in the runner, not in the fixture. You cannot change it after interpreter start.

2. Write canonical snapshots, not stdout

Do not hash pytest output. It contains durations and ordering that vary legitimately. Hash the artifact your test produces.

def emit(tmp_path, name, payload):
    """Write one canonical JSON file per characterization case."""
    import json
    path = tmp_path / f"{name}.json"
    path.write_text(json.dumps(payload, sort_keys=True, indent=2) + "\n")
    return path

Sorted keys, one trailing newline, no timestamps in the file itself. If a snapshot needs a time, read it from the frozen clock so it stays constant.

3. Prove stability with twenty runs

One passing run proves nothing. Run the characterization directory twenty times. Hash the concatenated snapshots each time. You must see exactly one unique digest.

for i in $(seq 1 20); do
  rm -rf /tmp/snap && mkdir -p /tmp/snap
  PYTHONHASHSEED=0 pytest -q tests/characterization \
    --snap-dir=/tmp/snap >/dev/null 2>&1
  find /tmp/snap -name '*.json' -print0 \
    | sort -z | xargs -0 cat | sha256sum
done | sort -u | tee digests.txt
wc -l < digests.txt   # must print 1

If it prints more than one, you have not sealed everything. Bisect by running one test file at a time. Most failures at this stage are unseeded RNG inside a third-party library, not your code.

This loop costs twenty times your suite runtime. Scope it to the characterization directory, not the whole test suite.

4. Record the baseline only now

After twenty identical digests, copy the snapshots into the repo. Commit them in a one-line diff that touches nothing else. That commit is your reference point for the rest of the refactor.

Rules for the baseline commit:

1. No source changes in the same commit.
2. Record the exact command and seed in the commit message.
3. Note which checks were skipped, and why.

A stable baseline can still be wrong. Determinism does not imply correctness. It only means the next diff is attributable.

5. Make the smallest safe change

Now the refactor. One seam per commit. The characterization digest must be byte-identical before and after. If it is not, the change was not behavior-preserving, or your baseline leaked.

Candidate change	Lines touched	Gate	Verdict
Add a clock or RNG parameter to one function	~10	digest identical	Do it first
Move one call site behind an interface	~30	digest identical	Do it second
Replace a dict with a dataclass across a module	~150	digest identical	Split into three commits
Rename a public function used by 40 call sites	~90	digest identical + import check	Only if callers are in-repo
Reorder side-effect calls inside one function	any	digest identical + call-log diff	Needs a second gate

Use a diff budget. If a commit exceeds roughly 40 changed source lines, stop and split it. Large commits hide the line that broke the digest.

6. Where a free model and free server fit

Candidate minimal diffs are cheap to generate and expensive to trust. I use MonkeyCode's free model access to draft two or three candidate seams for a single function. The determinism loop then decides, not the model. The free server option is where I run the twenty-run loop, so the repeat cost does not land on my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The split is deliberate. Models propose. The digest gate disposes. Anything that cannot pass twenty identical runs is not a candidate, however clean the suggestion reads.

Limitations and who should skip this

• Sealing hides real time dependence. Retention windows and expiry logic need a separate, unfrozen integration suite.
• Patching datetime module-wide breaks libraries that did from datetime import datetime. Prefer explicit injection where you can add it.
• Twenty runs catches flakiness, not a wrong expectation. Review the snapshots by hand once.
• The loop multiplies runtime. On a slow suite, cap it at ten runs and say so in the commit message.
• Skip this if behavior is legitimately nondeterministic and untested. Seal what you can, and quarantine the rest.

If your suite cannot survive twenty identical runs, the refactor will not survive review. Fix the baseline first. Then cut one small diff.