Freeze the Clock First: A Nondeterminism Gate for Messy Refactors

The short version

Never pin a snapshot your recorder cannot reproduce. Run the recorder ten times before you touch any code. If raw output differs between runs, that difference is your first bug — not the refactor you planned.

Characterization tests fail for a boring reason. The recorder captures the clock, the PID, the temp path, or a duration. Two runs, two hashes, and the team quietly deletes the assertion. Then the extraction ships blind.

The gate below is small. It answers one question: is the noise in my snapshot explained by a declared rule, or is it unexplained? Only run the refactor when the answer is zero.

Step 1 — Run K times and hash the raw output

Start with raw bytes. No normalization, no cleverness. You want the true volatility count first.

#!/usr/bin/env bash
set -euo pipefail
K="${K:-10}"
mkdir -p runs
for i in $(seq 1 "$K"); do
  python -m recorder.cli --out "runs/raw-$i.txt" >/dev/null 2>&1
done

# How many distinct outputs did K runs produce?
sha256sum runs/raw-*.txt | awk '{print $1}' | sort -u | wc -l

If that number is 1, stop reading and go refactor. Your recorder is deterministic enough.

If it is greater than 1, do not normalize yet. Blind normalization hides real behavior changes along with the noise. Classify first, then decide.

Step 2 — Declare your volatile fields as a rule table

Write the rule table down in a file, not in your head. Each rule is a name, a regex, and a replacement token.

# volatility.py
import hashlib
import re

# Order matters: most specific patterns first.
RULES = [
    ("uuid", re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.I), "<UUID>"),
    ("ts",   re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?"), "<TS>"),
    ("tmp",  re.compile(r"/(?:tmp|var/folders|private/var)\S*"), "<TMP>"),
    ("home", re.compile(r"/(?:home|Users)/[^/\s]+"), "<HOME>"),
    ("addr", re.compile(r"0x[0-9a-f]{6,}"), "<ADDR>"),
    ("pid",  re.compile(r"\bpid[=: ]?\d+\b", re.I), "pid=<PID>"),
    ("dur",  re.compile(r"\b\d+(?:\.\d+)?(?:ms|us|µs|s)\b"), "<DUR>"),
    ("port", re.compile(r":\d{4,5}\b"), ":<PORT>"),
]


def classify(chunk: str) -> str:
    """Name the rule that explains this diff chunk, or UNEXPLAINED."""
    for name, rx, _ in RULES:
        if rx.search(chunk):
            return name
    return "UNEXPLAINED"


def normalize(text: str) -> str:
    for _, rx, repl in RULES:
        text = rx.sub(repl, text)
    return text


def digest(text: str) -> str:
    return hashlib.sha256(normalize(text).encode()).hexdigest()[:12]

Two pitfalls bite here. First, rule order: an address regex can eat part of a UUID. Second, the duration rule will rewrite real numeric output if your records contain bare numbers with units. Test the table against one known-good run before trusting it.

Step 3 — Gate on unexplained volatility, not total volatility

Now measure. Diff every run against run 1. Keep only chunks no rule explains.

# census.py
import difflib
import glob
import json

from volatility import classify


def unexplained(a: str, b: str) -> list[str]:
    out = []
    diff = difflib.unified_diff(a.splitlines(), b.splitlines(), lineterm="", n=0)
    for line in diff:
        if line.startswith(("+++", "---")):
            continue
        if line[:1] not in "+-":
            continue
        body = line[1:].strip()
        if body and classify(body) == "UNEXPLAINED":
            out.append(body)
    return out


runs = sorted(glob.glob("runs/raw-*.txt"))
base = open(runs[0]).read()
report = {r: unexplained(base, open(r).read()) for r in runs[1:]}
print(json.dumps(report, indent=2))
print("unexplained chunks:", sum(len(v) for v in report.values()))

Set the policy at zero unexplained chunks. Not "few". Zero.

Every unexplained chunk is either a missing rule or a genuine nondeterminism source in the code under test. Both must be resolved before the refactor starts. A map iteration order difference is a real finding, not noise.

Step 4 — Normalize and pin the digest

Once the census reports zero, hash the normalized text and store it.

python - <<'PY'
from volatility import digest
print(digest(open("runs/raw-1.txt").read()))
PY

Commit that digest next to the rule table. The rule table is now versioned evidence, not a personal habit. When someone adds a rule six months later, the digest changes and the diff is reviewable.

Step 5 — Run the sensitivity check

The gate is worthless if it cannot fail. Mutate the code on purpose and confirm detection.

Deliberate mutation	Signal you should see	If you see nothing
Swap two writes	Order change in output	Rule table is too broad
Drop one write	Missing line	Recorder misses the channel
Change an exit code	Non-zero code chunk	Harness ignores status
Add a debug timestamp	Normalized <TS>	Correct, no alarm
Reorder dict keys	Key-order chunk	Unexplained volatility remains

Record two numbers, not one. The false-alarm rate is how often a stable tree fails. The blind-spot count is how many mutations survive. You want zero on both; anything else is a known, written limitation.

Step 6 — Run the K-loop somewhere disposable

Local runs carry machine-specific volatility. Your shell history, your user config, your timezone, your locale. A clean environment removes several noise sources at once.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode exposes free model access and a free server option, per the operator's description. Two honest uses fit this workflow. First, run the ten-run loop on the free server so the environment is fresh and uniform rather than your laptop. Second, use free model access to draft candidate regex rules for the unexplained chunks in the census report.

Treat model-proposed rules as drafts only. A model will happily suggest \d+ and erase half your signal. Every accepted rule must survive the Step 5 mutation table. The model proposes; the sensitivity check decides.

Availability, limits, and quotas for free tiers change. Verify them on the product's own current documentation before you depend on them.

Limitations and who should skip this

Normalization hides changes by design. That is its job and its risk. Line-level classification can mask a real change that shares a line with a timestamp, so split high-risk lines into tokens before classifying.

Ten runs is a sample, not a proof. Rare races survive it. Timing-heavy code, such as a scheduler or a retry loop, resists snapshot pinning entirely — the pinning cost exceeds the value.

Skip the normalization stages if your recorder is already stable. Go straight to the digest gate. Skip the whole approach if the observed output is genuinely time-valued, like an audit log, where the timestamp is the product. There, normalize the format and assert ordering instead.

Order matters more than tooling

Freeze volatility, then measure the digest, then cut the code. Reversing that order gives you a refactor you cannot verify and noise you will learn to ignore. Ten runs, one rule table, one digest, one mutation check. That is the whole gate.