Twenty Repeat Runs Before You Move One Function: A Determinism Gate for Messy Repos

Your characterization tests can lie. They pass once, then the refactor breaks production.

I trust no pinned output after a single run. A messy repo needs twenty runs before you move one function.

This article describes a determinism gate. It is not a guide to writing golden files, and it is not a benchmark.

1. One green run proves almost nothing

Messy repos hide several sources of drift. Dict and set ordering changes when the hash seed changes. Timestamps, timezones and locales reformat the same string differently.

Float formatting drifts after a library bump. Row order without ORDER BY drifts under load. Absolute paths and temp directories drift between machines.

Any of these turns a golden file into a coin flip. The refactor then takes the blame for a flake that already existed.

2. Step 1: Pin the smallest observable surface

Record one payload, one exit code, one stderr line. Keep the fixture small enough to read in review.

A 4 KB JSON file beats a 4 MB log. You will diff it twenty times.

Freeze only behavior you can describe. If you cannot name a field's meaning, do not pin it yet.

3. Step 2: Run the repeat matrix

The gate is twenty runs under different environments. Each run hashes the same output tree.

#!/usr/bin/env bash
# gate.sh - run the pinned suite N times, count distinct output hashes
set -uo pipefail

RUNS="${1:-20}"
ARTIFACTS="${2:-./artifacts}"

rm -rf "$ARTIFACTS"
mkdir -p "$ARTIFACTS"

for i in $(seq 1 "$RUNS"); do
  if [ $((i % 2)) -eq 0 ]; then TZ_VAL="UTC"; else TZ_VAL="Asia/Tokyo"; fi

  PYTHONHASHSEED="$i" \
  TZ="$TZ_VAL" \
  LC_ALL="C" \
  pytest -q tests/characterization > "$ARTIFACTS/run-$i.log" 2>&1
  echo "run=$i exit=$?" >> "$ARTIFACTS/exit-codes.txt"

  (cd out && find . -type f -print0 | sort -z | xargs -0 sha256sum) \
    > "$ARTIFACTS/run-$i.sha"
done

echo "--- exit codes ---"
sort "$ARTIFACTS/exit-codes.txt" | uniq -c

echo "--- distinct output hashes ---"
for f in "$ARTIFACTS"/run-*.sha; do sha256sum "$f"; done \
  | awk '{print $1}' | sort -u | wc -l

Read the last number, not the green checkmarks. One distinct hash means the output tree is stable.

More than one hash means drift exists before you touch anything. Fix that first.

The find . | sort part matters. Without it, file order changes the hash even when content does not.

4. Step 3: Scrub only fields you can name

Volatile fields must be normalized before hashing. Blanket scrubbing hides real drift.

"""normalize.py - normalize captured outputs before hashing."""
import re
import sys

PATTERNS = [
    (re.compile(r'"(timestamp|created_at|trace_id|etag|duration_ms)"\s*:\s*("[^"]*"|[-\d.eE+]+)'),
     r'"\1":"<volatile>"'),
    (re.compile(r'\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(\.\d+)?Z?\b'), '<ts>'),
    (re.compile(r'\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b'), '<uuid>'),
    (re.compile(r'\b\d+\.\d{5,}\b'), '<float>'),
]

def scrub(text: str) -> str:
    for pattern, replacement in PATTERNS:
        text = pattern.sub(replacement, text)
    return text

if __name__ == "__main__":
    sys.stdout.write(scrub(sys.stdin.read()))

Usage is one line in the matrix loop.

python normalize.py < out/response.json | sha256sum

Every pattern needs a written justification in the commit message. If a reviewer cannot explain why a field is volatile, it stays pinned.

5. Step 4: Perturb the environment, not the code

The matrix already varies hash seed, timezone and locale. That covers the most common ordering bugs.

Add two knobs when the code is concurrent or cached. Raise and lower the worker count. Clear the cache on odd runs.

If output changes under exactly one knob, you found the dependency. That dependency is the real refactor risk.

6. Step 5: Take the smallest possible change

Start refactoring only after the matrix returns one hash. Then change one thing.

Copy the body into a function. Pass every hidden input as an explicit argument. Commit nothing else in that change.

Re-run the matrix. New hashes must equal the pinned hashes. If they differ, revert and inspect before continuing.

7. Decision table

Matrix result	Meaning	Next action
20/20 pass, 1 hash	Behavior is pinned	Extract the function
20/20 pass, many hashes	Unnamed volatile field	Fix the normalizer, re-run
Mixed pass and fail	Nondeterminism in the code	Stop, fix the flake first
Stable on one machine only	Environment dependency	Pin the dependency, re-run

8. Where free model access and a free server fit

Model-assisted extraction is cheap now. Verification is the bottleneck, not the edit.

I use a model for two narrow jobs in this gate. First, list the hidden inputs of a messy function. Second, draft the perturbation matrix configuration.

MonkeyCode is an open-source project. The operator states a free allowance of 10 million tokens and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quotas, availability and terms change over time. Check the project's own page before planning around any number.

Keep the model output as a proposal. Run the matrix before trusting it. A suggested extraction boundary is a hypothesis, not evidence.

The free server option helped for one practical reason. A 20-run matrix is slow on a laptop, and laptop-only runs share one environment.

Running the matrix elsewhere tests whether the pins survive a different environment. That is the whole point of the gate.

If you want to reproduce this without provisioning anything, start with the free tier and the checklist below.

9. Limitations and who should not use this

Golden files pin current behavior, including bugs. They are not a specification.

Scrubbing a field that actually matters will hide real drift. Keep the scrub list short and reviewed.

Twenty runs are too slow for long end-to-end suites. Sample the fastest tests instead.

Repos with no deterministic seam should skip this. Fix test isolation before pinning outputs.

Do not use this gate to justify a large refactor. It only protects small changes.

If IDs are random by design, the gate cannot separate design from drift. Constrain the input first.

10. Reproduction checklist

1. Pick one small output tree.
2. Write one characterization test for it.
3. Run gate.sh with 20 runs.
4. Read the distinct-hash count.
5. Scrub only named volatile fields.
6. Re-run until the count is 1.
7. Make the smallest possible change.
8. Re-run and compare hashes.
9. Revert if the hashes differ.
10. Commit the pins together with the change.

A refactor is safe when the matrix says so. Not when the suite is green once.