Run It Twenty Times Before You Pin It: Characterization Tests for Messy Repos

A characterization test is only worth writing on output you can reproduce. Sample twenty runs before you pin one line.

Most "refactor safely" advice starts with "write a characterization test." That advice hides an assumption. It assumes the function returns the same value twice. Messy repos break that assumption daily.

Clocks, UUIDs, dict iteration, locale formatting, retry jitter, and float printing all leak into output. Pin that noise and your test goes red for reasons unrelated to your change. A red test you do not trust is worse than no test at all.

Below is the loop I run: sample, classify, freeze, then change one thing. The gate is a rule I call 20/3/2. Twenty runs, three hash seeds, two timezones.

Step 1 — Sample the entry point before touching the code

Do not import the module under test into your own process. Set PYTHONHASHSEED in a child process instead. It only takes effect at interpreter start.

# sample.py — run the entry point N times, count distinct outputs
import collections, os, subprocess, sys

CMD = [sys.executable, "entry.py"]  # entry.py prints exactly one JSON line

def sample(n: int = 20):
    seen = collections.Counter()
    for seed in range(n):
        env = dict(os.environ)
        env["PYTHONHASHSEED"] = str(seed)
        env["TZ"] = "UTC" if seed % 2 == 0 else "Asia/Tokyo"
        out = subprocess.run(CMD, env=env, capture_output=True, text=True, check=True)
        seen[out.stdout.strip()] += 1
    return seen

if __name__ == "__main__":
    n = int(sys.argv[1]) if len(sys.argv) > 1 else 20
    seen = sample(n)
    print(f"distinct outputs: {len(seen)} across {sum(seen.values())} runs")
    for payload, count in seen.most_common(3):
        print(count, payload[:200])

Run it with one argument and read the first line only.

python sample.py 20

Three outcomes matter, and each one changes your plan.

1. distinct outputs: 1 — the behavior is already pinned. Write the test now.
2. distinct outputs: 2..5 — a few fields are volatile. Classify them in step 2.
3. distinct outputs: 20 — nothing is pinned yet. Do not refactor this path today.

That third case is the useful one. It tells you the code has no observable contract. You are not refactoring here; you are designing.

Step 2 — Classify every unstable field

Diff the two most common payloads and list the fields that moved. Each field gets exactly one treatment.

Field kind	Example	Treatment	Why
Seedable	hash order, sampler seed	Set env or seed in the fixture	Same input, same output
Injected	clock, UUID factory	Pass a fake through a seam	Deterministic and realistic
Formattable	float precision, currency	Round in the fixture builder	Compare at contract precision
Quarantined	upstream request_id, latency	Regex-normalize, then assert the rule still matches	Keeps the field visible
Dropped	unbounded log lines	Exclude and record why	Nothing to assert

The quarantine row is the one people skip. A normalize rule that stops matching is a silent hole. You must test the rule itself, not just the payload.

Step 3 — Freeze only what survived

Write the normalizer as data, not as scattered inline edits. Three regexes beat thirty conditionals.

# normalize.py
import re

VOLATILE = {
    "request_id": re.compile(r'"request_id":\s*"[0-9a-f-]{36}"'),
    "created_at": re.compile(r'"created_at":\s*"[^"]+"'),
    "duration_ms": re.compile(r'"duration_ms":\s*[0-9.]+'),
}

def normalize(payload: str) -> str:
    for name, pattern in VOLATILE.items():
        payload = pattern.sub(f'"{name}": "<volatile>"', payload)
    return payload

Then the characterization test itself. It compares normalized output against a fixture committed in the same PR.

# test_invoice_characterization.py
import json
import pytest
from normalize import VOLATILE, normalize
from billing import build_invoice

CASES = json.load(open("fixtures/invoices.json"))

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["name"])
def test_output_matches_characterization(case):
    raw = json.dumps(build_invoice(**case["input"]), sort_keys=True, default=str)
    assert normalize(raw) == case["expected_normalized"]

def test_no_quarantine_rule_is_stale():
    blob = "".join(
        json.dumps(build_invoice(**c["input"]), default=str) for c in CASES
    )
    unused = [n for n, p in VOLATILE.items() if not p.search(blob)]
    assert not unused, f"quarantine rules match nothing: {unused}"

That second test is the anti-rot check. When a dependency stops emitting duration_ms, it tells you. Otherwise your normalizer quietly hides a real change six weeks later.

Regenerate fixtures by hand, in a separate commit, with the reason in the message. Never wire --update into a normal test run.

Step 4 — Change the smallest thing the green test allows

The order is fixed. Tests first, then one edit, then the same tests.

1. Confirm pytest -k characterization is green three times in a row.
2. Re-run python sample.py 20. It must still report one output for the touched path.
3. Make one change: a rename, an extracted function, or an early return.
4. Keep the test file byte-identical in that commit. If the test had to change, you changed behavior. Split that into its own commit.
5. Revert the commit locally and confirm the test still passes. A test that only passes forward is not characterization.

Step 5 catches a common mistake. People write tests that encode the new structure. Those tests fail on revert, so they prove nothing about the old behavior. Keep the assertion anchored to output, not to call shape.

Where a free model and a free server fit in this loop

The sampler decides what is trustworthy. A model cannot make that call for you, and it should not.

I use MonkeyCode's free model access for the boring first pass. It drafts characterization tests from the function body, plus a candidate volatile-field list. That draft is a starting point, not a verdict. The 20/3/2 run decides which drafted assertions survive.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option helps with the sampler, not with the thinking. Twenty subprocess launches with different env vars want a clean box. Running them remotely keeps locale, timezone, and PYTHONHASHSEED controlled. It also stops the run from competing with your local test suite.

Do not paste secrets or private customer payloads into any hosted model or remote machine. Strip fixtures down to structure first. If you cannot strip them, keep the whole loop local and accept the slower run.

Limitations, stated plainly

• Characterization tests pin current behavior, bugs included. They never tell you what is correct.
• Quarantine rules hide regressions if the anti-rot test is skipped in CI.
• Concurrency, streaming, and retry-heavy paths need property tests, not snapshots.
• The sampler costs twenty process launches per path. Budget a minute, not a millisecond.
• A model-drafted test can be confidently wrong. Treat every drafted assertion as unverified until the sampler agrees.

Who should skip this

Skip the loop if the entry point needs a live third-party service. Skip it if output is inherently time-shaped, like a scheduler tick log. Skip it if you cannot run one clean process in isolation. Isolate the seam first, then come back to this.

A 45-minute run sheet

1. Minutes 0–10: pick one function with a JSON or string output. Write entry.py that prints it.
2. Minutes 10–20: run sample.py 20. Record the distinct count in the PR description.
3. Minutes 20–30: classify each moving field with the table above.
4. Minutes 30–40: add normalize.py, one fixture, and the anti-rot test.
5. Minutes 40–45: make one edit. Confirm green, revert, confirm green again.

Publish the distinct count before and after your refactor. One number beats a paragraph of reassurance, and reviewers can reproduce it in a minute. If the sampler finds something surprising in your repo, that finding is the interesting part — write it down before you fix it.