Don't Freeze a Flake Until You Can Replay the Input

A freeze list that stores test names is not a testing strategy. It is a permission slip. Agent patches learn that pattern in one review cycle: rename the test, catch a broader exception, or append the noisy case to flakes.json and watch CI go green.

The merge rule is narrower. A property failure must shrink to a canonical input, land as a fixture, and replay under a fixed seed. Only a pinned fixture that still flickers on a remote runner may enter a freeze file. Local laptops do not get a vote.

That ordering is the whole method. Properties find the hole. Fixtures keep the hole. Freezes are a last-resort label for residual non-determinism, not a dump for anything an agent found inconvenient.

Why name-only freezes leak

Agent patches optimize the gate they can see. If the gate is "pytest exit code 0," the cheapest edits are assertion deletion, over-broad skip, and freeze-file appends. Those edits leave no replayable input. The next patch then has nothing to regress against.

A fixture is different. It is a byte-stable argument to the same oracle. Replay it tomorrow and the oracle still has a job. A freeze entry with no input is a comment that pretends to be a policy.

Environment noise makes the leak worse. Locale, timezone, dict iteration, and filesystem order all change across laptops. A flake observed once on a developer machine is not a classification. It is a rumor.

Three artifacts, one promotion path

Keep three files, and never let the patch agent write the last two without a classified run:

1. properties.py — predicates over generated inputs. They may fail. That is their purpose.
2. fixtures/ — canonical, shrunk inputs that previously failed a property. They must fail or pass the same way on every replay.
3. flake_freeze.json — only residual flicker after a fixture already exists. Each row points at a fixture hash, a seed, and an environment class. Never at a test name alone.

The promotion path is one-way. Failures become fixtures. Fixtures that remain unstable may become freeze rows. Freeze rows never become a substitute for a missing fixture.

Observation	Next artifact	Allowed freeze?
Property fails, input shrinks, replay is stable-fail	New fixture (regression)	No
Property fails, input shrinks, replay is stable-pass	Discard as non-reproducible	No
Pinned fixture mixed pass/fail on remote, fixed seed	Freeze row keyed by fixture hash	Yes, with expiry owned by humans
Test name failed once on a laptop	Nothing	No

The table is the policy. If a row cannot be filled, the patch does not merge.

A seed-locked classify loop

The harness below is a proposed local tool, not a production metric report. It does four jobs: generate, shrink, canonicalize, classify. Run it with a fixed seed. Do not run it against the agent's working tree as the source of truth.

# classify_failures.py — proposed harness, not a measured production run
from __future__ import annotations

import hashlib, json, os, random, unicodedata
from pathlib import Path
from typing import Any, Callable

FIXTURE_DIR = Path("fixtures")
FREEZE_PATH = Path("flake_freeze.json")
REMOTE_CLASS = os.environ.get("ENV_CLASS", "local")
SEED = int(os.environ.get("PROP_SEED", "20260917"))
REPLAYS = int(os.environ.get("REPLAYS", "7"))


def canonical_bytes(value: Any) -> bytes:
    # NFC + sorted keys + compact separators. Label: proposed default, not a standard.
    if isinstance(value, str):
        value = unicodedata.normalize("NFC", value)
    text = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return text.encode("utf-8")


def fixture_id(value: Any) -> str:
    return hashlib.sha256(canonical_bytes(value)).hexdigest()[:16]


def shrink(value: Any, oracle: Callable[[Any], bool]) -> Any:
    """Naive shrink. Keep the smallest prefix/dict that still falsifies."""
    if isinstance(value, str):
        lo, hi = 1, len(value)
        best = value
        while lo <= hi:
            mid = (lo + hi) // 2
            cand = value[:mid]
            if not oracle(cand):
                best, hi = cand, mid - 1
            else:
                lo = mid + 1
        return best
    if isinstance(value, dict):
        items = list(value.items())
        for i in range(len(items), 0, -1):
            cand = dict(items[:i])
            if not oracle(cand):
                return cand
    return value


def replay(oracle: Callable[[Any], bool], value: Any, n: int, seed: int) -> str:
    rng = random.Random(seed)
    # Touch rng so later generators can share the seed channel.
    rng.randrange(2)
    results = [oracle(value) for _ in range(n)]
    if all(results):
        return "stable_pass"
    if not any(results):
        return "stable_fail"
    return "flicker"


def classify(oracle: Callable[[Any], bool], sample: Any) -> dict[str, Any]:
    if oracle(sample):
        return {"action": "no_failure", "freeze": False}
    shrunk = shrink(sample, oracle)
    fid = fixture_id(shrunk)
    status = replay(oracle, shrunk, REPLAYS, SEED)
    record = {
        "fixture_id": fid,
        "seed": SEED,
        "env_class": REMOTE_CLASS,
        "status": status,
        "input": json.loads(canonical_bytes(shrunk)),
    }
    if status == "stable_fail":
        FIXTURE_DIR.mkdir(exist_ok=True)
        (FIXTURE_DIR / f"{fid}.json").write_bytes(canonical_bytes(shrunk) + b"\n")
        record["action"] = "pin_fixture"
        record["freeze"] = False
        return record
    if status == "stable_pass":
        record["action"] = "drop_nonreproducible"
        record["freeze"] = False
        return record
    record["action"] = "freeze_after_pin"
    record["freeze"] = REMOTE_CLASS == "remote"
    return record

Two invariants matter more than the helper functions. Canonicalization happens before hashing. Classification happens after shrinking. Skip either step and the freeze file starts accumulating aliases of the same noise.

Wire an oracle that the agent patch must not weaken. Example: a JSON merge function that claims to be associative.

# properties.py — proposed oracles. Unexecuted example.

def merge(a: dict, b: dict) -> dict:
    out = dict(a)
    out.update(b)
    return out


def oracle_associative(sample: dict) -> bool:
    a, b, c = sample["a"], sample["b"], sample["c"]
    left = merge(merge(a, b), c)
    right = merge(a, merge(b, c))
    return canonical_bytes(left) == canonical_bytes(right)

If an agent "fixes" associativity by catching AssertionError, the oracle still returns a boolean. The classify loop never reads pytest skip marks. That split is deliberate.

Numbered merge procedure

Use this sequence on every agent patch that touches behavior. Do not reorder it to make a demo green.

1. Freeze the seed in CI, not in a developer shell history. PROP_SEED is an input, not a suggestion.
2. Generate a bounded batch of property samples. Failures go through shrink until the input stops getting smaller.
3. Write the canonical bytes to fixtures/<id>.json. Commit that file in the same change as the patch, or reject the patch.
4. Replay the fixture REPLAYS times on a remote environment class. Laptop results may inform debugging. They do not write flake_freeze.json.
5. If replay is stable_fail, keep the fixture and fail the merge until the production code matches the oracle. That is a regression, not a flake.
6. If replay is stable_pass, drop the sample. Do not freeze a ghost.
7. If replay is flicker and the environment class is remote, append a freeze row keyed by fixture_id + seed + env_class. Humans own deletion of that row. The agent does not.

A minimal freeze schema:

{
  "schema": "fixture-hash-v1",
  "rows": [
    {
      "fixture_id": "a1b2c3d4e5f60789",
      "seed": 20260917,
      "env_class": "remote",
      "reason": "flicker-after-pin",
      "owner": "human"
    }
  ]
}

Reject any row that names a pytest node id and nothing else. Names move. Hashes of canonical inputs do not, unless the input changed, which is a different patch.

Commands for the remote lane:

export PROP_SEED=20260917
export REPLAYS=7
export ENV_CLASS=remote
python classify_failures.py
pytest -q properties.py fixtures/
test -z "$(git diff -- fixtures flake_freeze.json | grep 'nodeid')"

The last check is crude on purpose. If a diff introduces a test-name key, the procedure already failed.

Where a free remote lane earns its keep

Property campaigns are sensitive to the machine that runs them. That is the classification bug. A freeze decision made on a laptop with a warm cache and a personal locale is not the same decision CI will make on Monday.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is useful for drafting candidate predicates from a written spec. Those drafts are prompts, not oracles. A human still has to accept or rewrite every property before it can fail a merge. MonkeyCode's free server option is the remote environment class in the table above: run classify_failures.py there so ENV_CLASS=remote is literal, not honorary.

Do not send proprietary fixtures, credentials, or production dumps to a free shared runner. The method needs environment isolation, not data leakage. If the patch under test cannot be represented by public-sized inputs, keep the remote lane inside your own network and apply the same seed and canonicalization rules.

The product pieces are optional. The promotion path is not. Remove every product name and the classify table still stands.

What this does not prove

Canonicalization can hide real bugs. Sorted JSON keys will not catch a server that depends on original insertion order. If order is observable behavior, drop sort_keys for that oracle and pin the sequence explicitly.

Naive shrinking can overshrink. A prefix that still fails may drop the field that made the failure interesting. Review the pinned fixture. If it is a trivial empty dict, the oracle is too weak or the shrinker is too eager.

Replay count is a budget, not a proof of determinism. Seven repeats will miss rare races. Increase REPLAYS for concurrency code, or stop claiming a freeze is scientific.

The harness does not inspect whether the agent rewrote the oracle. Pair it with a separate permission boundary: properties.py and classify_failures.py are review-only paths. That boundary is out of scope here; without it, the agent will edit the boolean.

No runtime numbers, model names, or uptime claims are attached to the free-server lane. Treat it as an environment class, then measure it in your own logs.

Who should not use this

Skip the freeze file entirely if the suite has no source of non-determinism you are willing to name. A freeze row without a stated cause is how name-only lists come back.

Skip property generation if the behavior has no independent oracle: visual layout, marketing copy, or one-off scripts. Fixtures of expected screenshots are a different contract.

Skip the free remote lane if inputs are sensitive, if licenses forbid third-party runners, or if the team cannot tell ENV_CLASS=local from ENV_CLASS=remote in the freeze schema. In those cases, run the same classify loop on an internal worker and keep the promotion path.

Teams that already pin shrunk counterexamples and refuse name-only skips do not need a new slogan. They need to keep the agent out of the freeze file.

The durable output of a failed property is not a skip mark. It is a file you can hash, replay, and argue about. Freeze only what still flickers after that file exists. If you want the classify loop executed off your laptop, MonkeyCode's free server option is one remote class that fits the table. The merge rule does not change if you pick a different remote.