A Fixture Hash Separates Deterministic Misses From Eligible Flake Freezes

A property failure on an agent patch is a miss until a shrink proves otherwise. Freeze eligibility is a later question, and it stays closed until that miss is reduced to a minimal input, stored as a hashed fixture, and replayed on both the base revision and the candidate. Red-build counts do not enter the decision.

Generated edits often fail one sample and pass the next. That pattern looks like a flake. It is frequently a narrow input bug. The reverse mistake is just as costly: a test that already fails on the pinned base gets charged to the patch, and a clean candidate is dropped for noise it did not add.

Three records separate those cases. A property check samples a space under a seed. A fixture pins one input and the hash of that input.

A freeze is a time-bounded exception, legal for a single variance class. Drop any one of the three and the label "flaky" becomes a guess.

Variance classes, not failure totals

The table is a decision procedure for this workflow. It is not a score from a live suite, and it is not a benchmark. Two replays on each revision are the minimum the classifier accepts. They are not a tuned sample size for every property.

Class	Base on locked fixture	Candidate on locked fixture	Freeze eligible	Required next step
pass	all pass	all pass	no	keep other gates; do not open a freeze
deterministic_miss	all pass	all fail	no	reject the patch; keep the fixture as a regression lock
seed_unstable	all pass	mixed	no	draw a new local seed budget; do not freeze
env_unstable	both sides mixed	both sides mixed	yes, with owner and expiry	file the freeze against the fixture hash only
unresolved	any other matching-hash pattern	any other matching-hash pattern	no	add replays or hand the row to a human
fixture_drift	hash mismatch	hash mismatch	no	shrink again; discard the stale file

Read the table left to right. One row permits a freeze. The other rows reject the patch, demand more local evidence, or rebuild the fixture. A shared miss on base and candidate is not enough; both sides must be mixed, or the row stays unresolved.

1. Record the seed before the candidate runs

A sample without a seed cannot be replayed, so it cannot support a freeze. Bind the property id, the seed, and the base revision before the agent edit is applied.

from dataclasses import dataclass

@dataclass(frozen=True)
class SampleKey:
    property_id: str
    seed: int
    base_rev: str

def open_sample(property_id: str, seed: int, base_rev: str) -> SampleKey:
    if seed < 0:
        raise ValueError("seed must be a non-negative integer")
    if not property_id or not base_rev:
        raise ValueError("property id and base revision are required")
    return SampleKey(property_id, seed, base_rev)

The function is a reference sketch, and it has not been packaged as a library. Its job is to refuse an unbound sample so later steps have a key. Treat every later command in this article the same way: proposed shape, not a released CLI, and not a timed result.

2. Run that seed on base, then on the candidate

Same seed, two revisions. Store both outcomes before any shrink. If both pass, this seed is done. If both fail, keep the row, because the base is already dirty and a freeze is not yet justified.

python -m gate.sample --property order_total --seed 14021 --rev base
python -m gate.sample --property order_total --seed 14021 --rev candidate

Those commands describe the sketch only. No runtime was measured for them, and no public corpus was classified with them.

3. Shrink only a candidate-only miss

Shrink when the candidate fails and the base passes on that seed. The reduced input must still fail. A reduced input that passes is not a fixture.

import hashlib
import json

def project_minimal(failing_input: dict, keep: tuple[str, ...]) -> dict:
    missing = [key for key in keep if key not in failing_input]
    if missing:
        raise KeyError("cannot shrink; required keys missing")
    return {key: failing_input[key] for key in keep}

def fixture_id(minimal: dict) -> str:
    blob = json.dumps(minimal, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

keep is the property grammar, written by the test author. Projecting onto that tuple is a stand-in for a real reducer. Replace it with a library shrinker when you have one, then confirm the reduced value still fails. Until that replay is red, call the file a proposal, not a lock.

Canonical JSON matters as much as the shrink. sort_keys=True and compact separators stop key order from changing the digest. Without that, two identical inputs become fixture_drift, and the classifier never reaches a real class.

4. Lock the hash and replay the file

Write the minimal JSON and its SHA-256. Replay the file, not the original random sample. Two runs on base and two on the candidate feed the classifier.

Increase the repeat count when the property reads a clock, a temp directory, or process scheduling. Do not drop below two. A mismatched hash is drift, even if every assertion is green.

python -m gate.replay --fixture fixtures/order_total/locked.json --rev base --repeat 2
python -m gate.replay --fixture fixtures/order_total/locked.json --rev candidate --repeat 2

5. Classify, then open a freeze for one class only

from enum import Enum

class Variance(Enum):
    PASS = "pass"
    DETERMINISTIC_MISS = "deterministic_miss"
    SEED_UNSTABLE = "seed_unstable"
    ENV_UNSTABLE = "env_unstable"
    UNRESOLVED = "unresolved"
    FIXTURE_DRIFT = "fixture_drift"

def classify(base: list[str], candidate: list[str], hash_ok: bool) -> Variance:
    if not hash_ok:
        return Variance.FIXTURE_DRIFT
    if len(base) < 2 or len(candidate) < 2:
        raise ValueError("two replays are required on each revision")
    if any(item not in ("pass", "fail") for item in base + candidate):
        raise ValueError("outcomes must be pass or fail")
    base_fail = base.count("fail")
    cand_fail = candidate.count("fail")
    if base_fail == 0 and cand_fail == 0:
        return Variance.PASS
    if base_fail == 0 and cand_fail == len(candidate):
        return Variance.DETERMINISTIC_MISS
    if base_fail == 0 and 0 < cand_fail < len(candidate):
        return Variance.SEED_UNSTABLE
    if 0 < base_fail < len(base) and 0 < cand_fail < len(candidate):
        return Variance.ENV_UNSTABLE
    return Variance.UNRESOLVED

def freeze_eligible(kind: Variance) -> bool:
    return kind is Variance.ENV_UNSTABLE

Walk three hand-built rows. They are illustrations, not captured logs.

Base ["pass", "pass"] and candidate ["fail", "fail"], with a matching hash, yields deterministic_miss. freeze_eligible is false. Keep the fixture and reject the patch.

Base ["fail", "pass"] and candidate ["fail", "pass"] yields env_unstable. A freeze may be filed. It still needs an owner, an expiry, and the fixture hash. It does not approve the patch, because the base is already unstable.

Base ["pass", "pass"] and candidate ["fail", "pass"] yields seed_unstable. That is the row teams mislabel as flake.

The base is clean, and the locked file did not fail twice. Draw another seed, or raise the repeat count. Do not write a freeze to make the row quiet.

unresolved covers the rest. A candidate that always fails while the base is already dirty is in this bucket. So is a base-only failure. Add replays, or send the row to a human. Do not squeeze either pattern into env_unstable just to unblock a ticket.

6. Call a model only after the class exists

Local classification is the gate. A draft from a model is a new candidate, and it must re-enter steps 4 and 5.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Use the model access to draft a narrower diff after the ledger shows deterministic_miss or seed_unstable. Use the free server option to replay the locked fixture against that draft.

Those two options do not add a variance class, a quota figure, or a pass. This article does not claim a model list, a hardware shape, a time limit, or a measured save. Send a short request: the fixture hash, the assertion name, and the already bounded diff.

Ask for an edit that still fails the locked fixture if the bug remains. A sentence that calls the miss flaky is commentary. The function freeze_eligible remains the decision.

If you already store fixture hashes, add that function in front of the freeze ticket and keep model drafts behind it. That is the whole adoption step.

Ledger row

Persist one JSON object per classified fixture. Hashes in examples are placeholders, not digests of a published file.

{
  "property_id": "order_total",
  "seed": 14021,
  "fixture_sha256": "<sha256 of locked json>",
  "base_rev": "<pinned base>",
  "candidate_rev": "<pinned candidate>",
  "base_runs": ["pass", "pass"],
  "candidate_runs": ["fail", "fail"],
  "class": "deterministic_miss",
  "freeze_eligible": false
}

Reject rows that omit base_rev or fixture_sha256. An unbound row cannot be audited a week later, which is when a freeze is usually questioned. Pair the row with the canonical fixture bytes, not with a screenshot of a red job.

Who should skip this

Do not run the replays against live payment sandboxes, shared staging databases, or tests that send mail. Repeated fixture runs would duplicate side effects. Skip the workflow when base and candidate cannot be checked out at pinned revisions. Without those pins, env_unstable is a label with no referent.

Do not auto-wire freeze_eligible to merge. The legal product of a true result is a ticket with an expiry, not a commit. Authorization, encryption, and deletion patches need a human reader even when the class is pass. A green property sample is not a security review.

Shrink quality tracks keep. If the tuple drops the field that breaks the candidate, the fixture goes green and the original seed stays red. That split means the grammar is wrong. Fix keep before anyone discusses a freeze.

Two replays miss rare failures. For a property that already fails occasionally across a long local streak, either raise the repeat count or ban freeze eligibility for that property. This sketch states no universal streak length, because none was measured here.

Integrating a full shrinker from a property-based testing library is follow-on work. Do not cite this page as evidence that a particular library version behaves a certain way.

Start with one pure property

Choose a pure function whose inputs you can list. Record a seed, force one candidate-only miss, and confirm the classifier returns deterministic_miss. Then file no freeze. If that path is not boring yet, the hash is not stable, and the rest of the table will lie.