Accept a Flake Freeze Only When Base Replays the Same Miss

A flake freeze is an admission decision, not a test result. For an agent patch, that decision is valid only when the unmodified base commit misses the same property on the same seed, the property-source digest is unchanged, and the fixture epoch on both runs matches the epoch the score claims. Drop any one of those facts and the freeze becomes a label for a regression the patch introduced.

Agent diffs often edit product code and tests together. A skip list keyed only by test name cannot separate those edits. Property checks add a second split: a generator may rewrite the property between the failing run and the frozen run. The gate then scores a different statement from the one that failed.

The checker in this note is a proposed local function. Sample records are illustrative inputs for that function. They are not incident rates, model scores, or timing measurements.

Records the verdict may use

Four facts decide admission. Neighboring logs may be stored. They do not vote.

Field	Where it comes from	Reject when
base_misses	Bounded replay on the base commit	Zero
base_attempts	Same replay, passes included	Outside 1..cap
property_sha256	Hash of the property file after the run	Differs from base
fixture_epoch	Hash of the epoch lock after the run	Differs from base or from the claim

patch_misses explains why a freeze was requested. It never grants one. A patch can fail a property that base passes. That pattern is REJECT_PATCH_INTRODUCED, which is the outcome a quarantine file is most often misused to hide.

Codes, then the constants

Callers fail closed. An unknown code is a reject.

• ACCEPT_PREEXISTING means base missed at least once inside the cap, and both digests match the claim.
• REJECT_PATCH_INTRODUCED means the patch missed and base did not.
• REJECT_PROPERTY_DRIFT means the property file hash changed. Review the oracle edit. Do not freeze it.
• REJECT_FIXTURE_EPOCH means the epoch lock or the claimed epoch disagrees.
• REJECT_BUDGET means the attempt count left the allowed range.
• REJECT_INCOMPLETE means a required field was absent, or a replay line lacked a result.

ATTEMPT_CAP = 5 and MIN_BASE_MISSES = 1 are repository policy, not a tuned result. Raise the miss floor when the suite is noisy. Do not raise the cap to farm a miss. This note does not estimate how often real suites flake.

Proposed checker

The listing uses only the standard library. It is meant to be executed locally, then wired to a job. It has not been run against a production gate in this article.

from hashlib import sha256
from pathlib import Path

ATTEMPT_CAP = 5
MIN_BASE_MISSES = 1

REQUIRED = (
    "base_misses",
    "base_attempts",
    "patch_misses",
    "property_sha256",
    "base_property_sha256",
    "fixture_epoch",
    "base_fixture_epoch",
    "claimed_epoch",
)

def freeze_verdict(rec: dict) -> str:
    if any(k not in rec for k in REQUIRED):
        return "REJECT_INCOMPLETE"
    attempts = rec["base_attempts"]
    if not isinstance(attempts, int) or attempts < 1 or attempts > ATTEMPT_CAP:
        return "REJECT_BUDGET"
    if rec["property_sha256"] != rec["base_property_sha256"]:
        return "REJECT_PROPERTY_DRIFT"
    if rec["fixture_epoch"] != rec["base_fixture_epoch"]:
        return "REJECT_FIXTURE_EPOCH"
    if rec["fixture_epoch"] != rec["claimed_epoch"]:
        return "REJECT_FIXTURE_EPOCH"
    if rec["base_misses"] >= MIN_BASE_MISSES:
        return "ACCEPT_PREEXISTING"
    if rec["patch_misses"] >= 1:
        return "REJECT_PATCH_INTRODUCED"
    return "REJECT_BUDGET"

def file_sha256(path: str) -> str:
    return sha256(Path(path).read_bytes()).hexdigest()

def reduce_attempts(lines: list) -> dict:
    if any("result" not in row or "n" not in row for row in lines):
        return {"status": "REJECT_INCOMPLETE"}
    misses = sum(1 for row in lines if row["result"] == "miss")
    return {"status": "ok", "base_attempts": len(lines), "base_misses": misses}

Illustrative calls, using placeholder digests:

preexisting = {
    "base_misses": 2,
    "base_attempts": 5,
    "patch_misses": 3,
    "property_sha256": "aa",
    "base_property_sha256": "aa",
    "fixture_epoch": "epoch-token",
    "base_fixture_epoch": "epoch-token",
    "claimed_epoch": "epoch-token",
}
introduced = dict(preexisting, base_misses=0, patch_misses=4)
assert freeze_verdict(preexisting) == "ACCEPT_PREEXISTING"
assert freeze_verdict(introduced) == "REJECT_PATCH_INTRODUCED"
assert freeze_verdict({"base_misses": 1}) == "REJECT_INCOMPLETE"
assert reduce_attempts([{"n": 1, "result": "pass"}, {"n": 2}])["status"] == "REJECT_INCOMPLETE"

CPython 3.11 or newer runs the file without third-party packages. The asserts check control flow only. They do not certify an agent, a model, or a repository.

Attempt lines are append-only

Two workers must not share a summary file that can be rewritten in place. A rewrite can drop a pass and keep a miss. That manufactures ACCEPT_PREEXISTING. Append one JSON object per attempt, then reduce.

{"seed":"pinned-seed","n":1,"result":"pass"}
{"seed":"pinned-seed","n":2,"result":"miss"}

base_attempts is the line count. base_misses counts lines whose result equals miss. A line missing result fails the whole record as REJECT_INCOMPLETE. Truncation is a failed gate, not a shorter sample. Parallel workers should append to separate files and concatenate only after both processes exit.

Six steps before the quarantine file changes

Do these in order. Stopping at any reject code is the point of the procedure. A later step cannot repair an earlier hash miss.

1. Hash the claim on the scored tree. Store digest bytes, not a nickname. A renamed epoch with the same contents is the same epoch. A rewritten file that keeps the old nickname is not.

sha256sum fixtures/epoch.lock
sha256sum tests/properties/order_total.py
git rev-parse HEAD

On macOS, shasum -a 256 replaces sha256sum. Compare hex digits only. Ignore the tool's filename column.

1. Replay base in a second worktree. A dirty index in the patched checkout copies fixture edits into the supposed base result. Detach the worktree at the base commit.

git worktree add --detach ../replay-base "$BASE_SHA"
git -C ../replay-base rev-parse HEAD
git -C ../replay-base status --porcelain

status --porcelain must be empty before replay starts. If it is not, return REJECT_INCOMPLETE and delete the worktree. Do not clean unexpected files into the replay.

1. Pin the seed from the patch failure. A fresh draw is a different experiment. If the runner cannot accept that seed, the request is incomplete. Do not substitute another seed that happens to fail.

cd ../replay-base
python -m pytest tests/properties/order_total.py -p no:cacheprovider -q --tb=line

Adapt the seed flag to the runner you ship. The required shape is one property, one seed, no cache provider, and no extra options that hide skips. A skip is not a miss. Collection errors are not misses either.

1. Count starts and misses apart. Increment base_attempts when the process starts. Increment base_misses only when that property's assertion fails. Timeouts, collection errors, and skips stay out of the miss count. Stop at the cap. Failures after the cap are still REJECT_BUDGET.

2. Re-hash after the process exits. Hash the property file and the epoch lock on both trees after replay, not before. A loader that rewrites either file must show up as drift. Property mismatch is REJECT_PROPERTY_DRIFT. Epoch mismatch is REJECT_FIXTURE_EPOCH.

3. Write a freeze only for ACCEPT_PREEXISTING. Persist the seed, the base SHA, both hashes, both counts, and the verdict. A later job that cannot recompute those fields deletes the record. It does not inherit it.
   

{
  "property_id": "order_total_non_negative",
  "seed": "pinned-seed",
  "base_sha": "placed-by-the-job",
  "verdict": "ACCEPT_PREEXISTING",
  "base_misses": 2,
  "base_attempts": 5
}

That object is a schema sketch for the job. It is not a transcript copied from a live gate. Placeholder tokens such as epoch-token and pinned-seed are not fixture releases.

Remove the worktree after the verdict is stored.

git worktree remove --force ../replay-base

A leftover tree is how the next job inherits a dirty base. Removal belongs on both the accept path and every reject path.

Keep the draft off the replay worker

Drafting a property and executing it are different jobs. A model may propose candidate text. Only a clean worktree should run it. One shared workspace lets generated files and fixture caches cross the boundary the hashes are there to detect.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied availability claims used here are narrow: free model access for drafting candidate properties, and a free server option for the base-worktree replay, so the patched checkout is not also the generator scratch disk. This note does not assert a model id, a quota, a machine size, a retention window, or that either option stays free. Check the current product console before a gate depends on that path. If the path is unavailable, freeze_verdict still runs on any host that can add a second git worktree.

The admission rule does not change with the host. A matching fixture epoch remains necessary and is not sufficient. It shows sample identity. It does not show that the miss existed before the patch.

Limitations

The hash check is blunt on purpose. A legitimate oracle fix changes property_sha256 and is rejected. Route that edit through a reviewed digest update. Do not replace the equality check with a similarity score.

One miss inside five attempts is a policy knob, not a flake-rate finding. No suite variance is reported here. Noisy suites should raise MIN_BASE_MISSES and keep the cap small enough that unbounded reruns cannot earn a freeze.

Miss counts collapse different assertion failures under one property id. Split the property before freezing if those failures need different dispositions. Otherwise a timeout and a real invariant break share one quarantine line.

Live dependencies void the base replay. A property that calls a changing external service can miss on base for reasons the patch neither caused nor inherited. Keep those properties outside this gate until they are hermetic.

The checker does not parse the patch diff. That omission is deliberate. A diff heuristic can miss a generated file the worktree still executed. Post-run hashes see the bytes that ran.

Who should not use this

Skip the procedure when the base commit cannot be checked out, when properties are regenerated and discarded on every run, or when quarantine is a global release switch rather than a per-property record. Also skip it when the patch exists to replace the oracle. Those edits need a digest review, and the freeze file should stay untouched.

Teams that already refuse a score when fixture digests disagree still need this admission step. Digest agreement answers which sample ran. The base miss answers whether this failure was available without the patch. Only the second question justifies a freeze.