Close a Flake Freeze Only When the Replay Budget Hits Zero

An agent patch earns a flake freeze only when three records agree: a named property contract, one fixture identity across the replay set, and a replay budget that has already hit zero on a single failure signature. A quiet log from a free server does not satisfy that rule. A draft produced through free model access does not satisfy it either.

That split is the strategy. Execution cost and test authority are different lanes. Mixing them is how a blinking assertion becomes a permanent ignore, and how a real invariant break gets mislabeled as noise.

This article specifies a local decision procedure for that split. It does not report a benchmark, a quota, a hardware shape, or a model name. The Python gate later in the piece is an unexecuted reference implementation. Run it locally before you rely on the table. It performs no network call.

Why agent patches stress this gate

Agent patches arrive with a wide edit surface and a short explanation. The tempting response is to rerun the suite until it looks green, then freeze whatever still blinks. That response fails for a mechanical reason.

A blink can come from the assertion, from the fixture, from the runner class, or from the transport that delivered the patch. Those causes do not share a remedy. Treating them as one flaky bucket makes the ignore list a junk drawer.

Property checks answer a different question than regression names do. A property states an invariant over a declared input domain. A fixture identity states which inputs were actually bound. A replay budget states how many independent evidence runs remain before a repeated signature may be quarantined as a freeze candidate. If any of those records is missing, the gate stays shut.

Records the gate requires

Keep four fields, not a boolean.

1. contract_digest hashes the property name, the invariant text, and the input domain. A renamed test file does not preserve credit.
2. fixture_id hashes the bound inputs, not the wall clock. Two runs with different bound inputs are different experiments, even when the test name matches.
3. runner_class is either evidence or cost_probe. Only evidence may decrement the replay budget.
4. failure_signature is a stable string with a required prefix. invariant: means the property failed. flake: means the same check did not fail on every evidence run. transport: means the run never produced a verdict.

A freeze document may cite only a flake: signature collected on the evidence runner. It may not cite an invariant failure, a transport fault, or a cost-probe log. If the signature builder cannot emit one of those prefixes, the row is rejected evidence, not a pass.

Numbered workflow

1. Refuse a draft that has no property contract

Parse the patch proposal before any runner starts. Require a contract with a non-empty name, invariant, and input domain. If the draft is only a diff, stop.

Do not fill in a tautology such as assert True or result is not None and call it a property. Those checks do not constrain the domain, so they cannot justify a later freeze. A missing contract is a reject, not an invitation to spend replay budget.

Free model access is relevant here as a drafting aid, not as a judge. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied claim used in this workflow is narrow: free model access can help draft a patch and a property contract, and a free server option can host a cost probe. This article does not assert model identifiers, rate limits, hardware, retention, or how long either option remains available. If a draft omits the contract, discard it.

2. Bind the fixture identity before the first run

Compute fixture_id from the canonical JSON of the bound inputs. Store that id on the ledger next to contract_digest. Every later observation must echo both values.

A mismatch is not a flake. It is rejected evidence, because the replay set no longer describes one experiment. Do not let the patch under test rewrite the fixture file during the run. If the harness must generate inputs, generate them before the digest is sealed, then treat regeneration as a new identity.

3. Classify the runner before counting the result

Label the free server path cost_probe at admission, not after the log looks convenient. A cost probe may answer a cheap question: does the patch import, does the contract parse, does a single sample throw a transport error.

It must not decrement budget_remaining, and it must not appear in the freeze citation. Point evidence at a runner you control, with a recorded image digest and a fixed command. If you do not have that runner, you do not have a freeze. You have a draft.

4. Spend the replay budget only on evidence

Set a small integer budget before the first evidence run. Four completed evidence observations are enough for this reference gate to distinguish a first blink from a stable signature. You may choose another positive integer, but choose it before looking at failures.

Each completed evidence observation decrements the counter by one, whether it passed or failed. Stop early only for an invariant: signature or a digest mismatch. Those are terminal. Do not use up the budget to average away a property violation. A property failure rejects the patch. It never opens a freeze.

5. Close a freeze only on signature stability at zero

When the counter hits zero, read the evidence rows that still match the sealed fixture and contract. Ignore cost-probe rows while you do that read.

Mixed flake: signatures mean the failure is not stable. Hold the ledger open. Do not freeze. A single flake: signature, with budget already at zero, and with no invariant or transport row in the evidence set, is freeze-eligible. All passes mean no freeze is required. Accept the evidence set. Do not invent a freeze in case the next run blinks.

Any transport: row blocks the freeze. Retry policy belongs to the runner, not to the test-ignore list. Attach an expiry when a person accepts the freeze document. This gate does not choose that duration. Reopen the ledger when the fixture identity or the contract digest changes. A freeze does not travel with a new fixture.

Decision table

Read this table as the specification. The code that follows is only an encoding of it.

Evidence set	Budget	Decision
No evidence rows, only cost probes	any	ignore_probe
Contract or fixture digest mismatch	any	reject_evidence
Any invariant: on evidence	any	reject_patch
Any transport: on evidence	any	transport_fault
Unprefixed or empty signature	any	reject_evidence
Two or more flake: signatures	any	hold_open
One flake: signature	greater than zero	quarantine
One flake: signature	zero	freeze_eligible
All evidence rows passed	zero	accept
All evidence rows passed	greater than zero	hold_open

Quarantine is not a freeze. It is a hold while budget remains. Publishing an ignore during quarantine is a process bug, not a judgment call. accept is also not a waiver: it says this evidence set did not need a freeze. The next fixture identity starts a new ledger.

Reference gate

Save the following file as replay_budget_gate.py. It uses the standard library only. The listing is unexecuted until you run the checks in the next section. Annotations are postponed so the file stays a straight import on interpreters that accept from __future__ import annotations.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from enum import Enum

class RunnerClass(str, Enum):
    EVIDENCE = "evidence"
    COST_PROBE = "cost_probe"

class Decision(str, Enum):
    ACCEPT = "accept"
    REJECT_PATCH = "reject_patch"
    REJECT_EVIDENCE = "reject_evidence"
    HOLD_OPEN = "hold_open"
    QUARANTINE = "quarantine"
    FREEZE_ELIGIBLE = "freeze_eligible"
    IGNORE_PROBE = "ignore_probe"
    TRANSPORT_FAULT = "transport_fault"

@dataclass(frozen=True)
class Contract:
    name: str
    invariant: str
    input_domain: str

    def digest(self) -> str:
        payload = json.dumps(
            {
                "name": self.name,
                "invariant": self.invariant,
                "input_domain": self.input_domain,
            },
            sort_keys=True,
            separators=(",", ":"),
        )
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()

@dataclass(frozen=True)
class Observation:
    runner: RunnerClass
    fixture_id: str
    contract_digest: str
    signature: str | None
    transport_fault: bool = False

def fixture_id(bound_inputs: dict) -> str:
    raw = json.dumps(bound_inputs, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

def decide(
    expected_digest: str,
    expected_fixture: str,
    budget_remaining: int,
    rows: list[Observation],
) -> Decision:
    if budget_remaining < 0:
        raise ValueError("budget_remaining must be >= 0")
    evidence = [row for row in rows if row.runner is RunnerClass.EVIDENCE]
    if not evidence:
        return Decision.IGNORE_PROBE
    flakes: list[str] = []
    for row in evidence:
        if row.contract_digest != expected_digest or row.fixture_id != expected_fixture:
            return Decision.REJECT_EVIDENCE
        if row.transport_fault or (row.signature or "").startswith("transport:"):
            return Decision.TRANSPORT_FAULT
        if (row.signature or "").startswith("invariant:"):
            return Decision.REJECT_PATCH
        if row.signature is None:
            continue
        if not row.signature.startswith("flake:"):
            return Decision.REJECT_EVIDENCE
        if row.signature not in flakes:
            flakes.append(row.signature)
    if len(flakes) > 1:
        return Decision.HOLD_OPEN
    if len(flakes) == 1 and budget_remaining > 0:
        return Decision.QUARANTINE
    if len(flakes) == 1 and budget_remaining == 0:
        return Decision.FREEZE_ELIGIBLE
    if budget_remaining == 0:
        return Decision.ACCEPT
    return Decision.HOLD_OPEN

A passing row carries signature=None. A flake row carries a flake: prefix. An invariant row never becomes eligible, even if the same string repeats until the budget is gone. Cost-probe rows are filtered out before any of those branches.

Commands that lock the table

The checks below are the acceptance tests for the decision table. They are not a performance study, and they do not call a hosted model or a hosted server.

python - <<'PY'
from replay_budget_gate import (
    Contract, Observation, RunnerClass, Decision, decide, fixture_id,
)
c = Contract("order_total", "total == sum(lines)", "non-empty line lists")
fid = fixture_id({"lines": [2, 2], "seed_set": [7, 11]})
digest = c.digest()
probe = Observation(RunnerClass.COST_PROBE, fid, digest, "flake:timeout")
assert decide(digest, fid, 0, [probe]) is Decision.IGNORE_PROBE
bad = Observation(RunnerClass.EVIDENCE, "other", digest, "flake:timeout")
assert decide(digest, fid, 0, [bad]) is Decision.REJECT_EVIDENCE
inv = Observation(RunnerClass.EVIDENCE, fid, digest, "invariant:total")
assert decide(digest, fid, 0, [inv]) is Decision.REJECT_PATCH
one = Observation(RunnerClass.EVIDENCE, fid, digest, "flake:order-timeout")
assert decide(digest, fid, 2, [one, one]) is Decision.QUARANTINE
assert decide(digest, fid, 0, [one, one]) is Decision.FREEZE_ELIGIBLE
mixed = Observation(RunnerClass.EVIDENCE, fid, digest, "flake:other")
assert decide(digest, fid, 0, [one, mixed]) is Decision.HOLD_OPEN
ok = Observation(RunnerClass.EVIDENCE, fid, digest, None)
assert decide(digest, fid, 0, [ok, ok]) is Decision.ACCEPT
assert decide(digest, fid, 1, [ok]) is Decision.HOLD_OPEN
print("replay budget gate: table checks passed")
PY

Read the printed line. Do not treat a missing line as a pass. If you later wrap this gate in CI, store contract_digest, fixture_id, budget_remaining, and the decision string next to the patch id. Store cost-probe logs under a separate prefix so a reviewer cannot cite them by accident.

Limitations

This gate classifies records you already trust. It does not discover properties, generate fixtures, or prove that a signature is semantically meaningful. A stable flake: string can still be the wrong abstraction. If two distinct faults share one signature, the freeze will hide both. Review the signature builder with the same care you would give an assertion.

The budget is a count of completed evidence observations, not a measure of statistical confidence. Four matching failures are not a confidence interval. They are a process threshold that prevents a freeze on the first blink. Raise the threshold when the assertion is security-sensitive or moves money. Do not lower it because a free runner was slow or because a draft was cheap to produce.

Transport faults stay outside the flake lane on purpose. An empty body, a timeout, or an admission error is a delivery problem. This article does not prescribe a retry or lease policy for that class of fault. It only forbids promoting the fault into a test ignore.

No latency, cost, or quality comparison is offered. Free access, as supplied for this draft, is an availability claim only. It can make a cost probe easier to start. It cannot make that probe authoritative. Nothing here should be cited as a current quota, a supported model list, or a permanence guarantee. Those facts were not supplied, so they are out of scope.

Who should not use this approach

Skip this gate if you cannot name an evidence runner you operate. A freeze built only from free-server logs will launder runner noise into policy.

Skip it if the change sits under a regulatory or safety standard that requires a formal verification plan. A four-row replay budget is a software process control, not a certification artifact.

Skip it if the team wants the agent to edit the ignore list. The freeze document should be written by the gate, reviewed by a person, and rejected when the fixture identity moves. An agent that can both fail a test and authorize the ignore has no counterparty.

Skip it for suites that have no properties, only snapshot filenames. There is nothing to bind a contract digest to, so the table collapses into taste. In that case, write properties first. Do not start with a freeze.

Where a free draft lane still helps

Use free model access to draft the contract text and the patch, then throw that draft at step 1. Use the free server only as the cost_probe runner in step 3. Keep the evidence counter on infrastructure you can rebuild from a recorded image digest.

If a cost probe and an evidence run disagree, believe the evidence run. Do not open a freeze to reconcile them. If you already have those two free options, park them on the draft lane and the probe lane, and leave the zero-budget decision to the evidence set.