An agent patch does not earn a flake freeze from one red run on a wide fixture. The freeze may open only after the property predicate, the stopped witness, and a disagreeing rerun series are all on the same record. A stable failure is a reject, and a stable pass is not a freeze.
That rule is narrower than rerunning a suite until it turns green. A rerun hides a regression when the input is still large, the seed changes, or the property text moves with the patch. Shrink first, then repeat, then classify.
A digest match on the original fixture is not the decision. The digest that matters is the one taken after shrinking stops, on the patch that actually ran.
Bind four identifiers before you classify
Every attempt carries a patch digest, a property id, a fixture digest, and a seed. Missing any one of them is a hold, not a pass and not a freeze. The property id must point at a predicate a reviewer can read without opening a model transcript. The fixture digest hashes the input that actually ran, not the directory the agent was given at the start.
A property is a pure check over an explicit fixture and the observed output. Wall clocks, network calls, and map iteration order stay out of the predicate unless they are fixture fields. If the patch edits the predicate, the property id changes. An oracle that the diff rewrote cannot certify the edit that rewrote it.
Hash the patch before the first run. If the worktree does not match that hash, stop. Later reruns that include a small fix are a different patch and need a new record. The gate does not inherit evidence across digests.
Shrink until the failure stops moving
A failing fixture is often larger than the bug. Unused flags, extra rows, and stale optional keys make the next run look random when one field is doing the work. Shrinking deletes or narrows a piece only when the same property still fails. The loop stops when the next candidate no longer fails, or when the candidate is identical to the current fixture.
The stopped fixture is the witness. Later reruns must match its digest. A rerun on the original wide file goes back to hold, even if the property id is unchanged. Otherwise the freeze record cites a test that was not the one under dispute.
This stop condition does not prove the witness is minimal in the mathematical sense. It proves only that the supplied shrink function had no further failing step within a fixed step budget. That is a bound on the tool. It is not a claim about every smaller input a person might invent.
Keep the failure identity stable while shrinking. If the property returns a reason code, a shrink step that changes the code is refused and the previous fixture remains the witness. A new reason code is a different failure, and it should not be folded into the original freeze discussion.
Decision table
The table below is the whole classifier. It has no hidden score.
| Observed state | Gate |
|---|---|
| Patch digest, property id, fixture digest, or seed is empty | Hold |
| Shrink loop has not stopped, or the step budget ended without a local stop | Hold |
| Fewer reruns than the policy minimum on that witness | Hold |
Outcome labels are outside pass and fail
|
Hold |
| Every recorded rerun passes | Pass |
| Every recorded rerun fails | Reject |
| Reruns disagree, shrink stopped, expiry timestamp present | Freeze candidate |
| Reruns disagree, expiry timestamp empty | Hold |
The sample defaults are policy fields, not measurements.
| Policy field | Value in the sample | What it is not |
|---|---|---|
min_reruns |
3 | a confidence interval or a false-accept rate |
max_steps |
32 | a proof that no smaller failing input exists |
freeze_expires |
required non-empty string | a recommended duration |
Three reruns is the smallest series that can show agreement, total failure, or disagreement without treating one run as a story. Teams that already require five reruns should pass that number in, and should not treat the default as a benchmark.
A freeze candidate is not a merge approval. It records that one witness was unstable under one seed and one digest. Expiry is an operator-supplied timestamp, and this article does not recommend a duration. An empty expiry keeps the candidate closed, so a freeze cannot become a permanent skip by omission.
Numbered procedure
- Compute the patch digest and refuse to run if the worktree is dirty relative to it.
- Select a property id the diff does not touch. If the predicate file is in the diff, stop and review the oracle before any rerun or shrink.
- Execute the property once on the submitted fixture, with the seed recorded in the fixture header or the job spec. Store the outcome and the fixture digest.
- On failure, shrink until a local stop or until the step budget ends. If the budget ends without a stop, hold. Do not freeze a witness the shrinker was still changing.
- Hash the stopped witness. Rerun the same property on that digest with the same seed until the policy minimum is met. Do not draw a new seed between attempts.
- Apply the decision table. Persist the four identifiers, the outcomes, and the expiry field in one record. A pull-request comment is not a substitute for that record.
Initial passes still need an explicit sample plan when the patch claims to fix a prior flake. One green run on a wide fixture does not retire a freeze candidate from an older digest. The old candidate expires on its own timestamp, or a reviewer closes it by pointing at a new stable pass on the same property id and a recorded witness. It does not vanish because the branch is green today.
Proposal code
The module below is a proposal. It was not executed against a production suite for this article. It does not call a model, and it does not open a network connection. fails and shrink_once are injected by the harness you already trust.
import hashlib
from dataclasses import dataclass
from enum import Enum
class Gate(str, Enum):
PASS = "pass"
REJECT = "reject"
HOLD = "hold"
FREEZE_CANDIDATE = "freeze_candidate"
@dataclass(frozen=True)
class Witness:
patch_digest: str
property_id: str
fixture_digest: str
seed: str
shrunk: bool
outcomes: tuple[str, ...]
freeze_expires: str
shrink_budget_hit: bool = False
def sha256_bytes(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def bounds_check(fixture: dict, output: dict) -> bool:
return fixture["lo"] <= output["value"] <= fixture["hi"]
def drop_optional(fixture: dict):
if "optional_note" not in fixture:
return None
nxt = dict(fixture)
del nxt["optional_note"]
return nxt
def decide(w: Witness, min_reruns: int = 3) -> Gate:
if min_reruns < 2:
raise ValueError("min_reruns must compare more than one attempt")
required = (w.patch_digest, w.property_id, w.fixture_digest, w.seed)
if any(not item for item in required):
return Gate.HOLD
if w.shrink_budget_hit or not w.shrunk:
return Gate.HOLD
if len(w.outcomes) < min_reruns:
return Gate.HOLD
if any(item not in {"pass", "fail"} for item in w.outcomes):
return Gate.HOLD
unique = set(w.outcomes)
if unique == {"pass"}:
return Gate.PASS
if unique == {"fail"}:
return Gate.REJECT
if not w.freeze_expires:
return Gate.HOLD
return Gate.FREEZE_CANDIDATE
def shrink_until_local_stop(fixture, fails, shrink_once, max_steps=32):
current = fixture
for _ in range(max_steps):
if not fails(current):
return None
nxt = shrink_once(current)
if nxt is None or nxt == current or not fails(nxt):
return current
current = nxt
return current
bounds_check is only a shape example. Replace it with the predicate the patch claims to satisfy. drop_optional shows the shrink contract: return None when nothing optional remains, and never drop lo or hi in the same step. shrink_until_local_stop returns None when the current fixture does not fail, which signals an unsound step or a mismatched predicate. Callers should hold in that case rather than invent a witness.
A fixture header that can be hashed as canonical JSON looks like this. Sort keys before hashing so two equivalent objects do not produce two digests.
{
"patch_digest": "<64 lowercase hex chars>",
"property_id": "bounds_check",
"seed": "17",
"fixture": {"lo": 0, "hi": 10, "optional_note": "drop-me"}
}
A branch check can sit beside the module. The command is an example, not a reported run from a published build. The timestamp only shows that the expiry field is populated.
python - <<'PY'
from gate_witness import Witness, Gate, decide
base = dict(patch_digest="p"*64, property_id="bounds_check",
fixture_digest="f"*64, seed="17", shrunk=True,
shrink_budget_hit=False)
flake = Witness(**base, outcomes=("fail", "pass", "fail"),
freeze_expires="2026-10-19T00:00:00Z")
assert decide(flake) is Gate.FREEZE_CANDIDATE
stable = Witness(**base, outcomes=("fail", "fail", "fail"),
freeze_expires="2026-10-19T00:00:00Z")
assert decide(stable) is Gate.REJECT
no_end = Witness(**base, outcomes=("fail", "pass", "fail"), freeze_expires="")
assert decide(no_end) is Gate.HOLD
print("branches ok")
PY
sha256sum witness/minimal.json
Use the expiry your review policy already requires. Do not copy the sample timestamp into a real record.
Separate the draft from the rerun
Writing a predicate and executing it are different jobs. A fluent draft can look like a test while still encoding the bug it is supposed to catch. Keep generation on one side of the record and execution on the other.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode's free model access can draft a candidate predicate or a candidate shrink step as text. That text is untrusted input. It enters the repository only after the same review you would apply to a human patch, and it must not be a file the agent diff already rewrote.
MonkeyCode's free server option can host the rerun command so the witness digest is computed outside a worktree that still has uncommitted fixture edits. This article does not state quotas, hardware, model names, or how long either option remains available. If the remote job cannot print the patch digest it executed, discard the outcomes and hold.
Pin the command, the seed, and the witness file in the job spec. A runner that rebuilds a lookalike fixture is not repeating the witness. Similarity is not a digest match.
Limitations
The shrinker is unsound when shrink_once removes the field that caused the failure and fails still returns true for a different reason. The gate will then reject or freeze the wrong witness. Prefer shrink steps that delete one optional key at a time, and refuse a step whose reason code changes.
Disagreement across the minimum reruns does not name the flake source. The cause may be a race, hash-ordered iteration, or a clock the fixture forgot to pin. The candidate only blocks automatic merge. It is not a root-cause report.
A pass on the stopped witness says nothing about inputs the shrinker never produced. If the property is meant to cover a range, keep a separate broader sample. That sample must cite the same patch digest and the same property id, or it belongs to a different gate and cannot be added into this one.
No false-accept rate is claimed for the branch checks. The shell snippet locks control flow only. It does not inject faults, and it should not be cited as a detection percentage.
Who should leave this gate closed
Do not point a shrink loop at live third-party services. Repeated calls create load, and remote jitter masquerades as disagreement.
Do not use the table when the patch exists to change the property file. Review that oracle by hand, then mint a new property id.
Do not open a freeze during incident response if the failing check is the only signal on a known bad deploy. Incidental disagreement is not a reason to silence that signal. The stable-fail branch is the relevant output there, and a freeze is the wrong one.
Also skip the procedure if the four identifiers cannot be stored beside the result. A green badge without a witness digest is not evidence the property ran.
If a free server seat is already how reruns are hosted, print the patch digest and the witness digest before the gate word, and keep the shrink log in the same commit as that patch digest. The decision table still outranks the runner.
Top comments (0)