A Flake Freeze May Emit Inconclusive. It May Not Mint a Pass.

An agent patch does not earn a pass because a flaky check was silenced. It earns a pass only when every claimed property returns pass, and every freeze in the ledger is an inconclusive bound to a failure signature already reproduced on the base revision under the same fixture digest. A transform that turns fail into pass is not a freeze. It is a broken gate.

Property checks decide behavior. Fixture digests decide whether two runs are comparable. A flake freeze decides only whether a pre-registered signature may be scored as inconclusive. Those three jobs stay separate, or the suite starts certifying its own exceptions.

Three outcomes, not a mute button

Most CI wrappers collapse results into green and red. A freeze then becomes a filter that deletes red. After the filter, green is indistinguishable from "we agreed not to look."

Agent patches make that collapse worse. The same diff can edit production code, a fixture, and the property that is supposed to judge both. If the freeze applies after that edit, the patch can move the oracle until the frozen check no longer means what the ledger recorded.

Keep three verdicts, and no others. pass means the oracle ran and held. fail means it ran and broke, or it did not run, or its definition moved. inconclusive means a registered signature matched on base, the fixture digest matched, and the oracle file for that id was absent from the diff.

This is the same separation a strict expected-failure mark already draws: an expected failure is not a pass, and an unexpected pass is not something a quarantine should quietly absorb. Deleting the test is a third, worse option. It removes the counterexample instead of labeling it.

Runner status	Freeze match	Oracle edited in this diff	Emitted verdict	Promotion for this id
pass	irrelevant	no	pass	counts as a real pass
fail	no	no	fail	blocks
fail	signature, digest, and base revision match	no	inconclusive	allowed only as non-pass
fail	match	yes	fail	freeze void
pass	irrelevant	yes	fail	catalog drift, blocked
missing run	any	any	fail	blocked

Read the last two rows before tuning anything else. A missing run is not a pass. A property whose definition moved in the same diff cannot inherit an old signature, even if the new body happens to return green.

The table is a specification you can implement. It is not a measured flake rate, and it is not a report from a production gate.

1. Split the diff before any freeze is consulted

Do this before tests, not after a red job has already been muted. The split is what makes the later transform decidable.

git fetch origin main
git diff --name-only origin/main...HEAD > /tmp/patch.paths
git diff --numstat origin/main...HEAD -- src props fixtures

Assign each path to one bucket, and write the buckets down.

1. src/ is behavior. The patch may change it.
2. props/ is the oracle. An edit here voids freezes for the affected ids.
3. fixtures/ is the comparability lock. It must be digested, not described.
4. Other paths are out of scope. They must not be required for the verdict.

If a property id's file appears under props/, add that id to oracle_edited. The freeze ledger is not allowed to soften it. Split the patch, or drop the claim. Do not temporarily freeze an oracle the diff just rewrote.

2. Lock the fixture digest before the run

Comparable runs share bytes, not intentions. Hash paths and contents in a stable order so two machines either agree or fail the match.

import hashlib
import pathlib

def fixture_digest(root: pathlib.Path) -> str:
    hasher = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        hasher.update(path.relative_to(root).as_posix().encode())
        hasher.update(b"\0")
        hasher.update(path.read_bytes())
        hasher.update(b"\0")
    return hasher.hexdigest()

Write the digest beside the run id. A freeze record with a different digest does not match, even when the test name matches. Name equality is not fixture equality.

python3 -c 'from pathlib import Path; import fixture_lock; print(fixture_lock.fixture_digest(Path("fixtures")))' \
  | tee "$RUN_DIR/fixture.digest"

The helper is a proposed lock, not a claim that any particular repository already ships it. If a fixture embeds a timestamp or an absolute path, normalize that field before hashing or the digest will flap for reasons unrelated to behavior.

3. Reproduce the signature on base

A flake freeze without a base reproduction is a nickname for a failure. Collect the signature on the base revision with the candidate fixture digest, not on the patched tree.

git worktree add --detach /tmp/base-replay origin/main
cd /tmp/base-replay
python3 -m propcheck \
  --fixture-digest "$(cat "$RUN_DIR/fixture.digest")" \
  --only-id prop.order_total \
  --signature-out "$RUN_DIR/base.sig.json"

Store four fields only: prop_id, signature, fixture_digest, and base_revision. The signature should hash the assertion id plus a minimized counterexample. Drop timestamps, hostnames, and temporary directories. Keep the values that would still identify the failure tomorrow.

If base does not emit that signature, refuse the freeze. The candidate result stays an ordinary fail. A signature seen only on the patched tree is evidence of a regression, which is the opposite of a pre-existing flake.

4. Run the catalog the patch did not edit

Keep claims in a catalog that humans review. The runner may load it. The runner must not invent it at merge time.

id: prop.order_total
oracle: props/order_total.py
claim: "sum(lines) == total for every fixture row"
mutates_fixture: false

Skip execution for ids listed in oracle_edited, and emit fail with reason oracle_moved. That fail is deliberate. An edited oracle needs a separate review, not a softer score.

Properties outside the diff still run. A patch that only touches src/tax.py can still break prop.order_total. Sampling only the lines the diff mentions is a weaker gate, and it is not this one. The catalog entry is the claim under test. The diff is only the list of files that might void a freeze.

5. Apply the freeze as a pure transform

This module is the artifact. It is unexecuted example code: copy it, run the tests, and reject any change that lets a freeze emit pass.

from dataclasses import dataclass
from typing import Dict, Optional

PASS, FAIL, INCONCLUSIVE = "pass", "fail", "inconclusive"

@dataclass(frozen=True)
class Freeze:
    prop_id: str
    signature: str
    fixture_digest: str
    base_revision: str

@dataclass(frozen=True)
class Run:
    prop_id: str
    status: str
    signature: Optional[str]
    fixture_digest: str
    base_revision: str
    oracle_edited: bool

def apply_freeze(run: Run, freeze: Optional[Freeze]) -> str:
    if run.oracle_edited or run.status not in (PASS, FAIL):
        return FAIL
    if run.status == PASS:
        return PASS
    if freeze is None:
        return FAIL
    matched = (
        freeze.prop_id == run.prop_id
        and freeze.signature == run.signature
        and freeze.fixture_digest == run.fixture_digest
        and freeze.base_revision == run.base_revision
    )
    return INCONCLUSIVE if matched else FAIL

def promote(verdicts: Dict[str, str], claimed: set) -> bool:
    if set(verdicts) != set(claimed) or not claimed:
        return False
    if any(v not in (PASS, INCONCLUSIVE) for v in verdicts.values()):
        return False
    return any(v == PASS for v in verdicts.values())

promote refuses an empty claim set. It also refuses a ledger that is only inconclusive. At least one unedited property must actually pass. Otherwise a patch can freeze every check it might break and ship on silence.

Pin the forbidden transitions next to the function.

def test_freeze_cannot_mint_pass():
    run = Run("prop.order_total", FAIL, "sig-a", "digest-1", "rev-9", False)
    freeze = Freeze("prop.order_total", "sig-a", "digest-1", "rev-9")
    assert apply_freeze(run, freeze) == INCONCLUSIVE

def test_edited_oracle_voids_freeze():
    run = Run("prop.order_total", FAIL, "sig-a", "digest-1", "rev-9", True)
    freeze = Freeze("prop.order_total", "sig-a", "digest-1", "rev-9")
    assert apply_freeze(run, freeze) == FAIL

def test_digest_mismatch_stays_fail():
    run = Run("prop.order_total", FAIL, "sig-a", "digest-2", "rev-9", False)
    freeze = Freeze("prop.order_total", "sig-a", "digest-1", "rev-9")
    assert apply_freeze(run, freeze) == FAIL

def test_all_inconclusive_does_not_promote():
    assert promote({"prop.order_total": INCONCLUSIVE}, {"prop.order_total"}) is False

python3 -m pytest -q test_verdict_gate.py

If pytest is absent, the same assertions fit under unittest. The dependency does not matter. The forbidden transition does. A patch to apply_freeze that returns PASS on a matched freeze should fail review even if every other test is green.

6. Order the commands so the freeze cannot outrun the digest

Parallelism is fine after the digest file exists. It is not fine before that write. One sequence keeps the inputs auditable.

set -euo pipefail
python3 fixture_lock.py fixtures > "$RUN_DIR/fixture.digest"
python3 split_diff.py origin/main...HEAD > "$RUN_DIR/oracle_edited.json"
python3 reproduce_on_base.py \
  --revision origin/main \
  --digest "$(cat "$RUN_DIR/fixture.digest")" \
  --out "$RUN_DIR/base.sig.json"
python3 -m propcheck \
  --catalog catalog/ \
  --edited "$RUN_DIR/oracle_edited.json" \
  --digest "$(cat "$RUN_DIR/fixture.digest")" \
  --out "$RUN_DIR/runs.json"
python3 verdict_gate.py \
  --runs "$RUN_DIR/runs.json" \
  --freezes freezes/ledger.json \
  --claimed catalog/claimed.json

Exit non-zero unless promote returns true. Archive fixture.digest, oracle_edited.json, base.sig.json, and runs.json with the build. A green checkbox without those four files is not evidence. The ledger entry below is the match key, not a certificate.

{
  "prop_id": "prop.order_total",
  "signature": "sig-a",
  "fixture_digest": "<sha256 of fixtures/>",
  "base_revision": "<git revision of origin/main>"
}

Illustrative walkthrough, not a recorded run

The ids below show the combiner. They are not a benchmark and not an incident report.

Candidate claims are prop.order_total and prop.tax_round. Neither oracle file is in the diff. The fixture digest is digest-1 on both the base replay and the candidate run. Base revision rev-9 already emits sig-a for prop.order_total. The candidate run fails that same signature, and prop.tax_round passes.

apply_freeze emits inconclusive for the first id and pass for the second. promote returns true because the claim set matches, nothing is fail, and at least one id actually passed.

Change only the candidate digest to digest-2. The freeze no longer matches. prop.order_total stays fail, and promotion stops. Change only the oracle path so props/order_total.py is in the diff. The freeze is void even if the signature matches, and the emitted verdict is fail with reason oracle_moved.

Remove prop.tax_round from the claim set and leave a single inconclusive. Promotion stops again. Silence is not a ship state.

Where a free model draft and a free server replay fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two operator-supplied options fit this workflow without changing the rule. Free model access can draft a property stub from the diff: an id, a one-line claim, and a suggested oracle path. Treat that draft as a proposal. A human still commits it under catalog/ before the gate will load it. Free server access can host the detached base replay so the signature is collected away from a laptop that may not match CI.

The server does not get to skip the digest check. If the digest on that host differs from the digest recorded for the candidate run, apply_freeze stays on fail. No model name, quota, hardware shape, or retention period is assumed here. If either option is down, run the same commands locally. The gate does not depend on them.

If you already use MonkeyCode, point the free model at the diff only to draft catalog stubs, and point the free server at the base replay. Commit the stubs and the signature ledger yourself. Do not ask either option to emit the final verdict.

Limitations

Signature normalization can hide a real drift. Strip too much from the counterexample and two different failures share a hash, so the freeze matches incorrectly. Keep the assertion id and the minimized counterexample. Drop only fields that are not part of the failure identity.

The table does not prove the catalog claim is the right claim. An unedited tautology still passes this gate if it returns pass. Review new catalog entries in a separate change. This workflow only stops a freeze from laundering a fail, and it stops an edited oracle from inheriting an old signature.

promote allows a mix of pass and inconclusive. That is a product choice for low-risk branches, not a universal merge policy. Teams that cannot ship with any inconclusive should change the combiner so every claimed id is pass. Leave the freeze transform alone. It should still be unable to mint a pass.

Calendar expiry is out of scope. Binding the record to digest, signature, and base revision already makes a stale fixture fail the match. Adding a clock is a different policy, and this article does not specify one.

Nothing here reports a latency number, a flake percentage, or an endpoint measurement. The code is a specification you can execute. It is not evidence about any vendor runtime.

Who should skip this

Skip it if you cannot name a base revision. Without origin/main or an equivalent pin, the signature has nowhere honest to reproduce, and the freeze record cannot be checked.

Skip it if failures have no stable signature, only a job URL. A URL is not a match key. Skip it if every agent patch rewrites props/ and src/ together and review cannot split them. The honest output is then a stream of oracle_moved fails, which is accurate and useless until the split exists.

Skip it on safety-critical paths where inconclusive is not an acceptable ship state. Use the stricter combiner, or do not automate the freeze at all. Skip it if the goal is to raise a dashboard score. The gate is designed to stay red when evidence is missing. That will look slower than a filter that deletes red tests.

What to keep in the repository

Commit verdict_gate.py, the four negative tests, catalog/, and the freezes/ledger.json schema. Do not commit a generated pass certificate. The next patch must recompute the digest and re-run the transform.

The review rule fits in four lines. Pass means the oracle held. Inconclusive means the signature was already on base under this digest. Fail means everything else, including a freeze that tried to become a pass. If a proposed change cannot say which of those three it emitted, it is not ready to merge.