A Flake Freeze Starts From a Shared Property Failure

An agent patch does not become flaky because the new tree failed a property twice. A flake-freeze proposal is legal only when the parent commit fails that same property under the same contract. If the parent passes and the patch fails, the class is regression. Further reruns on the patch do not move it.

This gate sits beside fixture locks, input shrinking, and assertion-waiver rules. It does not replace them. It answers one question those checks leave open: was the failure already present on the parent?

One conjunction, seven fields

All seven fields must hold before a proposal file may be written.

1. Property spec hash matches across parent and patch.
2. Fixture closure hash matches.
3. Seed schedule matches, including order and count.
4. Parent status for that property is fail.
5. Patch status for that property is fail.
6. Normalized witness class matches.
7. The schedule is not uniformly red: at least one seed disagrees with another on the parent or on the patch.

A missing field is a reject. There is no close-enough row.

Write the contract before the first run

Two logs without a shared contract cannot be scored. Put the contract in the review directory and ignore any run that cites another hash.

contract_version: 1
parent_sha: "abc123"
patch_sha: "def456"
spec_hash: "spec-9f2c"
fixture_closure_hash: "fix-44ab"
seeds: [11, 29, 47]
properties:
  - id: prop.order_total_non_negative
    oracle: "total >= 0"

Those hashes are inputs from earlier checks. This note does not redefine them. If the agent edits the property text or the fixture set, the hashes diverge and this gate stops. Spec review is a different queue.

Steps

1. Separate the trees

Run parent and patch from different worktrees. Reconstructing the parent result from memory is not a baseline.

git worktree add ../parent-tree "$PARENT_SHA"
git worktree add ../patch-tree "$PATCH_SHA"

2. Emit one JSON object per seed

A missing line is an incomplete contract, not a pass. Keep status in pass or fail only.

python3 run_properties.py \
  --tree ../parent-tree \
  --contract differential-contract.yaml \
  --out parent.jsonl
python3 run_properties.py \
  --tree ../patch-tree \
  --contract differential-contract.yaml \
  --out patch.jsonl

3. Normalize, do not shrink

Map each fail to a witness class you already trust: property id, exception type, and a stable assertion id. Leave input minimization to the shrinking gate. If the classes differ, the patch introduced a new failure mode even when both sides are red.

4. Score locally

The scorer reads the two JSONL files and the seed list. It writes a decision object. It does not edit tests, and it does not delete a failing assertion.

5. Require a human and an expiry

propose_freeze is a review ticket. A person adds an expiry and a reason code. When the expiry lapses, delete the ticket. The next patch must earn a new one.

Copying the file forward is a process bug, not a renewal.

How to read the table

Parent	Patch	Witness	Seed split	Result
pass	pass	n/a	n/a	accept, no freeze file
pass	fail	any	any	reject, regression
fail	pass	n/a	n/a	accept path, no freeze file
fail	fail	same	yes	freeze proposal allowed
fail	fail	same	no	reject, uniform fail
fail	fail	different	any	reject, new failure mode
missing	any	any	any	reject, incomplete contract

A seed split means the schedule contains both pass and fail for that property on at least one side. Uniform red is not flake evidence. It is a stable miss, and this path does not waive stable misses.

One regression rejects the whole patch, even if a second property looks like a shared flake. The patch is the unit of merge. Partial credit would let a real break ride in beside an unrelated noisy check.

Reference scorer

This module is a reference implementation. Run it yourself before you trust it. It is not a report of production counts, and it does not call a remote model.

#!/usr/bin/env python3
"""Score parent vs patch property runs. Unexecuted until you run it."""

from __future__ import annotations

import json
from pathlib import Path
from typing import Iterable

def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text().splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def split(statuses: Iterable[str]) -> bool:
    seen = set(statuses)
    return "pass" in seen and "fail" in seen

def decide(parent: list[dict], patch: list[dict], seeds: list[int]) -> dict:
    if not parent or not patch:
        return {"decision": "reject", "reason": "incomplete_contract", "results": []}

    def index(rows: list[dict]) -> dict:
        return {(r["prop"], int(r["seed"])): r for r in rows}

    pmap, cmap = index(parent), index(patch)
    props = sorted({r["prop"] for r in parent} | {r["prop"] for r in patch})
    results = []

    for prop in props:
        p_rows = [pmap.get((prop, s)) for s in seeds]
        c_rows = [cmap.get((prop, s)) for s in seeds]
        if any(r is None for r in p_rows + c_rows):
            results.append({"prop": prop, "decision": "reject", "reason": "missing_seed"})
            continue

        def side_status(rows: list[dict]) -> str:
            return "fail" if any(r["status"] == "fail" for r in rows) else "pass"

        def witnesses(rows: list[dict]) -> list[str]:
            return sorted({r.get("witness", "") for r in rows if r["status"] == "fail"})

        p_status, c_status = side_status(p_rows), side_status(c_rows)
        p_wit, c_wit = witnesses(p_rows), witnesses(c_rows)

        if p_status == "pass" and c_status == "pass":
            item = {"prop": prop, "decision": "accept", "reason": "both_pass"}
        elif p_status == "pass" and c_status == "fail":
            item = {"prop": prop, "decision": "reject", "reason": "regression"}
        elif p_status == "fail" and c_status == "pass":
            item = {"prop": prop, "decision": "accept", "reason": "parent_fail_cleared"}
        elif p_wit != c_wit:
            item = {"prop": prop, "decision": "reject", "reason": "witness_mismatch"}
        elif not (split(r["status"] for r in p_rows) or split(r["status"] for r in c_rows)):
            item = {"prop": prop, "decision": "reject", "reason": "uniform_fail"}
        else:
            item = {"prop": prop, "decision": "propose_freeze", "reason": "shared_variant_fail"}
        results.append(item)

    if any(r["decision"] == "reject" for r in results):
        return {"decision": "reject", "results": results}
    if any(r["decision"] == "propose_freeze" for r in results):
        return {"decision": "propose_freeze", "results": results}
    return {"decision": "accept", "results": results}

def main() -> None:
    parent = load_jsonl(Path("parent.jsonl"))
    patch = load_jsonl(Path("patch.jsonl"))
    contract = json.loads(Path("differential-contract.json").read_text())
    print(json.dumps(decide(parent, patch, contract["seeds"]), indent=2))

if __name__ == "__main__":
    main()

Commands that exercise the branches

Regression case. The parent line is green. The patch line is red. Reading the branches, that pair selects regression.

printf '%s\n' '{"prop":"prop.order_total_non_negative","seed":11,"status":"pass","witness":""}' > parent.jsonl
printf '%s\n' '{"prop":"prop.order_total_non_negative","seed":11,"status":"fail","witness":"AssertionError:total"}' > patch.jsonl
printf '%s\n' '{"seeds":[11]}' > differential-contract.json
python3 differential_gate.py

Shared variant case. Seed 11 fails on both sides with the same witness. Seed 29 passes on the parent and fails on the patch. That is a split, so the function can emit propose_freeze. The file is still a proposal.

cat > parent.jsonl <<'EOF'
{"prop":"prop.order_total_non_negative","seed":11,"status":"fail","witness":"AssertionError:total"}
{"prop":"prop.order_total_non_negative","seed":29,"status":"pass","witness":""}
EOF
cat > patch.jsonl <<'EOF'
{"prop":"prop.order_total_non_negative","seed":11,"status":"fail","witness":"AssertionError:total"}
{"prop":"prop.order_total_non_negative","seed":29,"status":"fail","witness":"AssertionError:total"}
EOF
printf '%s\n' '{"seeds":[11,29]}' > differential-contract.json
python3 differential_gate.py

Count lines before you argue about the color. A review note that cannot show these counts is incomplete.

python3 - <<'PY'
import json
from collections import Counter
from pathlib import Path
for name in ("parent.jsonl", "patch.jsonl"):
    c = Counter(
        json.loads(line)["status"]
        for line in Path(name).read_text().splitlines()
        if line.strip()
    )
    print(name, dict(c), "lines", sum(c.values()))
PY

What the counts are allowed to mean

Publish object counts, not a flake rate you did not measure. Two fails out of two seeds is a uniform fail under this contract. It is not a suite-wide flake rate, and it is not evidence about other properties.

If you lengthen the schedule, change seeds in the contract and rerun both trees. Adding patch-only reruns until a green appears is selection, not measurement.

Cleared parent failures need a second look that this function does not do. parent_fail_cleared means do not open a freeze. It does not mean the patch preserved the oracle. If the diff edits a bound, a comparator, or a fixture, send it to spec review even when the table says accept.

Free model drafts, free server runs, local score

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use MonkeyCode's free model access before the gate, as a drafting aid. Ask it for candidate properties from the patch summary, then edit that text until it is the oracle you are willing to enforce. Commit the edited spec. The scorer never takes a model label as an input.

A paragraph that calls a red log timing noise cannot override a parent pass.

Use the free server option only as capacity for step 2, when local CPU is the constraint. The job spec must carry parent_sha, patch_sha, spec_hash, fixture_closure_hash, and the seed list, and it must return both JSONL files. Score those files on a machine you control, with the module in this article. Discard a response that is empty, partial, or tied to a contract hash you did not submit.

A transport success is not a test result.

This note states no model names, quotas, hardware sizes, or end dates. Free access and a free server are operator-supplied options, and either can be absent on a given day. If they are absent, run the same contract elsewhere. The table does not loosen to fit the outage.

Keep the classifier in the repository you review, and treat the remote run as a replaceable worker.

Limits

A short schedule can miss a rare parent failure. The gate then reports regression for a patch fail that might have been shared. That false block is the conservative error, and it is accepted. The opposite error, waiving a real break, is not accepted.

Do not remove the parent run to clear a block.

Witness equality is only as sharp as the normalizer. Timestamps, memory addresses, and temp paths split one failure into many classes and force witness_mismatch. Strip those fields first.

Shared external services can fail on both trees and look like a shared variant. If the property calls a network, a clock you do not control, or another team's staging host, do not use this gate. Hermetic fixtures are a precondition, not a footnote.

The function also ignores coverage. A patch can delete a call site and still show both_pass on a property that no longer executes the risky branch. Pair this table with a coverage or mutation check if that risk matters in your codebase. This article does not supply that second check.

Who should skip it

Skip the gate when there is no parent commit, as in a first import of generated tests. There is no baseline to difference. Skip it when spec hash or fixture closure hash changed.

Skip it for screenshot and pixel tests whose failures do not reduce to a stable witness class. Skip it if your merge bot treats a freeze file as an automatic override. This design refuses that workflow.

Also skip it when you cannot store the JSONL files. A prose comment that the parent failed too is not a substitute. Without the files, a later reader cannot audit the conjunction, and the proposal is void.

Review note, four lines

Store the contract hash, the top-level decision, the per-property reason codes, and the JSONL line counts. Those four lines are the evidence packet. If any line is missing, do not merge on the strength of a flake label.

The parent tree is the control. The patch tree is the treatment. A freeze is a time-boxed ticket about a shared variant failure, not a mute button for a regression the control never showed.