Only SCHEDULER May Enter the Flake Freeze Ledger

An agent patch earns a flake freeze only after the failing run is labeled, and only for the seed that was classified. The only legal label is SCHEDULER. PROPERTY and FIXTURE failures stay on the reject path, even when a later retry goes green.

That rule is the merge gate. A freeze ledger that accepts unlabeled retries will hide a regression the candidate introduced, then describe the hide as a stability fix.

This note specifies the labels, the decision table, and a Python classifier you can run on one JSON record. No pass rate is claimed. The listing is a proposal, not a measured result.

What the three labels mean

PROPERTY means the oracle rejected the candidate on a fixture the base revision still owns. The assertion fired, the digest matched, and the same seed is red on unmodified base, or that replay was never stored as pass or fail. A second green attempt on the candidate does not erase the first red oracle.

FIXTURE means the bytes under test are not the bytes you named. A hash mismatch, or a generator path sitting inside the candidate diff, lands here. A green score on a rewritten fixture does not transfer, and it must not be filed as noise.

SCHEDULER means one triple failed and then cleared under a cap that was fixed in advance. The triple is property path, fixture digest, and seed. The same seed passed on unmodified base, and the declared wall-time and sample budgets still held.

Only SCHEDULER may enter the freeze ledger. The entry needs an expiry and an owner. Anything else is a reject, or a pass that does not need a freeze.

INCOMPLETE is a hard stop, not a mild flake. A missing digest, a missing seed, a false transcript flag, or a base result outside the set {pass, fail} ends the decision. Do not promote a gap.

CONTAMINATED is the row that still rejects when every hash matches. If the property file or the fixture generator sits in the candidate diff, the patch is grading an edit it just made. Split that edit into another review, or drop the score.

Decision table

Read the rows from top to bottom. The first match wins. The function later in this note encodes that order, and it does not emit a confidence score.

Signals in hand	Label	Freeze eligible
Any required field missing, or transcript_complete is false	INCOMPLETE	no
Property path or fixture generator path is listed in diff_paths	CONTAMINATED	no
fixture_sha256 differs from base_fixture_sha256	FIXTURE	no
budget_ok is false	BUDGET	no
base_seed_result is neither pass nor fail	INCOMPLETE	no
Candidate failed, base seed passed, a retry inside the cap passed	SCHEDULER	yes, one triple, with expiry
Candidate failed for any other reason	PROPERTY	no
Candidate passed	PASS	no freeze to grant

A budget miss is not a flake. Freezing it would move a slow or unbounded patch into the pass set without a new oracle result. Keep BUDGET on the reject path.

Workflow

1. Emit the run record before the word flake is allowed

Each attempt writes one JSON object. Required keys are run_id, seed, property_path, property_sha256, fixture_sha256, base_fixture_sha256, diff_paths, candidate_result, base_seed_result, retry_results, retry_cap, budget_ok, and transcript_complete.

Add fixture_generator_path when the suite has a generator. Do not default a missing base replay to pass. Absence is not evidence.

Write retry_cap before retries start. A cap chosen after you have seen a green retry is not an input. It is a conclusion, and this gate does not accept it.

2. Refuse a self-graded oracle

A property path listed in diff_paths cannot certify the patch that edited it. The same ban applies to the fixture generator. The check is syntactic. It does not prove that an untouched property is strong.

If the production change and the property change are both legitimate, land them in separate reviews. Score the production change only against property hashes that were fixed before that change.

3. Classify with one function

Save the listing as classify_failure.py. It is a proposal. It has not been executed against a production suite for this article.

#!/usr/bin/env python3
"""Classify one agent-patch run before any flake freeze is written.

Proposal. Not a measured benchmark.
"""

from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

LABELS = (
    "INCOMPLETE",
    "CONTAMINATED",
    "FIXTURE",
    "BUDGET",
    "PROPERTY",
    "SCHEDULER",
    "PASS",
)

REQUIRED = (
    "run_id",
    "seed",
    "property_path",
    "property_sha256",
    "fixture_sha256",
    "base_fixture_sha256",
    "diff_paths",
    "candidate_result",
    "base_seed_result",
    "retry_results",
    "retry_cap",
    "budget_ok",
    "transcript_complete",
)


def _sha256(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def classify(record: dict) -> dict:
    missing = [key for key in REQUIRED if key not in record]
    if missing or not record.get("transcript_complete"):
        return {"label": "INCOMPLETE", "freeze": False, "missing": missing}

    if not isinstance(record["diff_paths"], list) or not isinstance(
        record["retry_results"], list
    ):
        return {"label": "INCOMPLETE", "freeze": False, "reason": "type"}

    diff = set(record["diff_paths"])
    watched = {record["property_path"], record.get("fixture_generator_path") or ""}
    watched.discard("")
    overlap = watched & diff
    if overlap:
        return {
            "label": "CONTAMINATED",
            "freeze": False,
            "paths": sorted(overlap),
        }

    if record["fixture_sha256"] != record["base_fixture_sha256"]:
        return {"label": "FIXTURE", "freeze": False}

    if not record["budget_ok"]:
        return {"label": "BUDGET", "freeze": False}

    if record["base_seed_result"] not in ("pass", "fail"):
        return {"label": "INCOMPLETE", "freeze": False, "reason": "base_seed_result"}

    candidate_failed = record["candidate_result"] != "pass"
    if not candidate_failed:
        return {"label": "PASS", "freeze": False}

    cap = int(record["retry_cap"])
    retries = record["retry_results"][:cap]
    retry_passed = any(item == "pass" for item in retries)
    if record["base_seed_result"] == "pass" and retry_passed:
        return {
            "label": "SCHEDULER",
            "freeze": True,
            "seed": record["seed"],
            "property_path": record["property_path"],
            "fixture_sha256": record["fixture_sha256"],
        }
    return {"label": "PROPERTY", "freeze": False}


def _self_check() -> None:
    """In-process examples. Not a production log."""
    base = {
        "run_id": "r1",
        "seed": 1402,
        "property_path": "tests/props/test_retry_budget.py",
        "property_sha256": "p" * 64,
        "fixture_sha256": "f" * 64,
        "base_fixture_sha256": "f" * 64,
        "diff_paths": ["src/retry.py"],
        "candidate_result": "fail",
        "base_seed_result": "pass",
        "retry_results": ["pass"],
        "retry_cap": 2,
        "budget_ok": True,
        "transcript_complete": True,
    }
    assert classify(base)["label"] == "SCHEDULER"
    assert classify(dict(base, fixture_sha256="e" * 64))["label"] == "FIXTURE"
    graded = dict(base, diff_paths=["tests/props/test_retry_budget.py"])
    assert classify(graded)["label"] == "CONTAMINATED"
    assert classify(dict(base, transcript_complete=False))["label"] == "INCOMPLETE"
    assert classify(dict(base, budget_ok=False))["label"] == "BUDGET"
    assert classify(dict(base, base_seed_result="fail"))["label"] == "PROPERTY"
    assert classify(dict(base, candidate_result="pass"))["label"] == "PASS"
    assert classify(dict(base, base_seed_result="not_run"))["label"] == "INCOMPLETE"


def main(argv: list[str]) -> int:
    if len(argv) == 2 and argv[1] == "--self-check":
        _self_check()
        print("self-check ok")
        return 0
    if len(argv) != 2:
        print("usage: classify_failure.py RUN.json | --self-check", file=sys.stderr)
        return 2
    record = json.loads(Path(argv[1]).read_text(encoding="utf-8"))
    decision = classify(record)
    raw = json.dumps(record, sort_keys=True, separators=(",", ":"))
    decision["record_sha256"] = _sha256(raw)
    json.dump(decision, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if decision["label"] in LABELS else 1


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Process exit status is 0 for every known label, including PROPERTY. A successful classification is not a passing patch. The merge job has to read label and freeze from stdout.

Ignoring stdout and trusting the exit code will admit every reject. That is why the hash of the record is printed beside the label. A traceback is not a pass. Repair the record and rerun.

Run the built-in examples with one command. They cover the table rows that do not need a git checkout. They are not a flake-rate study.

python3 classify_failure.py --self-check

4. Replay the failing seed on base

A base_seed_result typed from memory is not a replay. Check out the pinned base revision in a clean worktree and run that seed once.

# Proposal. replay_seed.py is a one-shot wrapper you own.
# It must not retry, and it must not invent upstream pytest flags.
git worktree add ../base "$BASE_SHA"
python3 replay_seed.py --workdir ../base --property "$PROPERTY" \
  --seed "$SEED" --expect-fixture-sha256 "$FIXTURE_SHA"
python3 classify_failure.py run.json

Map wrapper exit 0 to pass and exit 1 to fail. Any other exit, including a missing digest, is incomplete, and the classifier will refuse SCHEDULER.

Do not retry base until it passes and then store pass. A base retry would drag a red property onto the SCHEDULER row.

A candidate cap of two is a workable default for this policy, not a measured optimum. A cap of "until green" is not a cap. If the cap is raised after a failure, rewrite the record. Do not patch it in place.

5. Persist a freeze only for SCHEDULER

A legal freeze names one triple, an expiry, and an owner. It does not mute the property file. It does not mute other seeds.

{
  "label": "SCHEDULER",
  "property_path": "tests/props/test_retry_budget.py",
  "seed": 1402,
  "fixture_sha256": "<base digest>",
  "expires_after_runs": 20,
  "owner": "<reviewer id>",
  "record_sha256": "<hash printed by the classifier>"
}

On each later run, decrement expires_after_runs when that triple is skipped. At zero, delete the freeze and require a new classification. A freeze with no expiry is a skip, and a skip is outside this policy.

The value 20 in the example is a policy sketch, not a measured half-life. If the run record changes after the decision, record_sha256 no longer matches. Treat the freeze as void. Do not re-hash quietly and keep the waiver.

6. Keep the rest of the property in force

A freeze covers one seed on one digest. It does not retire the property. Run the same property at the seeds that are not listed in the freeze file.

If any of those seeds classifies as PROPERTY, reject the patch. A green retry on seed 1402 cannot vouch for seed 1403. That is why the freeze record stores the seed, not only the file path.

Keep the check order identical to the table. Contamination comes before digest. Digest comes before budget. Budget comes before the scheduler row. Reordering those tests mints false SCHEDULER labels.

Worked records

The cases below are constructed. They are not logs from a live service, and they are not customer data. The same branches are what --self-check asserts.

Digest mismatch, every other field present. fixture_sha256 is abc, base_fixture_sha256 is def. Label FIXTURE. Freeze is false. Pin the fixture or reject the patch. A candidate retry does not repair a different input.

Candidate fail, base seed fail, digests equal, property outside the diff, budget_ok true. Label PROPERTY. The oracle is red on both sides, or it was already red on base. Neither case is scheduler noise.

Candidate fail, base seed pass, retry_results equal to ["pass"], retry_cap 2, budgets held, property path absent from diff_paths, digests equal, transcript complete. Label SCHEDULER. A freeze may be written for that seed only. The next seed stays in scope.

If base_seed_result is not_run, the label is INCOMPLETE even when a candidate retry passed. Fill the replay. Do not infer it.

Host the base replay away from the agent worktree

The classifier is only as honest as the base replay. A shared dirty worktree can turn fixture drift into a false SCHEDULER label, because both commands see the same mutated files.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free server option is a practical host for that base replay when the classification step should not share a worktree with the agent runner. Free model access can draft extra property ideas for a human to edit and accept.

Both are availability claims supplied for this assignment. This note does not name models, quotas, hardware, duration, or permanence, and none of those were verified here.

A drafted property is not a passing run. Review it, hash it, and keep that hash outside the candidate diff for any run that uses it. If the free server cannot check out the same base revision and the same fixture digest, do not write base_seed_result from it.

A convenient host that cannot reproduce the digest is a FIXTURE row, not a faster freeze. Use that host for the one replay job only when the pin holds. Otherwise keep the replay on a machine that can.

Limitations

This workflow classifies one record. It does not estimate a flake rate, and it does not publish a stability threshold.

It will not catch a weak property that already lived outside the diff. A tautology still passes this table. Review oracles in a separate pass.

Suites that cannot replay a seed cannot emit SCHEDULER. Leave them on PROPERTY or INCOMPLETE until seeding exists.

Do not use the table when the base revision is unpinned. Do not use it when fixture bytes are generated during the candidate run and never digested. Do not use it when one patch may edit the property, the generator, and the freeze ledger together. In those repositories the SCHEDULER row has no honest inputs.

Do not treat a scheduler label as a security waiver. The label says nothing about authorization, data loss, or migration safety. Those changes need a different gate.

The label is also wrong when base and candidate differ in worker count, clock budget, or filesystem. Record those three beside the seed. If they differ, force BUDGET or INCOMPLETE rather than SCHEDULER.

What to put on the merge path

Add classify_failure.py as a required check named failure-class. Fail the check when label is INCOMPLETE, CONTAMINATED, FIXTURE, BUDGET, or PROPERTY.

Allow PASS. Allow a skip only for a triple that already has a non-expired SCHEDULER freeze whose record_sha256 still matches. Keep that file small, reviewed, and counted down by runs rather than by a vague later.