Five Budget Codes Before a Flake Freeze on an Agent Patch

An agent patch does not earn a flake freeze because one test went red. It earns a freeze only when three checks agree: the parent fixture digest matches, a property oracle still holds on that digest, and a fixed rerun budget comes back mixed. Every other combination is a regression, a changed input, or an unfinished measurement.

This is a protocol, not a case study. The snippets are unexecuted examples. They do not report a pass rate from a production gate, and they do not ask you to treat one red cell as evidence of flakiness.

Keep three records apart

A property result says whether an invariant holds on a known input. A fixture digest says which bytes that input was. A flake observation says whether one command stayed stable across a seed list you committed in advance.

Those are different questions. Store them in different files. If one document holds the digest, the oracle output, and the freeze decision, a patch can edit the conclusion while it edits the evidence.

The merge job should recompute a single budget code from the three files. A reviewer should be able to repeat that arithmetic without opening a chat transcript.

Five codes, no soft bypass

Use five codes. A sixth value such as review-later becomes a bypass the moment a queue is busy.

Code	Parent digest	Property on parent fixture	Rerun budget	Merge meaning
FIXTURE_DRIFT	mismatch	not scored	not started	Reject. Score nothing else.
PROPERTY_MISS	match	fail	not started	Reject. Leave the freeze file untouched.
STABLE_FAIL	match	pass	0 passes in N	Reject. The miss is stable.
STABLE_PASS	match	pass	N passes in N	Do not write a freeze row.
FLAKE_CANDIDATE	match	pass	1 to N-1 passes	One row, one test id.

INSUFFICIENT_BUDGET is a process error, not a product code. An early stop leaves the ledger unchanged and the merge closed.

Step 1. Canonicalize, then hash

Hash a fixture only after canonical encoding. Raw key order and float spelling create false drift. The same instability can also hide a real field edit inside formatting noise.

The helper below is a proposal. Change the float policy to match your contract before you trust it. If key order is meaningful, skip sort_keys, hash the raw file bytes, and treat any rewrite as drift.

import hashlib
import json
from pathlib import Path

def canonical_bytes(payload: dict) -> bytes:
    return json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=True,
        allow_nan=False,
    ).encode("utf-8")

def digest(payload: dict) -> str:
    return hashlib.sha256(canonical_bytes(payload)).hexdigest()

def load_digest(path: Path) -> str:
    payload = json.loads(path.read_text(encoding="utf-8"))
    return digest(payload)

Keep the parent digest beside the fixture. A digest written inside the document changes the document, which changes the digest.

git show HEAD:fixtures/order.json > /tmp/parent-order.json
python3 scripts/fixture_digest.py /tmp/parent-order.json > /tmp/parent.sha256
python3 scripts/fixture_digest.py fixtures/order.json > /tmp/head.sha256
cmp /tmp/parent.sha256 /tmp/head.sha256

If cmp exits non-zero, stop at FIXTURE_DRIFT. Do not launch the flaky test to see whether it still fails. The input is no longer the input under measurement. Use a sorted pretty-print only as a review aid.

python3 -m json.tool --sort-keys /tmp/parent-order.json > /tmp/parent.canon.json
python3 -m json.tool --sort-keys fixtures/order.json > /tmp/head.canon.json
diff -u /tmp/parent.canon.json /tmp/head.canon.json

The gate still hashes canonical_bytes. The unified diff is for humans. Do not parse the diff to decide the code.

Step 2. Score the property on the parent fixture

The oracle is a pure function of fixture plus output. It is not the integration test you suspect of flaking. It should not read the clock, the network, or a shared account.

If the oracle fails on the parent fixture plus the agent output, emit PROPERTY_MISS and stop. A later mixed rerun cannot repair a broken invariant. The freeze file is not a second score column for that failure.

def property_holds(fixture: dict, output: dict) -> bool:
    # Proposal only. Replace this with the invariant you ship.
    total = sum(line["qty"] for line in fixture["lines"])
    if "qty" not in output or "status" not in output:
        return False
    return output["qty"] == total and output["status"] in {"ok", "held"}

The body is a stand-in. A production oracle also needs a rule for extra keys and for wrong types. A chain of missing-key defaults can turn an absent field into a false pass, so the example checks required keys first.

Score the parent fixture even when the branch edited fixtures/order.json. The branch file describes a new input. A freeze is a claim about the old input. Mixing them produces a code you cannot reproduce on the parent commit.

Step 3. Spend a fixed seed budget

Choose N before the run, and commit the seeds. Eight seeds is a pre-merge proposal, not a tuned optimum. It is large enough to refuse a single red cell, and small enough to script.

If eight runs are too expensive for the blocking lane, move them to the offload job in Step 5. Do not shrink N to 1 and keep the same codes. One sample cannot support this table.

seeds: [11, 23, 47, 71, 97, 131, 173, 211]

Runtime random seeds make the next patch unable to recompute the code. An inner retry on a failed seed spends the budget twice and biases the result toward FLAKE_CANDIDATE.

python3 scripts/budget_run.py \
  --test tests/test_order_hold.py::test_hold_window \
  --seeds 11,23,47,71,97,131,173,211 \
  --out artifacts/budget.json

Classification uses counts, not log adjectives.

def budget_code(passes: int, n: int) -> str:
    if n < 8 or passes < 0 or passes > n:
        return "INSUFFICIENT_BUDGET"
    if passes == n:
        return "STABLE_PASS"
    if passes == 0:
        return "STABLE_FAIL"
    return "FLAKE_CANDIDATE"

STABLE_FAIL is still a reject after a green property check. The encoded invariant held, and the integration test failed on every seed. That usually means the oracle is narrower than the test, or the product change is consistently wrong.

Widen the oracle, or fix the product code. Do not write a freeze row to hide a stable miss. STABLE_PASS means this budget does not support a flake claim. Remove any existing row for that test id instead of extending it.

Step 4. Write a row only for a candidate

A valid row names one test id, the parent digest, the seed list, the pass count, an expiry you chose, and the commit the observation ran against. It does not name a directory. It does not copy a freeze file from another branch.

{
  "test_id": "tests/test_order_hold.py::test_hold_window",
  "parent_digest": "replace-with-sha256",
  "seeds": [11, 23, 47, 71, 97, 131, 173, 211],
  "passes": 5,
  "budget": 8,
  "code": "FLAKE_CANDIDATE",
  "expires_on": "2026-10-09",
  "opened_against": "replace-with-parent-sha"
}

The date 2026-10-09 is a sample field, not a platform rule. Fourteen days is a window you commit in your own repo. When the date passes, the row is absent, and the next patch spends a new budget.

The expiry does not carry evidence forward. Only a new budget.json does. Reject the row when the same diff also changes the fixture, the oracle, or the test body. Those edits make the observation describe the new check, not the behavior you intended to hold still.

Refresh a fixture in its own commit. Do not attach a freeze row to that commit.

git diff --name-only origin/main...HEAD

If that list contains both fixtures/order.json and flake-ledger/order_hold.json, fail closed. Path coupling is a cheap guard. You still need the digest compare when the fixture path looks untouched, because a generator can rewrite bytes without a review-friendly diff.

Read the counts as arithmetic

The following three traces are illustrations of the table. They are not measurements from a live suite.

Trace A: digest compare exits 0, property_holds returns true, passes is 5, and budget is 8. The code is FLAKE_CANDIDATE. Five passes is not mostly green. It is a mixed result inside a budget fixed before the run.

Trace B: same digest and property result, and passes is 0. The code is STABLE_FAIL. A row left from an older commit does not travel with the new patch. The new patch spends a new budget.

Trace C: digest compare exits 1. Stop. Ignore later fields. A green property score on the branch fixture must not be written into the parent column.

Step 5. Offload the reruns, keep the verdict local

The blocking lane can stay short. The eight-seed job can run elsewhere if it uploads budget.json and both digest files. CI recomputes the code. It does not accept a remote status string as the merge decision.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free server option is a practical place for that observation job when you want the reruns off the main runner. MonkeyCode's free model access can draft a short note that maps log lines onto one of the five codes. Treat both as optional capacity. Nothing here assumes a quota, a hardware shape, a model id, or a permanent tier.

The note is a comment. The Python classifier is the gate. If they disagree, keep the classifier and leave the note on the review. A generated line that says the failure looks unstable must not create a ledger row.

If you already store a parent fixture and a pure property check, run the budget script once on a known unstable test and read the code you get before you add a freeze file.

Who should skip this

Skip the protocol when you have no parent fixture to hash. Skip it when the oracle reads the network, the clock, or a shared staging account. Skip it when the pipeline can afford only one sample. One sample cannot separate a flake from a regression.

Skip it when reviewers are expected to override PROPERTY_MISS or STABLE_FAIL from a chat summary. The table is empty once a side path ignores it.

Canonical JSON hides order-sensitive bugs. Hash raw bytes when order or whitespace is part of the contract. A fixed seed list misses failures that need other seeds, a loaded host, or a different clock.

The budget detects instability inside the list you committed. It does not prove the test is sound, and it does not replace reading the agent diff. No historical pass rate belongs in the ledger. The only figures that should decide the code are the ones this job just wrote: digest match or mismatch, oracle pass or fail, and passes out of N.