An Extra Call Keeps the Flake Freeze Shut

Matching output is not a pass. Not when the patch spent an extra call, sleep, or read to produce that output. A flake freeze stays shut until a shrink yields one stable fixture and a bilateral replay fails the same assertion on both revisions, with every accepted effect budget still holding. A retry that only repairs the returned dict meets none of those tests.

Agent patches blur that line in a few edits. The diff adds a loop or a second read, the happy-path assertion turns green, and the red budget check is filed as a flake. That filing is a classification error.

Flakes are unstable observations of one assertion. Regressions are stable disagreements, including disagreements about how often a dependency was touched. This workflow keeps the two classes in separate lanes.

The steps below are a proposal. Nothing in this draft was executed, and no pass rate is claimed. Thresholds are parameters you fill from your own logs.

Separate the lanes before you name the failure

The property lane may reject a patch. It owns invariants, call budgets, and oracle identity. A failure here blocks merge, even when the returned value looks familiar.

The flake lane may only queue a case for later observation. Entry requires the eligible row of the decision table, an expiry, and a fixture digest. Eligibility is not merge approval, and it does not cover other failing checks in the same diff.

The patch this gate is built to catch

Consider a helper an agent might edit, normalize_event(payload, now). The base revision calls now once and returns a dict. A weak patch catches a parse miss, calls now again, and returns that same dict.

Output equality holds. The call budget does not. An output-only flake freeze would hide the extra call. The gate below is built so that hide fails.

1. Pin revisions before anyone drafts a test

Record base_rev, patch_rev, the test command, and the runner image digest. A floating image tag is not evidence. Write those four values into a run log before a model sees the diff.

Do this even when the runner is a laptop. The pin is what makes the later replay comparable, not the brand of the machine.

2. Accept properties only after a mutant dies

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access matters here only as a source of candidate property text. The free server option matters only as a possible host for shrink and replay.

This article does not name a model, a quota, a machine size, or a duration. Copy those fields from the console on the day you run, or leave them blank and use another runner you already operate.

Treat every drafted property as untrusted text. Accept it only when three local checks pass.

1. The statement names an input and a relation on an output or an effect.
2. A hand-written mutant that is still in the tree makes the statement fail.
3. The statement is not a tautology: no constant assert, and no comparison of a function with its own fresh return.

For this helper, the budget property is the mutant killer. The clock is a counter. One public call must increment it once.

# Proposal. Not executed for this draft.
def test_clock_budget(normalize_event, clock):
    payload = {"id": "evt-1", "ts": "1970-01-01T00:00:00Z"}
    normalize_event(payload, now=clock)
    assert clock.calls == 1

If that assertion fails on the patch, the lane is property regression. Do not add a freeze file in that same change.

3. Shrink until the minimum repeats

Capture the payload that broke an accepted property. Reduce it with a deterministic function: drop optional fields, shorten strings, and replace timestamps with one fixture epoch. Run that reducer at least twice. Call the shrink stable only when every run produces the same digest.

required_identical is your parameter. This draft sets no recommended constant. If you have logs, count how often a second shrink changed the minimum across recent failures, and set the parameter above that count. If you have no logs, use 2 and mark the lane provisional.

# Proposal. Not executed for this draft.
import hashlib
import json

def digest(payload):
    raw = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode()).hexdigest()

def shrink_event(payload):
    event_id = payload.get("id", "evt")
    return {"id": event_id, "ts": "1970-01-01T00:00:00Z"}

def shrink_report(payload, runs):
    minima = [digest(shrink_event(payload)) for _ in range(runs)]
    last = minima[-1]
    identical = sum(1 for item in minima if item == last)
    return {
        "minimal_digest": last,
        "attempts": runs,
        "identical_minima": identical,
        "stable": identical == runs and runs >= 2,
    }

The reducer is deliberately small so the stability rule stays visible. Swap in the reducer for your own type. Keep the report keys. Store the unshrunk payload beside the minimum for one review cycle, because shrinking can collapse two bugs into one fixture.

4. Replay the minimum on both revisions

Check out the pinned revisions in two worktrees. Copy the same fixture into both. Run the same command. Record assertion id, pass or fail, fixture digest, and clock-call count for each side.

# Proposal commands. Substitute your revisions and image digest.
git worktree add /tmp/base "$BASE_REV"
git worktree add /tmp/patch "$PATCH_REV"
cp fixtures/min.json /tmp/base/fixtures/min.json
cp fixtures/min.json /tmp/patch/fixtures/min.json
docker run --rm -v /tmp/base:/src -w /src "$IMAGE_DIGEST" pytest -q tests/test_budget.py
docker run --rm -v /tmp/patch:/src -w /src "$IMAGE_DIGEST" pytest -q tests/test_budget.py

Use a free server for that pair only if the console still lists the option for the account that will run the job. Pin IMAGE_DIGEST from your registry. Do not infer isolation, persistence, or a rental lifetime from the label free.

If the option is absent, run the same two commands on a runner you already operate. The decision table does not change.

5. Apply the decision table

Read rows from top to bottom and stop at the first match. Call inequality is judged only after the oracle and the shrink are acceptable. A changed oracle invalidates both sides, so it never becomes a freeze.

Base	Patch	Shrink	Same assertion	Calls	Lane
any	any	any	any	oracle file changed	reject the run
any	any	unstable	any	any	rerun shrink; do not classify
any	any	stable	any	not equal	property regression; freeze denied
pass	fail	stable	yes	equal	property regression; freeze denied
fail	fail	stable	no	equal	different bug; freeze denied
fail	fail	stable	yes	equal	flake queue eligible
fail	pass	stable	yes	equal	fix candidate; other properties still apply
pass	pass	stable	yes	equal	no freeze

Row 3 is the retry disguise. The dict can match, and the patch is still a regression, because the effect budget moved. Row 6 is the only path into the flake queue. Equal calls are required there. An extra sleep that preserves the return value still dies on row 3.

# Proposal. Not executed for this draft.
def classify(base_passed, patch_passed, same_assertion, shrink_stable,
             base_calls, patch_calls, oracle_changed):
    if oracle_changed:
        return "reject_run"
    if not shrink_stable:
        return "rerun_shrink"
    if patch_calls != base_calls:
        return "property_regression"
    if base_passed and not patch_passed:
        return "property_regression"
    if (not base_passed) and (not patch_passed) and not same_assertion:
        return "different_bug"
    if (not base_passed) and (not patch_passed) and same_assertion:
        return "flake_queue_eligible"
    if (not base_passed) and patch_passed and same_assertion:
        return "fix_candidate"
    if base_passed and patch_passed:
        return "no_freeze"
    return "property_regression"

def test_classifier_examples():
    assert classify(True, True, True, True, 1, 2, False) == "property_regression"
    assert classify(False, False, True, True, 1, 1, False) == "flake_queue_eligible"
    assert classify(False, False, True, False, 1, 1, False) == "rerun_shrink"
    assert classify(True, False, True, True, 1, 1, False) == "property_regression"
    assert classify(False, True, True, True, 1, 1, False) == "fix_candidate"
    assert classify(True, True, True, True, 1, 1, True) == "reject_run"
    assert classify(True, True, True, True, 1, 1, False) == "no_freeze"

test_classifier_examples checks the rule, not a product. A green local run of that function does not mean your reducer is faithful. It only means the table and the function agree.

python -m pytest -q tests/test_classify.py

6. Write a freeze entry only from the eligible row

The entry has five fields: fixture digest, assertion id, base revision, patch revision, and an expiry you choose. Drop the entry if any field is missing. Do not let a model fill the expiry.

The entry waives nothing else in the patch. It does not renew when a draft is regenerated. When the expiry passes, delete the entry or replace it with a fresh bilateral replay. A missing follow-up is an open risk, not a quiet pass.

# Proposal schema. Not a recorded run.
base_rev: "<git sha>"
patch_rev: "<git sha>"
image_digest: "sha256:<pin>"
fixture_digest: "sha256:<from digest()>"
assertion_id: "test_clock_budget"
lane: "flake_queue_eligible"
expiry: "<ISO-8601 you choose>"
console_model_id: "<paste or leave blank>"
console_server_offer: "<paste or leave blank>"

Counts worth keeping

For the next 20 agent patches, record four integers from your own runs. Do not borrow figures. This draft has none to offer.

1. Properties drafted, versus properties that killed a mutant.
2. Failures whose second shrink changed the minimal digest.
3. Cases where output matched and call count did not.
4. Freeze entries still present after expiry.

Count 3 is the health check for output-only suites. If it is above zero, matching dicts are not evidence for this helper. Count 2 tells you whether required_identical is too low. Count 4 tells you whether the queue is becoming a side door.

Plot the four counts as a simple weekly tally. A rising count 3 with a rising count 4 means reviewers are parking call-budget regressions in the flake queue. Tighten acceptance, not the retry loop inside the patch.

Limitations

Shrinkers throw information away. Two defects can share one minimal payload, so the table can under-count bugs. That is why the unshrunk payload stays attached for a review cycle.

The budget check needs an injectable clock or counter. If the patch builds a private sleeper inside a closure you cannot observe, the gate will not see the wait. Instrument that seam first. Until you can, do not claim a call-budget property.

Bilateral replay is the wrong tool when the patch changes the contract on purpose. The oracle file then differs, and the table says reject the run. A reviewer writes a new property and repeats the mutant check. Updating a snapshot inside the agent diff is not that review.

Free-model drafts can be tautologies with serious names. Step 2 is the countermeasure. Skip it, and later lanes will certify whatever sentence was drafted.

A disposable server does not improve a bad fixture. Byte-unstable inputs, live clocks, and unlocked network calls make both the digest and the bilateral compare meaningless. Freeze the fixture before you freeze the test.

Equal call counts can still hide a behavior change if the counter is coarse. A clock that increments once per public call will miss a nested retry that reuses the same tick. Split the counter by dependency name before you trust row 6.

Who should skip this gate

Skip it when no base revision exists. One side cannot separate a flake from a regression. Wait until the parent commit is identifiable.

Skip it for browser tours that cannot pin a fixture digest. The shrink rule needs byte-stable input. A live backend does not provide that.

Skip it as an incident hotfix when waiting would extend an outage. Write down the skipped step, the revision, and a follow-up date. Do not invent a freeze entry afterward and treat it as evidence.

Skip it when every property compares a function to itself. The classifier will return no_freeze. That answer is consistent, and it does not protect the patch.

Closing check

Feed the classifier a mutant that adds one clock call and returns the original dict. The expected lane is property_regression. If you observe flake_queue_eligible or no_freeze, the gate is wrong, and the patch stays closed.

If MonkeyCode supplies the draft or the runner, confirm on that account's console that free model access and the free server option are still listed, then paste those console facts into the run log. The lane rule stands without either offer.