A Freeze Is a Ceiling on Evidence, Not a Floor: Priority Rules for Agent-Patch Test Gates

Three test gates can each return a defensible answer and still leave you with no verdict.

Property checks say fail. The fixture lock says the measurement instrument moved. The flake list says this node is frozen until Thursday. Individually each signal is honest; combined, they invite the worst reflex in CI: rerun until something looks green.

The fix is not a better gate. It is a precedence order, written down before you have a conflict, plus one invariant that survives every future gate you add: no combination of non-clean signals may produce an accept.

Why order has to be fixed in advance

When a gate conflict is resolved after the results are visible, the resolution is chosen by the person who wants the merge. That is selection bias with extra steps, and it is invisible in the log because the log only records the final verdict.

A precedence table moves the choice upstream. You decide once, in a review, with no patch in front of you, that instrument integrity outranks results and that a freeze withholds evidence in both directions. Later you only apply it.

The concrete rules I would start from:

1. Fixture drift or a missing fixture outranks every result. If the fixture set changed, the test did not measure what it was pinned to measure. Hold.
2. A frozen node never contributes evidence — not a pass, not a fail. Freeze removes a node from the sample; it does not vote. Hold.
3. Only an executed, failing property check may reject. not_run is not a pass and never was.
4. Accept requires all three signals clean, at every node. Anything short of that is hold or reject.

Rule 2 is the one people push back on. If a frozen test fails, isn't rejecting the safe move? Safety, yes; accuracy, no. You cannot attribute a failure from a node you have declared unreliable, so rejecting on it teaches the queue that frozen failures are actionable — and the next person un-freezes the test to unblock the merge instead of fixing it.

Step 1: make each gate emit a typed record

Free-text gate output is the reason conflicts get argued instead of computed. Emit a small closed vocabulary per node:

{
  "node": "tests/props/test_ledger.py::test_balance_monotone",
  "property_check": "fail",
  "fixture_lock": "drifted",
  "flake_status": "stable",
  "fixture_manifest": "9f2c...",
  "seed": 8817,
  "freeze_expires": null
}

Note seed sitting next to the manifest hash. A fixture set can hash identically while a seed change reshuffles the generated inputs, which is drift that the hash alone will not catch.

Step 2: a precedence table, not a scoring model

Scores invite tuning. A total order does not.

property_check	fixture_lock	flake_status	verdict	reason code
any	drifted / missing	any	HOLD	fixture_unpinned
any	match	frozen	HOLD	frozen_node
fail	match	stable	REJECT	property_fail
not_run	match	stable	HOLD	property_not_run
pass	match	stable	ACCEPT	clean

The any rows are the precedence. They are also the rows a reviewer should argue about, because everything below them is mechanical.

Step 3: the arbiter, as a pure function

Keep the decision pure so you can test it without running a single property. The version below is illustrative and unexecuted — the record shape is the part worth copying, not the syntax.

# illustrative, not executed
from dataclasses import dataclass
from enum import Enum

class Verdict(str, Enum):
    ACCEPT = "accept"
    HOLD = "hold"
    REJECT = "reject"

@dataclass(frozen=True)
class Signals:
    property_check: str   # "pass" | "fail" | "not_run"
    fixture_lock: str     # "match" | "drifted" | "missing"
    flake_status: str     # "stable" | "frozen"

def arbitrate(s: Signals) -> tuple[Verdict, str]:
    # 1. instrument integrity outranks every result
    if s.fixture_lock != "match":
        return Verdict.HOLD, f"fixture_{s.fixture_lock}"
    # 2. a freeze caps evidence: it withholds, it never grants
    if s.flake_status == "frozen":
        return Verdict.HOLD, "frozen_node"
    # 3. only an executed failure may reject
    if s.property_check == "fail":
        return Verdict.REJECT, "property_fail"
    if s.property_check == "not_run":
        return Verdict.HOLD, "property_not_run"
    return Verdict.ACCEPT, "clean"

Aggregation has one trap. Reject outranks hold, hold outranks accept — but the empty sample must not fall through to accept:

# illustrative, not executed
def aggregate(per_node: list[tuple[Verdict, str]]) -> Verdict:
    if not per_node:
        return Verdict.HOLD          # no evidence is not a pass
    verdicts = {v for v, _ in per_node}
    if Verdict.REJECT in verdicts:
        return Verdict.REJECT
    if Verdict.HOLD in verdicts:
        return Verdict.HOLD
    return Verdict.ACCEPT

That four-line guard catches the most common silent pass in agent-patch queues: a test-selection step that selected nothing, therefore reported no failures.

Step 4: pin fixtures so drift is a distinct verdict

Drift must never arrive as a test failure, because a failure is actionable and drift is not. Pin the fixture tree and check it as its own step, before the properties run:

# illustrative
find tests/fixtures -type f -print0 | sort -z | xargs -0 sha256sum > tests/fixtures.sha256
sha256sum -c tests/fixtures.sha256

Two operational notes. Regenerating the manifest is a reviewed commit, never a side effect of the agent patch. And when a fixture edit is genuinely part of the patch, the manifest change and the code change land in the same commit so the drift is visible as an intentional act rather than absorbed into a green run.

Step 5: scope the freeze and let it expire mechanically

A freeze without an owner and an expiry date is a permanent exemption wearing a temporary label. Record three fields per frozen node: reason, owner, expiry. The arbiter reads freeze_expires; past that timestamp the node returns to stable and starts blocking again, whether or not anyone remembered.

Scope matters as much as duration. Freeze the narrowest node that reproduces the nondeterminism — a single property, not the file, not the directory. A frozen directory takes real evidence offline along with the flaky assertion.

Step 6: verify the arbiter itself

The decision table is a specification, so test it as one. The invariant worth encoding is that accept is reachable from exactly one combination:

# illustrative, not executed
import itertools
import pytest
from arbiter import Signals, arbitrate, Verdict

COMBOS = itertools.product(
    ["pass", "fail", "not_run"],
    ["match", "drifted", "missing"],
    ["stable", "frozen"],
)

@pytest.mark.parametrize("prop,fixture,flake", list(COMBOS))
def test_accept_requires_all_three_clean(prop, fixture, flake):
    verdict, _ = arbitrate(Signals(prop, fixture, flake))
    if verdict is Verdict.ACCEPT:
        assert (prop, fixture, flake) == ("pass", "match", "stable")

Eighteen cases, one assertion each. If someone later adds a fourth signal and wires it in as an override, this test fails the moment the new signal can be bypassed — which is the point. You are not testing that the gates work; you are testing that nothing can quietly outrank them.

Where the agent side fits

Patch generation and patch verification want different machines, and mixing them is how nondeterminism enters the harness. Keep generation in one place and arbitration in a place whose load you control.

MonkeyCode offers free model access and a free server option (operator-supplied availability). Reasonable split: use free model access to produce candidate patches, and use the free server option for the CPU-bound half of the arbitration harness — fixture hashing over a large tree, plus property runs that are insensitive to wall-clock timing. Keep anything timing-sensitive on hardware you control, and re-check your assumptions rather than assuming a shared environment is quiet.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations, and who should skip this

A precedence table is worth its review cost only if you have gate conflicts to resolve. If your suite is small enough that a human reads every failure, this is ceremony — you already are the arbiter, and the table only adds a file to maintain.

Skip it too if you have no flake history yet. Freezing nodes speculatively removes evidence you never had a reason to distrust, and rule 2 will then hold patches that a plain run would have cleared honestly.

The real cost is cultural, not technical. Rule 2 forces a choice: fix the frozen test, or accept that its node contributes nothing until you do. Teams that treat freezes as a way to keep the pipeline green will find the arbiter blocks them, correctly, and they will want to edit the table instead of the test. Write the table in a review, link it from the CI config, and make changes to it a separate pull request with its own reviewer.

If you map your own gate outputs onto this table, the interesting output is not the verdict count — it is which reason codes accumulate. That distribution tells you whether your next investment is fixture pinning or flake removal.