When Three Gates Disagree: A Precedence Router for Agent-Patch Evidence

Most agent-patch pipelines already have more than one gate: a property check, a fixture suite, and some handling for tests that are currently frozen as flaky. The failure mode is not that a gate is missing. It is that nobody decided, in advance, what a disagreement means.

Conclusion first: adjudicate the verdicts, do not average them. Three gates over four states produce 64 combinations, and exactly four of them permit a merge. Encode that ladder as data, make disagreement a hard stop, and the review conversation stops being about opinions.

1. A freeze does not make a test green

This is the part that quietly corrupts most aggregations. When you freeze a flaky test, you have not resolved it. You have declared its verdict inadmissible for now.

Those are different states, and they must stay different:

• pass — the check ran and held.
• fail — the check ran and a counterexample exists.
• unknown — the check was frozen, skipped, or errored without a verdict.
• missing — the gate never ran at all.

If you map unknown onto pass, every frozen test becomes silent consent. If you map it onto fail, a single flake blocks a release and the team starts disabling gates. Neither is acceptable. unknown has to survive into the adjudication step as its own thing.

2. The artifact: an evidence document and a router

Each gate writes a small JSON document. Nothing else is allowed to influence the decision.

{
  "patch_id": "pr-1841",
  "property_check": "fail",
  "fixture_check": "pass",
  "flaky_freeze": "unknown"
}

The router is deliberately boring Python. No model, no heuristics, no scoring.

#!/usr/bin/env python3
"""gate_router.py - one disposition from three independent patch gates."""
import json
import sys
from dataclasses import dataclass
from enum import Enum


class V(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"   # frozen, skipped, or errored: no verdict
    MISSING = "missing"   # gate did not run


class Disposition(str, Enum):
    MERGE_OK = "MERGE_OK"
    BLOCK = "BLOCK"
    ESCALATE = "ESCALATE"


@dataclass(frozen=True)
class Evidence:
    prop: V
    fixture: V
    frozen: V

    @classmethod
    def load(cls, path):
        raw = json.load(open(path))
        return cls(V(raw["property_check"]),
                   V(raw["fixture_check"]),
                   V(raw["flaky_freeze"]))

    def verdicts(self):
        return (self.prop, self.fixture, self.frozen)


def route(ev):
    """Ordered rules. First match wins. The order is the policy."""
    verdicts = ev.verdicts()
    if V.MISSING in verdicts:
        return Disposition.ESCALATE, "evidence-gap"
    if V.FAIL in verdicts:
        return Disposition.BLOCK, attribute(ev)
    if V.UNKNOWN in verdicts:
        return Disposition.ESCALATE, "inadmissible-evidence"
    return Disposition.MERGE_OK, "all-gates-held"


def attribute(ev):
    """Only called when at least one gate failed. Names where to look first."""
    if ev.prop is V.FAIL and ev.fixture is V.FAIL:
        return "behavior-and-invariant"
    if ev.fixture is V.FAIL:
        return "concrete-behavior"
    return "invariant-or-over-specified-property"


EXIT = {Disposition.MERGE_OK: 0, Disposition.BLOCK: 1, Disposition.ESCALATE: 2}


if __name__ == "__main__":
    ev = Evidence.load(sys.argv[2])
    disp, why = route(ev)
    print(f"{disp.value} attribution={why}")
    sys.exit(EXIT[disp])

python gate_router.py --evidence evidence.json
# BLOCK attribution=invariant-or-over-specified-property
echo $?   # 1

The exit code is the API. A blocked patch is a 1, an escalation is a 2, and a 2 must never be treated as a soft pass by the merge bot.

3. The decision table

Read this as the whole contract. Every row is a reachable state.

property	fixture	frozen tests	disposition	attribution
pass	pass	pass	MERGE_OK	all gates held
fail	fail	*	BLOCK	behavior + invariant
fail	pass	*	BLOCK	invariant or over-specified property
pass	fail	*	BLOCK	concrete behavior
*	*	unknown	ESCALATE	inadmissible evidence
*	*	missing	ESCALATE	evidence gap

Two design choices are worth defending. First, fail is absorbing: one admitted counterexample overrides any number of passes, so there is no vote to win. Second, missing outranks fail in the rule order, because a patch that blocked on a failure it can explain is safer than one that would have merged had the un-run gate agreed with it.

4. Steps to adopt it without re-platforming

1. Make each gate emit the four states. The common bug is a boolean. A fixture runner that returns False for "timed out after three attempts" is lying, and that lie propagates. Map timeouts, skips, and collected-zero-tests cases to unknown or missing explicitly.
2. Freeze by patch scope, not by test name. A frozen test only earns unknown for a patch that touches the code it covers. If it does not cover the diff, it stays pass-irrelevant, not unknown, and you do not escalate on it.
3. Put the router in front of the merge button. Run it as the last step of your required check, after all three gates have written their documents. Do not let the router re-run anything; it reads only the artifacts.
4. Log every non-MERGE_OK disposition with the eventual fix commit. This is the only data that lets you calibrate attribution later, and it costs nothing at decision time.
5. Review the ladder, not the results. Quarterly, look at the escalation rate. If 40% of patches escalate on inadmissible-evidence, your freeze set is too broad, not your router.

5. Test the router like a gate

The router is the thing deciding your merges, so it gets a table-driven test. This runs without any fixtures or network.

# test_gate_router.py
import pytest
from gate_router import Evidence, V, Disposition, route

CASES = [
    ((V.PASS, V.PASS, V.PASS), Disposition.MERGE_OK),
    ((V.FAIL, V.FAIL, V.PASS), Disposition.BLOCK),
    ((V.PASS, V.FAIL, V.PASS), Disposition.BLOCK),
    ((V.FAIL, V.PASS, V.PASS), Disposition.BLOCK),
    ((V.PASS, V.PASS, V.UNKNOWN), Disposition.ESCALATE),
    ((V.PASS, V.PASS, V.MISSING), Disposition.ESCALATE),
    ((V.FAIL, V.PASS, V.MISSING), Disposition.ESCALATE),
]

@pytest.mark.parametrize("verdicts,expected", CASES)
def test_route(verdicts, expected):
    ev = Evidence(*verdicts)
    assert route(ev)[0] is expected

Note case seven. A patch with a real counterexample and a gate that never ran escalates rather than blocking, because the missing evidence might have been the one that explained the failure. That is a judgment call, and the test is where you argue about it, once.

6. Calibrate attribution from your own history

I am not going to give you a hit rate for invariant-or-over-specified-property. Yours will differ from mine, and a borrowed number would make the table look more empirical than it is.

Measure it instead. For every patch routed to BLOCK, record three fields: the attribution string, the files touched by the commit that eventually fixed it, and whether the fix changed production code or the check itself. Then compute, per attribution class, the share of fixes that landed where the attribution pointed. If a class is consistently wrong, the ladder order is probably fine and the attribution function is what needs work.

That is a ten-line query against a log you are already writing, and it turns a subjective table into something you can defend in a retro.

7. Where a free model tier and a free server actually help

Two steps in this workflow benefit from hosted capacity, and both are optional.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises free model access and a free server option; confirm the current terms and limits on the vendor's own documentation before you depend on them, since availability claims age quickly.

The model helps in exactly one place: when property_check fails and the counterexample is unreadable, paste the failing input, the diff, and the property source into a session and ask for a plain-language summary plus candidate invariants you did not state. Treat every candidate as a proposal. It only becomes real after a human turns it into an executable check that fails on the current patch and passes on the parent commit. A model that writes properties without that two-commit proof is just generating more tautologies.

The server matters when you have no always-on runner. The router is stateless and needs a Python interpreter and three JSON files, so a small hosted box that accepts POST /route with the evidence document is enough to unblock teams whose CI minutes are the bottleneck. If you already run required checks in CI, put the router there and skip this entirely.

8. Limitations and who should skip this

• Small single-gate repos. If you have one trustworthy test suite, a precedence ladder is overhead. Use it when you have at least two independent evidence sources that can contradict each other.
• Aggressive freeze sets. If most of your suite is frozen, the router will escalate almost everything. Fix the freeze scope first, or you will conclude the router is too strict when the real problem is upstream.
• Teams that want a score. This returns a disposition, not a confidence number. If your process needs a percentage to argue about, this will frustrate you.
• Model-generated properties without a two-commit proof. Unproven invariants inflate fail rates for reasons that have nothing to do with the patch.

The ladder is not a smarter gate. It is a written answer to the question your reviewers currently answer differently every time: when the evidence disagrees, who wins. Write the answer down once, test it, and let the exit code carry it.

If you run this against your own patch history, the disagreement table is the interesting artifact — I would like to see which attribution class turns out to be wrong most often.