Which Gate Should Run First? Ordering Agent-Patch Checks by Yield per Second

Gate order is a cost decision, not a taste decision. If your most expensive check runs first, you pay full-suite prices for diffs that a scope read would have routed differently in under a second. This post treats the ordering itself as the artifact: a three-tier ladder, a verdict model that distinguishes failure from inconclusive, and an evidence rule for freezing flaky tests instead of freezing them by reflex.

It does not re-argue whether fixture edits need their own gate — I have written about that separately. Here the question is narrower: given several checks you already trust, which one do you run, in what order, and how do you know the order is working?

The ordering mistake that hides in plain sight

A test suite is not a single check. It is a bundle of checks with wildly different costs and wildly different failure semantics. When you hand an agent a failing test and ask for a patch, you get a diff you must classify, and classification is cheapest when it happens before execution.

The failure mode is subtler than "slow CI". If tier 2 (the full suite with pinned fixtures) runs first and passes, you have learned almost nothing when the patch also touched the fixtures — the oracle moved with the code. If it fails, you cannot tell whether the patch is wrong or the fixture changed underneath it. You paid the most and got the least information.

1. Define tiers by cost, then state what each tier cannot answer

Tier	Question it answers	Cost order	Non-zero exit means
t0-scope	Does this diff touch test-adjacent paths?	~0.5 s	Not a bug. It routes, it never "catches".
t1-properties	Do the deterministic properties still hold?	seconds	Caught, if exit code is exactly 1
t2-fixtures	Does the pinned-fixture suite agree?	minutes	Caught, if exit code is exactly 1

Two things fall out of that table. First, t0-scope produces no catch, so it has no yield — its only value is seconds it saves downstream. Do not put it in a catch-rate report and then wonder why it scores zero. Second, only exit code 1 is a test failure. Everything else is a different verdict.

2. Read the diff before you execute anything

# gate_ladder.py (part 1)
from __future__ import annotations
import json, subprocess, time
from dataclasses import dataclass, asdict
from pathlib import Path

SCOPE_MARKERS = ("conftest.py", "pytest.ini", "tox.ini", "fixtures/", "tests/")

def changed_files(repo: Path, base: str, head: str) -> list[str]:
    out = subprocess.run(
        ["git", "-C", str(repo), "diff", "--name-only", f"{base}...{head}"],
        check=True, capture_output=True, text=True,
    ).stdout
    return [line for line in out.splitlines() if line.strip()]

def is_test_adjacent(path: str) -> bool:
    return any(marker in path for marker in SCOPE_MARKERS) or Path(path).name.startswith("test_")

The routing rule is the whole point of tier 0: if any changed file is test-adjacent, t1-properties cannot vouch for the patch. A property check that passes after the fixture was edited is evidence about the edited fixture. So a scope hit skips tier 1 and goes straight to tier 2, where the contract check lives.

3. Give "inconclusive" its own verdict

This is the cheap bug that makes ladders lie. pytest returns 1 for test failures, but also 2 for interruption, 3 for internal error, 4 for usage error, and 5 for "no tests collected". Collapsing all non-zero codes into "caught" inflates your catch rate with infrastructure noise, and collapsing them into "pass" is worse.

# gate_ladder.py (part 2)
@dataclass
class Gate:
    name: str
    cmd: list[str]
    timeout_s: float

TIERS = {
    "t1-properties": Gate("t1-properties", ["pytest", "-q", "tests/properties", "-x", "--timeout=60"], 120),
    "t2-fixtures":   Gate("t2-fixtures",   ["pytest", "-q", "-p", "no:randomly", "--timeout=180"], 900),
}

def run_gate(gate: Gate, cwd: Path) -> tuple[str, float]:
    start = time.monotonic()
    try:
        rc = subprocess.run(gate.cmd, cwd=cwd, capture_output=True, text=True,
                            timeout=gate.timeout_s).returncode
    except subprocess.TimeoutExpired:
        rc = None
    seconds = round(time.monotonic() - start, 3)
    if rc == 0:
        return "pass", seconds
    if rc == 1:
        return "caught", seconds
    return "error", seconds        # 2, 3, 4, 5 and timeouts are not catches

error must short-circuit the ladder the same way caught does, but it lands in a different column. A tier that errors tells you the harness is broken, not the patch.

4. Write one ledger row per attempt

# gate_ladder.py (part 3)
def run_ladder(repo: Path, base: str, head: str, ledger: Path) -> None:
    files = changed_files(repo, base, head)
    plan = ["t2-fixtures"] if any(map(is_test_adjacent, files)) else ["t1-properties", "t2-fixtures"]
    for name in plan:
        verdict, seconds = run_gate(TIERS[name], repo)
        ledger.open("a").write(json.dumps({
            "patch": head, "tier": name, "verdict": verdict,
            "seconds": seconds, "files": files,
        }) + "\n")
        if verdict in {"caught", "error"}:
            return

Because the loop returns on the first non-pass, a row exists only for tiers the patch actually reached. caught therefore means "this tier was the first to reject", which is exactly the quantity you want to divide by.

5. Report yield, not pass rate

python gate_ladder.py --repo . --base origin/main --head HEAD --ledger ledger.jsonl
python gate_ladder.py --report ledger.jsonl

The report prints tier, runs, caught, errors, median_s, and you compute yield as caught / total_seconds per tier. I am deliberately not printing a filled-in table here. Any number I typed would be a number I did not measure on your codebase, and the ordering you should adopt depends almost entirely on your suite's fixed cost — whether tier 2 is 40 seconds or 40 minutes changes the answer.

The readable signal is the ratio between tiers. If tier 1 has a decent catch count at a low median cost and tier 2 catches almost nothing new, tier 2 is buying you a contract check, not bug detection — keep it, but stop pretending it earns its minutes. If tier 2 catches nearly everything tier 1 catches and more, tier 1 is decorative and should be deleted rather than reordered.

6. Freeze a flaky test only with evidence

Freezing is the most overused response to flakiness, because it is the only one that takes ten seconds. The problem: freezing preserves a test, and some tests deserve deletion instead. A test that flips between outcomes and has never failed on a seeded bug is not flaky in a useful way — it is noise with a history.

def freeze_eligible(outcomes: list[bool], caught_seeded_bug: bool,
                    min_runs: int = 20, min_flip_rate: float = 0.02) -> tuple[bool, str]:
    if len(outcomes) < min_runs:
        return False, "insufficient history; keep observing"
    flips = sum(1 for a, b in zip(outcomes, outcomes[1:]) if a != b)
    flip_rate = flips / (len(outcomes) - 1)
    if flip_rate < min_flip_rate:
        return False, f"not actually flaky ({flip_rate:.3f})"
    if not caught_seeded_bug:
        return False, "no sensitivity evidence; rewrite or delete"
    return True, f"freeze with expiry, flip_rate={flip_rate:.3f}"

Three conjunctive requirements, and all three matter. History first: a test that failed once is not flaky, it is unfinished. Flip rate second: a test that fails every run is deterministic and broken, not flaky. Sensitivity third: the seeded-bug check is what separates a test worth preserving from a test worth removing. When all three pass, the freeze still needs an expiry date and a named re-entry condition — a freeze with no expiry is a deletion you did not have the nerve to make.

7. Where free model access and a free server actually help

The ladder needs two things a solo maintainer usually lacks: a corpus of patches to order the tiers against, and a place to run the expensive tier that is not a laptop about to sleep.

For the corpus, free model access is the relevant part — the point is not one better patch but many varied patches from the same failing test, so the ledger has enough rows for the ratio between tiers to mean something. For the runner, a free server option lets the ladder and its JSONL ledger live outside your machine, which makes the ordering decision a team artifact rather than a local habit. Both are operator-supplied availability claims; neither implies a specific model, quota, or hardware spec, so treat them as "the tier exists and I can use it", not as a capacity plan.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two things I would not do: run untrusted agent-generated code on a shared free host without a sandbox or network isolation, and treat a free tier as a guarantee that tier 2 finishes inside a merge window. Wall-clock behavior under contention is exactly what a free host does not promise.

Limitations and who should not use this

This ladder orders checks, it does not validate them. If a property is a tautology — it passes whether or not the bug is fixed — tier 1 will have a beautiful yield and catch nothing real. Fix the properties first.

A short-feedback suite (tier 2 under about 30 seconds) does not need a ladder; the ordering overhead exceeds the savings. Teams without a stable property layer should not skip to tier 2 routing, because a scope hit will send everything to the most expensive check. And if you cannot record at least a few dozen patch attempts per month, the yield ratios will be dominated by noise and you should pick an order from judgment and revisit later.

Closing note

The artifact here is small on purpose: one routing rule, one verdict model, one freeze predicate. Run it for a month before you change anything else. Your ordering will come out different from mine, and the ledger is what will tell you why.