Budget the Gate, Not Just the Tests: Stage Caps and a Calibration Check for Agent Patches

A test gate for agent patches fails in two directions. It lets bad patches through, and it gets slow enough that people route around it. This post is about the second failure: how to give each stage of the gate a wall-clock budget, keep that budget honest across machines, and demote checks instead of deleting them when a patch pushes the suite past its cap.

The artifact is two small scripts — a calibration probe and a stage allocator — plus a decision table and a three-strike rule. It assumes you already have checks worth running. It does not re-derive what those checks should be.

The failure this prevents is bypass, not slowness

A slow gate is not a neutral inconvenience. Every extra minute of pre-merge latency raises the chance that someone merges with a local run, or adds a skip marker, or approves a patch they did not read. Agent-authored patches make this worse because they arrive in volume: ten patches a day that each add "one small test" and nobody notices the gate doubled.

Budget creep is measurable, which is the good news. Record the total wall clock of each stage on every run, and you get a curve. The rule I use is boring and effective: if a stage exceeds its cap on three consecutive runs, the next merge window is a budgeting task, not a feature task.

Step 1: Measure per-check cost and normalize it

Absolute seconds do not travel. A runner with a cold cache, a noisy neighbour, or different disk will produce different numbers, and a budget measured on one machine is fiction on another. Normalize instead.

Add a calibration check whose purpose is to measure the machine, not the code. It should be deterministic, CPU-bound, and touch no network or disk:

# calibration_check.py — one number that makes timings portable
import hashlib, json, time

def calibration_seconds(rounds: int = 200_000) -> float:
    payload = b"agent-patch-gate"
    start = time.perf_counter()
    h = payload
    for _ in range(rounds):
        h = hashlib.sha256(h).digest()
    return time.perf_counter() - start

if __name__ == "__main__":
    print(json.dumps({"calibration_s": round(calibration_seconds(), 4)}))

Run it once at the start of the gate, then express every check's cost in calibration units: cost_rel = p95_seconds / calibration_seconds. Use p95, not the mean, because the checks that break your budget are the tail ones.

Two honest caveats. First, this proxy tracks CPU throughput, not I/O or parallel contention; a check that is 80% database round-trips will not normalize properly. Second, run the calibration probe on the same host class you intend to compare, or accept a wide error bar.

Collect the raw numbers into a file you commit:

pytest -q --durations=0 --durations-min=0.5 | tee durations.txt
# then transform to timings.json:
# {"calibration_s": 0.51,
#  "checks": [{"id": "props.core", "p95_s": 41.2},
#             {"id": "fixtures.golden", "p95_s": 12.8},
#             {"id": "audit.flake_ledger", "p95_s": 96.0}]}

The numbers above are illustrative inputs, not measurements of anyone's suite.

Step 2: Fit checks into stage budgets

State the caps once, in calibration units, and let a script assign checks. A cap of 5 minutes with a 0.5 s calibration unit is 600 units. Pick the caps from how long a reviewer will actually wait, not from how long the suite happens to take today.

#!/usr/bin/env python3
"""check_budget.py — place each check in the earliest stage whose cap can hold it.

Exit 2 when a check fits nowhere. Silent truncation is how gates rot.
"""
import json, sys
from pathlib import Path

STAGES = ("pre-merge", "post-merge", "nightly")
BUDGET_REL = {"pre-merge": 600.0, "post-merge": 2400.0, "nightly": 8000.0}
UNMEASURED_REL = 2.0  # conservative charge for a check with no timing

def rel_cost(check, cal):
    p95 = check.get("p95_s")
    return UNMEASURED_REL if p95 is None else float(p95) / cal

def main(src="timings.json", dst="schedule.json"):
    data = json.loads(Path(src).read_text())
    cal = float(data["calibration_s"])
    remaining, schedule, overflow = dict(BUDGET_REL), {s: [] for s in STAGES}, []

    for check in sorted(data["checks"], key=lambda c: rel_cost(c, cal)):
        cost, placed = rel_cost(check, cal), None
        for stage in STAGES:
            if cost <= remaining[stage]:
                remaining[stage] -= cost
                placed = stage
                break
        if placed is None:
            overflow.append(check["id"])
        else:
            schedule[placed].append({"id": check["id"], "cost_rel": round(cost, 3)})

    for stage in STAGES:
        used = BUDGET_REL[stage] - remaining[stage]
        print(f"{stage:11s} {used:7.1f}/{BUDGET_REL[stage]:<7.0f} units  "
              f"{len(schedule[stage])} checks")
    if overflow:
        print("fits no stage:", ", ".join(overflow), file=sys.stderr)
        return 2
    Path(dst).write_text(json.dumps(schedule, indent=2))
    return 0

if __name__ == "__main__":
    sys.exit(main())

The sort is cheapest-first, so the fast stage absorbs as many checks as possible and the expensive ones land where they belong. BUDGET_REL is committed alongside timings.json, so a PR that changes the caps is visible in review.

One more rule makes the ledger self-defending: an agent patch that adds a check must include that check's measured cost. No timing, no pre-merge slot — the check starts in nightly at the conservative charge and earns promotion after three real measurements.

Step 3: Demote, don't delete

When a check exceeds its stage cap, the wrong move is to delete it and the second-wrong move is to raise the cap without saying why. Demotion preserves coverage and moves the cost where latency does not matter.

Check kind	Default stage	First demotion	Deletion requires
Invariant / property checks	pre-merge	split by module, then post-merge	proof the property is subsumed
Golden-file fixture locks	pre-merge	post-merge	owner sign-off in the PR
Order and rerun sampling	post-merge	nightly, sampled	never — sample instead
Quarantine and flake audits	nightly	weekly	never — expire instead
Slow end-to-end replays	nightly	weekly	a dated review ticket

Write the reason into the schedule file. "reason": "overflow" on a nightly entry is the earliest signal that a suite needs splitting, and a bare list of check names tells the next person nothing.

Step 4: Where free model access and a free server actually help

The loop above has two cost centres: authoring the patches and executing the stages. I run agent-authored changes through MonkeyCode, using its free model access for the drafting and the free server option for the post-merge and nightly stages when my own runners are saturated. That keeps the expensive, latency-tolerant half of the gate off the machines people are waiting on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the product claims as the operator's: free model access and a free server option exist, and I am not going to quote quotas, hardware, or durations I cannot verify. The design consequence is what matters — if you do not control the machine, absolute budgets are meaningless, which is exactly why the calibration unit is in Step 1. Keep timings.json and check_budget.py in git so the numbers and the policy travel with the code instead of living in a CI dashboard nobody reads.

If you want to try the loop without provisioning anything, that combination is a reasonable place to start; run the calibration probe first and re-measure on your own suite before trusting any cap.

Limitations and who should not use this

• Suites under roughly 2 minutes of pre-merge time do not need an allocator. The overhead of maintaining the ledger exceeds the latency saved.
• I/O-heavy or network-dependent checks normalize poorly; their relative cost will drift, and you will re-measure often.
• The calibration proxy is a CPU throughput measure. On shared or throttled hosts the error bar is wide enough that a 10% budget change is noise.
• Teams without a place to review a committed budget file will watch it drift. If the schedule is generated but never read, this is ceremony.
• If your real problem is checks that never fail, budgeting them is optimizing the wrong axis. Fix the checks first.

The review checklist

Before you trust the gate again, confirm four things: every stage has a cap in calibration units, every check has a measured p95 or an explicit conservative charge, the three-strike rule has an owner, and every demoted check has a written reason and a return path.

Latency budgets are not bureaucracy. They are the reason the gate still gets read on a Friday afternoon.