Nobody complained about the gate. They complained about Tuesday.
Our merge queue runs the full eval suite before anything lands: 1,400 cases, about 240 of them scored by an LLM judge, the rest deterministic. Wall clock, 22 minutes. One PR at a time, because the suite pins the same eval dataset and the judge budget. Every engineer on the team would tell you 22 minutes is fine. Go get coffee.
Then one Tuesday I counted nine PRs sitting in the queue at 11:04. The last one merged at 14:22. Three hours and eighteen minutes for a change that took 40 minutes to write, and the gate itself never ran slow, never flaked, never failed. Every single run took its normal 22 minutes. The queue did the rest.
The math nobody ran
A serial gate is a single-server queue. What sets the wait is the ratio between the gate's duration and how fast work arrives. That ratio has a name, utilisation, and the wait it produces is not linear in it. It has a knee.
For a fixed 22-minute service time and randomly arriving merges, the steady-state average wait in queue is:
def gate_queue_wait(s_min: float, arrivals_per_hr: float) -> float:
"""Average queue wait (minutes) before a serial s_min-minute gate.
M/D/1: Wq = rho * s / (2 * (1 - rho)), rho = arrival rate * s."""
rho = arrivals_per_hr * s_min / 60
if rho >= 1:
return float("inf") # the queue never drains
return rho * s_min / (2 * (1 - rho))
for rate in (1.0, 1.5, 2.0, 2.25, 2.5, 2.6):
print(f"{rate:>4}/hr wait {gate_queue_wait(22, rate):6.1f} min")
Run it for our 22-minute gate and the knee is right there:
1.0/hr: utilisation 37%, avg wait in queue 6.4 min, wait + gate 28.4 min
1.5/hr: utilisation 55%, avg wait in queue 13.4 min, wait + gate 35.4 min
2.0/hr: utilisation 73%, avg wait in queue 30.2 min, wait + gate 52.2 min
2.25/hr: utilisation 82%, avg wait in queue 51.9 min, wait + gate 73.9 min
2.5/hr: utilisation 92%, avg wait in queue 121.0 min, wait + gate 143.0 min
2.6/hr: utilisation 95%, avg wait in queue 224.7 min, wait + gate 246.7 min
Capacity is 2.73 merges per hour. That is the whole budget a 22-minute serial gate gives you, ever.
At one merge an hour the gate is invisible. At two an hour, the average PR waits longer in the queue than it spends being tested. At two and a half, the average experience is over two hours, and that is the average on a quiet, evenly spaced day. The formula assumes arrivals sprinkled at random. Real teams merge in bursts, after standup, before the sprint cutoff, and a burst is strictly worse than the formula. Our nine-PR Tuesday was not an anomaly. It was nine arrivals hitting a server that clears 2.7 an hour, and 9 times 22 minutes is 3 hours 18. The math was never going to do anything else.
The part that stung: we had spent a month optimising the suite from 26 minutes down to 22, and we only understood what we had bought after running these numbers. At two merges an hour, the old gate held the average wait at 85 minutes; the new one holds it at 30. And on Tuesday-heavy stretches at two and a half, the 26-minute suite was past capacity entirely (60 over 26 is 2.3 an hour): that queue was not slow, it was diverging, and we had fixed it by accident. Service time matters exactly as much as the ratio says it does, and none of that is visible from the gate's runtime alone.
What we changed
Three tiers, nothing clever.
Tier 0, per commit, 90 seconds. Every deterministic check that does not need the full dataset: schema conformance, tool-call shape, regex and exact-match cases, token-budget ceilings. About 400 of the 1,400 cases, and historically the tier that catches most honest mistakes. Runs on push, before review, outside the merge queue entirely.
Tier 1, the merge queue, batched. The full 22-minute suite still gates every merge, but it gates batches, not PRs. As soon as the runner is free it takes whatever is queued, up to four PRs, and runs once against the batch head. Green, all four land. Red, the batch splits into pairs and reruns, then the failing pair splits again: standard bisection, four extra runs worst case to isolate one offender. A batch of four cuts effective service per PR to five and a half minutes, which moves us from 92 percent utilisation back to about 23 at the same merge rate; the average wait falls off the bottom of the table, into single-digit minutes. Red batches cost us the bisection, and at our failure rate the trade wins by a large margin; if your gate is red a third of the time, batching will hurt, and your problem is the failure rate anyway.
Tier 2, nightly. The expensive sweeps that never belonged in a merge path: full judge calibration against the human-labelled set, cross-model regression, the long-tail scenarios. Nightly, with a report in the morning, and a rule that a red nightly blocks the next day's releases rather than the next engineer's merge.
Did we lose per-PR attribution inside a green batch? Yes, and we decided we do not care: a green batch means every PR in it shipped against the full suite. Attribution only matters on red, and red is exactly when bisection runs.
What it cost
One week of one engineer, most of it spent classifying the 1,400 cases into deterministic versus judge-scored, and deciding which of the heavyweight sweeps belong in the new nightly tier. The queue infrastructure was our existing merge queue with batching turned on. The 22-minute suite itself did not change at all, which is the point: we stopped paying for it per PR without deleting a single case. The nightly sweeps run on top of the merge gate, not instead of it.
Median wait in the queue, the time before your run even starts, went from 74 minutes that Tuesday-heavy week to under six the week after. Add the 22 minutes the suite still takes and a typical merge now lands in under half an hour.
What I'd check first
- Your gate's service time against your busiest hour's arrival rate. Multiply them, service time in hours. Past 0.7, you have a queueing problem, not a test-speed problem, and shaving suite minutes will not save you.
- What fraction of the gate is deterministic checks that could run per commit. Ours was 29 percent of cases and they catch the majority of failures; they were queueing behind an LLM judge for no reason.
- Whether your queue serialises PRs that could gate as a batch. Serial-per-PR is the default in most setups. It is also the strictest possible service discipline, and the table above is the bill for it.
Top comments (1)
The concept of utilisation and its impact on wait times in a serial gate is really insightful, and I appreciate how you've applied the M/D/1 formula to illustrate this point. The example of how optimising the suite from 26 minutes to 22 minutes only marginally improved wait times at higher arrival rates is a great reminder that service time is only part of the equation. I'm curious to know more about how the introduction of tiered testing, particularly the batched Tier 1, has affected the overall merge queue wait times and utilisation - have you seen a significant reduction in wait times since implementing this change?