The problem
A cost efficient AI system sends easy work to a cheap model and only escalates hard work to an expensive frontier model. The trouble is knowing which is which. When a task has a test, like code with unit tests, you just run the test: if the cheap answer passes, serve it; if not, escalate. But most real prompts have no test. A question like "what time is the maintenance window" cannot be checked by running code. With no test, a careful system escalates almost everything, and you pay frontier prices for work a cheap model could have done.
We measured our own gateway and found exactly that. On no-test prompts in automatic mode, the system escalated to the frontier 100 percent of the time, at every context length. The cheap tier was capable, but the system did not trust it without a test, so it never served those answers.
The idea: agreement as a stand-in for a test
Instead of a test, ask a second, independent cheap model the same question. If the two cheap models agree, the answer is very likely correct, so serve it cheap. If they disagree, that is the genuinely hard case, so escalate to the frontier. Disagreement never serves a worse answer than before, because the disagreement path is the same escalation that used to happen anyway. Agreement only adds a chance to skip an unnecessary frontier call. The gate is conservative by construction, so its only failure mode is paying for an avoidable escalation, never serving a wrong answer, unless the two cheap models happen to agree on the same wrong answer. That single risk is the whole ballgame, so we measured it directly.
The one number that can break it
The only way this gate ever serves a wrong answer is if two cheap models agree on the same wrong answer. We call that P(wrong given agree). If it is zero, agreement is a safe stand-in for a test. So we stress tested it.
We ran two architecturally different cheap models across four task families, including a set of brand new hard traps we wrote ourselves so they could not be memorized from training data (a reverse-percentage trap, a rise-then-fall price trap, a buy-two-get-one trap, a clock-chime interval trap, a snail-in-a-well trap, and more). The result, at n=160: P(wrong given agree) was 0.00 in every one of the four families (retrieval, reasoning, multi-fact, and the new adversarial traps), with zero agree-and-wrong cases out of 160 total.
When the two cheap models agreed, they were correct 100 percent of the time, across every category, including the traps designed to fool them. They agreed about 76 percent of the time, and each cheap check took about 0.9 seconds at the median. One honest note on rigor: an early run showed a few apparent failures that turned out to be a formatting bug in our scoring, not real errors. We caught it, fixed it, and re-ran clean. The boring re-check is the work.
CORRECTION, added 2026-07-27. The 0.00 above is real but it does not mean what this piece says it means, and two phrases here are now retracted: "safe stand-in for a test" and "correct 100 percent of the time".
One. Zero wrong in 160 does not bound the error rate at zero. For a zero numerator the 95% upper bound runs about 3/n, roughly 1.9%. At a 76% agree-rate that permits on the order of 1.4% of everything served to be wrong and never checked. The point estimate was 0.00. The bound never was, and this piece leaned on the point estimate as if it were the bound.
Two. The probe was short answer arithmetic, where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail. Every row of that probe showed the primary correct, which should have been the tell and was not. Re-run on code, using the same agreement check with the benchmark's own tests as ground truth, the gate leaks low single digit percent, not zero.
Three. The word "independent" in the opening is doing work it cannot support. Agreement substitutes for correctness only when the two endpoints fail independently, and independence is a property of what the provider is serving, not of your code. Ours quietly stopped being independent: the witness was selected by model name, and all three of our cheap backends turned out to be the same underlying model under three different spellings. A same model pair agrees about 88% of the time against 70 to 75% for a genuinely different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.
And the traps cut the other way from how this piece reads them. We presented the adversarial families as evidence the gate was safe. They are the opposite. A bait works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate is least trustworthy on the cases that cost the most to get wrong.
What survives: disagreement is still a useful escalation signal, and checking is still cheaper than producing. What does not survive is treating agreement as a test. If you use this pattern, sample a slice of the agreed cases, escalate them anyway, and score whether the expensive model concurs. That turns a one time number into one that moves when a provider changes what sits behind a model name.
What it does to cost
We shipped the gate to production and re-measured escalation on no-test prompts:
context length frontier escalation before frontier escalation after accuracy after 1,000 tokens 100% 40% 100% 4,000 tokens 100% 20% 100% 16,000 tokens 100% 0% 100% 32,000 tokens 100% 0% 100%
At long context, where frontier calls cost the most, escalation went from total to zero with no loss of accuracy. Across live traffic the system now serves about 91 percent of requests on the cheap tier. Our blended measured cost is about 0.002 dollars per request, and a repeated question is served from cache at close to zero.
*Why it matters
*
The savings on verifiable work, like code with tests, were already real. This extends the same economics to the large class of work that has no test, which is most real questions, without guessing and without giving up accuracy. The hard cases still get the frontier, and those are exactly the cases worth caching a high quality answer for, so the next identical question is served cheap too.
Honesty and limits
Every case we proved is a single final answer. We have not yet proven the gate on multi-step reasoning where a final answer can be right by luck on a broken chain, or where two models could agree on the same wrong chain. That is the next frontier and we are not claiming it here. The result above is for two specific cheap models; a different pair could behave differently, and widening model diversity is a known lever we hold in reserve. We are publishing the result and the honesty, not the gating engineering.
Top comments (2)
The "conservative by construction" framing is the right instinct — making the only failure mode "pay for an avoidable escalation" is exactly how you want to bias a cost gate. The number I'd pressure-test isn't P(wrong|agree)=0 at n=160 (encouraging, but the Wilson interval at that n still leaves room for a rare correlated failure) — it's whether your two cheap models share a training-data blind spot. Architecturally different ≠ statistically independent; two models that both absorbed the same wrong Stack Overflow answer will agree confidently and wrongly, and your hand-written traps (smart, because they dodge memorization) can't surface the correlated errors neither you nor the models anticipated. Two questions: did you break the 76% agreement rate down by task family? If retrieval agrees 95% but reasoning agrees 50%, the cost win is concentrated and you could route by family. And have you considered a background cheap-vs-frontier disagreement-sampling loop to keep re-estimating P(wrong|agree) on live traffic, rather than trusting the offline 160? That turns the gate into something you can monitor for drift instead of a one-time proof.
Hi Max,
I am so sorry it took me this long to respond. I wish I had read this yesterday :) I will be honest with you, I am learning a lot of this as I go, and your insights are exactly the kind of thing that helps me get it right. Thank you for the sharp questions. Please keep them coming.
You were right that the number to really test was not the zero from our trap questions. We wrote those ourselves, so a perfect score there mostly shows we can dodge the obvious memorized stuff, not that agreement actually means the answer is correct. So we went and ran it on a real public test set we did not write, about 380 cases. The honest result is that when the cheap models agree, they are still wrong sometimes. About 10 percent of the time on question answering, and about 29 percent of the time on summarizing.
So agreement is a real signal, but it depends on the type of task. That is your first question answered. We broke it down by task type, and the savings land exactly where you guessed. On question answering, agreement gets us most of the benefit. On summarizing it barely helps and the agreement is weak. So the right move is to decide per task type, not treat agreement as one single switch.
On independence, you nailed the real weakness. Two different models are not truly independent if they both learned from the same internet. They pick up the same blind spots and can be confidently wrong together, and the questions we wrote ourselves will never catch the shared mistakes nobody saw coming. We are not claiming they are independent. The honest next step is to measure agreement between two different models on real data we did not write, broken down by task type, and report the real range instead of one rosy number. Where we said zero, we will say it again with the real number and the honest range.
Your second idea is the one we are building right now. A background check that quietly compares the cheap answer against the expensive one on live traffic, and keeps measuring how often agreement is wrong. That turns it from a one time claim into something we watch over time, and it warns us if things drift.
The one thing we keep no matter what is that the system is built so the worst case is paying for an extra check we did not need, never serving a wrong answer to save money.
Thank you again for pushing on this. It made the whole thing more honest and more useful.