The standard picture of a cheap AI system is a ladder. Small model at the bottom, big one at the top, switch up when the work looks hard. Almost every cost-saving story is a version of that, and the whole design question collapses into "when do I escalate".
We built that. Then we published that our router for it does not work. This is the follow-up we owed. Our router failing was the least of it: the ladder itself is the wrong axis to argue about, and we can now show that with a number.
The four layers
Every number in that figure is from our own runs, and the rest of this piece walks down it. Layer 1 is the only one a ladder can reach.
Layer 1: which model you pick
The result that changed our mind
We ran 2,400 tasks through both rungs. Same tasks, both arms pinned, three repetitions each, so every task has a cheap answer and a frontier answer side by side.
The frontier tier wins. It is 96.9% against 92.8%, an advantage of 4.12 points, 95% CI +3.12 to +5.21, McNemar exact p below 0.0001. It replicates an earlier, weaker result of ours that had never been demonstrated properly.
Then look at what is inside those 4.12 points.
Escalation fixed 133 answers and broke 34. The net is +99, which is where the 4.12 points come from. But 34 tasks were answered correctly by the cheap model and incorrectly by the expensive one.
The obvious objection is that those 34 are noise, and we checked. Of them, 10 were right in all three cheap repetitions and wrong in all three frontier repetitions. Sampling noise would have to land the same way six times running to produce that. Those are a model-by-item interaction: on those items the more capable model is reliably worse.
So escalation is not monotone. Going up the ladder hands you a different distribution of right and wrong, so you trade one set of failures for another set you have never looked at.
Every escalation gate ever built assumes monotonicity. That is what "escalate when it is hard" means: up is better. On our own production ladder, measured, up is better on average and worse 34 times out of 2,400.
What those points cost
The other half is the price. Measured from each row's own token counts, the top rung costs $0.1114 per additional correct answer over the bottom rung, about 21 times the cost per answer. An independent path through the vendor invoices came out at $0.1102 and 22.5 times, which is the only reason we trust the first figure.
Latency moves with it, from a p50 of 0.87s on the cheap rung to 5.99s on the frontier one, about seven times.
The ladder's whole proposition comes to this. Pay 21x, wait 7x longer, gain four points, and reproducibly break a few answers that were already right.
That can still be worth buying. It is a purchase at a price, and there are bigger levers in the system.
Layer 2: what you feed it
This is the one that reframed the project for us, and it is the direct comparison to the ladder.
Same model. Same tasks. Two arms. In the first, the agent found its own context: real navigation in a read-only sandbox, grepping and reading, which is what your agent does right now. In the second, we handed it the one right note and asked it to answer from that.
Mean 39,458 tokens down to 1,337. About thirty times cheaper, at identical accuracy, 75% in both arms. Eight tasks, one repository, judged the same way in both arms, and our test repo is small, so read thirty times as a floor rather than a headline.
Put that beside the ladder. Switching rungs bought four points for 21 times the cost per answer. Changing what we fed the same model cut cost by about thirty times at the same accuracy.
The delivery layer wins by roughly an order of magnitude.
Layer 3: which shape the task is, and whether your instrument can see it
This layer is where we caught ourselves, three hours before publishing, about to say something false about a named vendor's flagship model.
Separately from the ladder work we ran six frontier models over 120 standard tool-calling tasks and 120 trapped ones, twice, because a single-run ranking is noise even at temperature zero. Two runs on the identical item list, verified identical by ID and order, gives a run-to-run movement of at most 3 points on the standard set and 6 on the traps.
The trap half is twelve shapes of ten items. Nine came out at nine or ten of ten for every model, so they separate nobody. The whole between-model spread, from 119 of 120 down to 100, sat in the remaining three.
We had a story ready. The most expensive model in the set scored 1 of 10, twice, on a shape whose prompt is "I need to get to Austin. I'll be setting out from Dallas, give me driving directions." We were going to report that it inverts origin and destination nine times in ten.
Then a reviewer pointed out that our harness never scored inversions. It scored not equal to gold. Two different claims, and only one of them had been measured.
What it actually returned
So we read the arguments the model produced. Across all 18 failures, in both runs:
| what the model actually returned | count |
|---|---|
| origin and destination inverted | 0 |
correct cities, with the state added: gold Dallas, predicted Dallas, TX
|
16 |
| two calls, disambiguating Portland OR from Portland ME | 2 |
It had the origin and the destination right every single time. Our gold was a bare city string and our scorer did exact match, so "Dallas, TX" was marked wrong. The two models that scored a perfect 10 of 10 did so by echoing the bare city name.
That shape measures whether a model writes the state abbreviation, and says nothing about tool calling.
One of the other three shapes was already flagged in our own notes, seven weeks before this run, as having an ambiguous gold. So we removed both and scored the remaining ten shapes:
| trap score of 100, run 1 / run 2 | |
|---|---|
| sonnet-5 100/100 · gpt-5.2 100/99 · gpt-5.1 99/99 | grok-4.5 99/99 · opus-5.5 96/98 · grok-4.7 96/97 |
Ninety-six to one hundred. A four-point spread, against a six-point run-to-run floor. Six frontier models come out indistinguishable on our trap suite, and every apparent difference was our own formatting preference scored as capability.
The lesson belongs to us rather than to any model: an exact-match gold on a free-text argument measures your house style and reports it as capability. You can check for it before publishing: if one shape carries most of your spread, suspect the scoring first, because real capability differences rarely concentrate that neatly.
And it is the same point as the rest of this piece, one level up. The aggregate hid the shape. The shape hid the scorer.
Then we ran it properly, and the models do separate
The trap conclusion above stands on its own, and it is about the traps. The other half of that pilot was 120 standard tasks, and the flag we used to limit the run happened to draw all 120 from the easiest of three categories. We had been about to let "indistinguishable on traps" sit next to a thin standard set and read as a statement about the models.
The full run is 800 standard tasks per model across three categories, plus the traps. It changes the answer.
| category | spread across the six models |
|---|---|
| simple, one call | 10.8 points |
| multiple, several tools offered | 6.0 points |
| parallel, several calls required | 32.5 points |
The category the pilot skipped is the only one that separates anybody. And inside it there is a result worth the whole run: one model answered the first sub-request and stopped. Asked for two calls, it sent one, on 60 of 200 tasks. Every other model did that between 1 and 9 times.
Its per-category scores are 86.8% on simple, 88.0% on multiple, 61.0% on parallel. The aggregate reads 80.6% and sounds like "somewhat weaker". The truth is "competitive on single calls, unusable for parallel work", and only the per-category read says so.
We checked this one against the lesson above before believing it, because two findings the previous night had turned out to be scorer artifacts. On the exact 78 parallel items that model failed, the other five score 55 to 66 of 78, so the items are answerable. What fails is the call count, and the arguments it does send are fine.
And the sample size was the whole story on whether anything was visible at all. At 220 items, zero of the fifteen model pairs separated. At 800, eight of fifteen do. The bootstrap intervals still refuse to rank the top four against each other, which is its own honest result: the models that look best are tied, and the gap that matters is further down.
Layer 4: whether the rule arrives at all
The last layer is the one nobody benchmarks, and it has the largest failure rate of the four.
A rule only helps if it reaches the model at the moment it is needed. We measured retrieval surfacing a safety-relevant rule two times out of three. Always-on delivery scored 100%.
Two in three is worse than useless, and the miss is the smaller problem. The two successes are the bigger one, because they teach you to rely on it. So anything that must hold, holds by construction and never goes through the clever part.
Model choice cannot reach this layer at all. Put the best model in the world here and it answers confidently, without the rule nobody showed it.
A third failure mode, found by making the same mistake twice
We push short notes to the agent, bound to the action it is about to take, instead of hoping retrieval finds them. That fixes the two-in-three. It introduces something else.
Yesterday I shipped the same bug twice in one session: a string replacement that matches nothing returns the original unchanged, so the patch looks applied and does nothing. Afterwards I went looking for the note we must be missing.
The note existed. It was bound to exactly the right trigger. It described the bug precisely, down to the same function call. And on that action it is delivered as 675 characters of 3,751, because the channel caps each item, and the sentence I needed sits past the cut.
Across the whole set that is 98,715 characters which are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.
The reason is worth more than the number. Truncating from the front is safe when a note carries one idea, because the claim leads. It is silently lossy when a note carries two, because the second one lives in the tail. Our fix is one idea per note, which beats any cleverness in the cutting.
So Layer 4 has three states instead of two. The rule can be missing. The rule can arrive. And the rule can be present, correctly selected, and cut, which looks exactly like arriving from the inside and behaves exactly like missing. Ask what happens if retrieval misses a fact, and then ask the second question: what happens if it hits and arrives partial.
The actual claim
Models differ enormously, and Layer 3 is the proof: a 32-point spread on parallel calls against 6 on multiple, in the same six models, on the same day. One of them drops to 61% on a category where it scores 87 and 88 on the other two.
That proof has changed under us. The ninefold gap we opened with turned out to be our own scorer, and what replaced it came from a bigger run that we had to go and do. We kept the number that survived being attacked and discarded the one that failed, and they happened to point the same way, which is the only reason the section still has a point to make.
The claim is that a ladder optimises the wrong axis. A ladder is a scalar, cheap at the bottom and capable at the top, with one dial. The variation we can actually measure runs along three other axes: which shape the task is, what context you supplied, and whether the constraint arrived at all. A ladder has a knob for none of them.
Which is why "just use the better model when it matters" keeps almost working. It is directionally right on the aggregate, and blind in exactly the places the aggregate is blind.
What we do instead
Verify rather than select, because a cheap answer that passed a check is worth more than an expensive answer nobody checked, and checking costs less than escalating. Feed rather than let it search, because that is where the thirty times lives. Push the things that must hold instead of hoping retrieval surfaces them. And read results per shape, never per aggregate, because a 1-in-10 failure goes to the aggregate to hide.
Every one of those is a system decision rather than a model decision, which is the thesis: the model is one component, and the biggest gains sit elsewhere.
Where this could be wrong, including two places we already were
Our router failing is evidence about our router. It is equally consistent with "prediction-based escalation is hard" and with "we built it badly", and we cannot separate those.
The token result is n=8, one repository, one agent. Large effect, small sample.
We already published one piece of routing advice off this data and had to retract it. An earlier read said "escalate simple tasks, hold back parallel ones". That came from 4 of 5 observed breaks being parallel. At n=2,400 the real figure is 11 of 34, and parallel is net positive by 25. Anyone who followed our advice would have withheld escalation from the category it helps second most. Five events looked like a pattern and were not.
And a second one, this week. A re-pricing showed the rung we call cheap costing 3.4 times the rung we call value while getting fewer answers right, and we very nearly shipped that as "strictly dominant, swap them". A per-category read killed it: the value rung wins parallel and simple calls and fails 8 of 40 safety traps that the other passes 40 of 40, on one disposition that produces both results. The win and the loss are the same behaviour, so you cannot take half of it. Which is, again, the thesis: the honest answer was per shape, not per level.
Finally, the shape work here rests on a harness that, until we fixed it this week, was misnaming its own trap shapes. The label was arithmetic that assumed the shape numbers ran contiguously; they skip two. Every stored result named shapes from the third onward after a different trap. We caught it because a row's payload disagreed with its label, and it mattered: we were one step from publishing that the expensive model calls tools users explicitly forbade nine times in ten. It does not. The mislabelled shape was the directions one, which is an argument-formatting story instead of a safety story, and only reading the payload separated them.
Then the same move had to be made a second time on the corrected version, because "it inverts origin and destination" was also false, as the table in Layer 3 shows. Two readings of that shape were wrong before the one that held. The first cost a safety claim about a named vendor, the second an ordering claim, and both died the same way, by opening the arguments the model actually produced.
We added a check tying every label to the human-written description of that shape, then mutation-tested it against the old code to watch it go red. A measurement system that cannot catch its owners being wrong will not catch you being wrong either.
Run your own numbers against ours
Everything above is our system measured by us, which is the weakest position a claim can be in. If you want to check any of it, the endpoint is open.
https://api.tirtha.ai/v1, model tirtha/verified, OpenAI-compatible, so it is a base_url change and nothing else. Sign in with Google at tirtha.ai, mint a key, no card and no sales call. A free key is capped at 100 requests a month, and we will raise it for anyone who asks.
What would be genuinely useful is the thing this piece is about: point it at YOUR task distribution, split the results per shape rather than per aggregate, and tell us where it does worse. A category where we lose is worth more to us than another run where we win, because the whole argument here is that the aggregate hides the shape.
Top comments (1)
The ladder being the wrong axis matches what I've seen. The biggest swing in our costs came from what went into the window, not which model sat on top.