Cost Per Correct Answer Is Worse Than Useless, and It Will Tell You to Use the Cheap Model

A small model costs a fraction of a large one and gets most easy requests right. A router that could tell easy from hard would cut the bill dramatically. Every production LLM system reaches this point, and the argument that follows is always accuracy versus cost — which is the wrong frame, because you can buy any accuracy you like by spending more.

👉 Live, runs in your browser: https://dev48.in/ai/days/day81-cost-routing.html

The metric everyone proposes

"Use cost per correct answer." Here it is, on 2,000 requests, small model at 1 unit and big at 20:

strategy	accuracy	cost / request	cost / correct
always small	74.5%	1.00	1.34 ← the winner
route on difficulty	95.8%	6.42	6.70
oracle	95.8%	5.85	6.11
always big	95.8%	20.00	20.88

🔴 It picks the small model outright. A 74%-accurate model at 1 unit beats a 96%-accurate one at 20 on that metric, every time, and it always will. My first draft of this page asserted the opposite and the in-page check disproved it.

Cost per correct answer, on its own, hands the prize to the cheapest model that is vaguely adequate.

The question that does have an answer

Fix a quality floor first. Given the accuracy you have promised, what is the cheapest way to reach it?

the promise	cheapest that keeps it	spend vs always-big
70%	always small	5%
90%	cascade with a verifier	28%
95%	route on difficulty	32%

At a 90% promise the small model is not on the menu — it is not cheap, it is ineligible. And routing then matches always-big's accuracy exactly for under a third of the spend. That saving is the entire value of routing, and it only becomes visible once the quality bar is written down.

🔴 Routing on the small model's confidence

The obvious design: ask the small model, escalate when it says it is unsure. Cheap, needs no training, and it fails exactly where it matters.

the small model says	requests	actually right
95% confident	843	100%
90% confident	406	0%
60% confident	303	61%
55% confident	448	0%

There is a band where it is 90% confident and 0% correct. Those are precisely the requests escalation exists for, and confidence is blind to them by construction. Confidence measures fluency, not correctness.

The router is not free

Most comparisons quietly price the router at zero. It is a model call too.

for (const req of reqs) {
  const r = strategy(req, opts);
  cost += r.cost;
  routerCost += routerPrice;     // on EVERY request, whichever way it decides
}

🔴 In my first version that price was read from the options and never billed — no strategy returned a router field, so the column stayed at zero and the overhead table was meaningless. Difference two runs and a charge that never lands cannot hide.

The ceiling

An oracle router knows in advance which requests the small model will get right and sends only the rest. No real router beats it, and the gap between yours and the oracle is the only honest measure of how much routing skill you have left to gain.

Here the oracle matches always-big's accuracy for 29% of the spend. Routing on difficulty reaches 32%. There is not much left on the table.

Everything is reproducible

No API, no sampling. Difficulty is u² for u uniform, so P(difficulty < d) = √d and every number is re-derivable in closed form — which is how the verifier checks the simulation rather than trusting it.

184 verifier asserts, 34 in-page checks, 0 failures.