Run the cheap model. If it is confident, ship its answer. If not, escalate to the expensive one. You pay big-model prices on the escalated fraction only.
Every discussion of this focuses on the two accuracies. Look at what the code actually reads:
answer, confidence = small(x)
if confidence >= threshold: return answer
return big(x)
It reads confidence. It never sees whether the small model was right. So the quantity that decides whether a cascade works is not accuracy — it is whether that confidence score ranks the errors below the correct answers.
The whole frontier, computed live: https://dev48.infy.uk/ai/days/day66-model-cascades.html
Which is why calibration beats capability
At an identical 20% escalation budget:
| cheap model | calibration | cascade accuracy |
|---|---|---|
| 70% | perfect | 88.5% |
| 80% | poor | 84.4% |
The first knows exactly which 30% to hand over. The second escalates a random-ish selection and wastes most of the budget on cases it would have got right.
The ablation says the same thing. Holding escalation at 20%: +0.30 calibration buys 6.1 points, +0.10 on the cheap model's accuracy buys 5.8, and +0.05 on the expensive model buys 0.9. Calibration is usually also the cheapest of the three to improve — temperature scaling is a one-parameter fit.
A cascade has no single score
It has a curve. Two configurations can only be compared at the same cost, which is why the ablation pins the escalation rate rather than comparing free-floating accuracies.
A fixed threshold is not a fixed budget
You tune threshold = 0.85 and it escalates 20%. Then traffic gets harder and the same threshold escalates 33%. Nothing in your code changed and your bill moved.
const sorted = confidences.slice().sort((a, b) => a - b);
const threshold = sorted[Math.floor(rate * sorted.length)]; // a BUDGET, not a constant
One line converts a drifting threshold into a fixed budget. Which of accuracy and cost you want to hold constant is a real decision — the important part is making it deliberately rather than discovering it on an invoice.
"Run both, escalate on disagreement" is not a serving strategy
It costs 26.00 per request against 25.00 for always-expensive, at identical accuracy — because you already paid for the expensive call in order to compare. It is a fine technique for finding hard cases offline. The two get conflated constantly.
A bug worth mentioning
My first confidence model was calibration * correct + (1 - calibration) * noise. At calibration 1 every score collapses to exactly 0 or 1 — so a quantile threshold landed on a tie and escalated nothing at all, and the headline finding measured as zero.
Real confidence scores are continuous. The fix was to interpolate between "uniform over the whole range" and "uniform over the correct half": perfectly ordered, and tie-free.
Part of a from-scratch series — one AI concept a day, measured in-browser: https://dev48.infy.uk/aifromzero.php
Top comments (0)