If you serve a model cascade, escalation is your cost dial. Not your model choice, not your prompt,
not your context window. The single number that moves your bill is what fraction of requests climb to
the expensive tier.
So the obvious thing to build is a router: look at the incoming request, predict whether the cheap
model will get it right, and only pay for the expensive one when the answer is no. We built that. It
does not work, and the reason turned out to be more interesting than a working router would have been.
Publishing it because a negative result with a mechanism is worth more than a positive result with a
story, and because the last two times we published numbers, readers found the flaws faster than we did.
What we built and what it scored
Our gateway already embeds every prompt, because the semantic cache needs a vector. Reusing that
vector to predict difficulty is free. That is exactly why we did it, and that turned out to be the
founding design flaw.
Trained on 539 real coding tasks labelled by actual execution (428 easy, 111 hard):
held-out AUC 0.594
5-fold CV 0.55 to 0.57 (one fold below chance)
best threshold scores exactly what "never escalate" scores
A classifier that cannot beat a constant policy is not a weak classifier. It is not a classifier.
It is a feature limit, not a tuning problem, and we made it prove that
The tempting read is underfitting or overfitting, so we swept L2 regularisation across four orders of
magnitude, 0.01 to 100, with cross-validation at every step. Train AUC stayed at 0.94 to 0.98 while
test AUC stayed at 0.58 to 0.59, at every single setting. The gap never closed. A gap that survives
four orders of magnitude of regularisation is telling you the features do not contain the signal.
Then the result that actually explains it. We threw together 11 crude surface features, things like
length and token counts, no semantics at all:
11 crude surface features AUC 0.610
1024-dimension prompt embedding AUC 0.552
both together AUC 0.609 (the embedding adds nothing)
Eleven numbers you could compute with a ruler beat a 1024-dimension semantic embedding. The reason is
simple once you see it: the cache vector encodes topic, not difficulty. It is built to answer "have
I seen a question like this before", and it is good at that. Two questions about sorting a list sit
close together in that space whether one is trivial and the other is subtle. We reused it because it
was free, and free was the whole problem.
Two measurement traps, which are worth more than the router
One. Scoring a cost-saving router on accuracy alone marks it failed by construction. A router that
saves money by sending easy work to a cheap model will, correctly, be slightly less accurate than
always escalating. Our first yardstick printed DO-NOT-SHIP on a router that actually had skill. The
control you need is random routing at the same cost, not the expensive model. If you cannot beat
a coin flip that spends what you spend, you have nothing. If you can, you have something, even if your
raw accuracy went down.
Two. The ceiling is not "always escalate". We assumed the expensive model was the upper bound and
a perfect router would approach it. It is not. On our 539 tasks, escalating everything to the top tier
rescues 39 answers the cheap model got wrong and breaks 23 it had already got right. Net +16.
always escalate 82.4%
a PERFECT router 86.6%
A perfect router beats always-escalate by 4.2 points, because it also knows when not to escalate.
That headroom is real and no gate-local heuristic captures it. If you are benchmarking a cascade
against "just use the big model", you are measuring against the wrong ceiling.
What is still alive
Showing the router the cheap model's draft rather than only the prompt does better: AUC 0.640, and
81.8% accuracy at 36% escalation against 80.5% for an equal-cost random control. That is the right
shape. But permutation testing gives p=0.0375 unadjusted and roughly p=0.30 after Bonferroni, so
the honest verdict is promising, not proven. We are not going to claim it until it survives a
pre-registered run.
Which makes sense mechanically: the draft carries evidence of difficulty that the question alone does
not. A model that is about to be wrong often looks different while being wrong.
The questions, and this is why I am posting
I would rather hear from people who have hit this than keep guessing.
What features actually carry difficulty before generation? We have shown that a topic embedding
does not, and that crude surface stats beat it. That is a low bar. What clears it?Is pre-generation difficulty prediction possible at all, or is the draft the earliest honest
signal? It is entirely plausible that "will this model fail" is not a property of the question,
only of the interaction. If someone has evidence either way I would like to see it.If you run a cascade in production, what do you actually route on? I suspect the honest answer
for most teams is a hand-written rule about request type, and I suspect those rules do better than
our classifier did. That would itself be a finding.
Numbers, labels and the negative result are ours to share. If you want the setup in more detail, ask
and I will write it up.
Top comments (6)
This is the negative result I wish more router writeups kept. Accuracy against the expensive model is the wrong control if the thing you are buying is lower escalation. The useful baseline is cost-matched random routing, then verifier coverage on top of it. Otherwise the router gets blamed for not being an oracle.
You have put your finger on the exact thing that nearly made us bin a working router. Our first yardstick scored it on accuracy alone and printed DO-NOT-SHIP on a router that had real skill, because a router that correctly sends easy work to a cheap model will always look slightly worse than always escalating. Cost-matched random is the only control that answers the question you are actually asking.
On verifier coverage as the layer above it, one number that surprised us and is worth having before you lean on it. Coverage is not a single quantity, it is per workload family. Our agreement gate, meaning two different models answering and serving cheap when they concur, still lets through 1.7 to 3.5 percent wrong on code where executable tests are the ground truth. On faithfulness judgement, with nothing to execute, the same gate approves a wrong answer 27.5 percent of the time, 95 percent interval 16.1 to 42.8. Same mechanism, same code, an order of magnitude apart.
So the stack you are describing works, but the coverage term has to be measured on the traffic you actually serve rather than inherited from a benchmark. We published a 0.00 for that gate once and a reader took it apart in the comments, correctly, because it had been measured on arithmetic where the cheap model is essentially never wrong. A gate cannot be measured where the primary does not fail.
The other half of your point deserves stating plainly: the ceiling is not always-escalate either. On our 539 tasks the expensive model rescues 39 answers the cheap one got wrong and breaks 23 it already had right. A perfect router beats always-escalate by 4.2 points precisely because it also knows when not to escalate.
This is a useful negative result. Routing sounds simple until you realize the cheap-versus-expensive decision depends on uncertainty, domain risk, and failure cost, not just prompt shape. The router needs to know when it is allowed to be wrong.
That reframes the negative result better than my own write-up did, and I think you are pointing at why it failed rather than just adding to it.
We tried to predict difficulty as a property of the prompt. But cost of being wrong is a property of the caller's situation, not of the text, so no amount of prompt features can carry it. The same request is cheap-is-fine inside an exploratory script and needs the expensive model inside a customer-facing job, and those two are byte-identical on the wire. Half the decision was never in the input we were modelling.
Which suggests the classifier was mis-specified rather than merely weak, and that is a more useful conclusion than "our features were bad".
What we do in practice is the boring version of your point: the caller declares it. A per-request mode says how far this one is allowed to climb, defaulting to a cascade that stops as soon as a check passes. It puts the risk tolerance with the person who knows it instead of asking a model to infer it from wording. It is less clever than a router and it has the advantage of being correct.
The part I still find open is your last sentence, and I would take an opinion on it. If the router must know when it is allowed to be wrong, that knowledge has to arrive as an explicit signal, at which point you are not really routing, you are honouring a policy. Do you think there is a version where the system infers permissible failure rather than being told it, or is being told it the actual answer?
Do you think the failure comes from predicting task difficulty, or from the fact that model confidence is a poor proxy for correctness?
Ours failed upstream of confidence, and I think the distinction matters because the two failures have different fixes.
We never got as far as using the model's confidence. The router was trained on the prompt embedding, on the theory that if the cache already computes a vector we may as well reuse it. That was the founding mistake: the vector encodes topic, not difficulty. Two questions about sorting a list sit close together in that space whether one is trivial and the other is subtle. The tell was brutal, and it is in the article: eleven crude surface features, things like lengths and counts with no semantics at all, scored AUC 0.610 while the 1024-dimension embedding scored 0.552, and combining them added nothing. When a ruler beats an embedding, the embedding is not measuring the thing.
So for us it was difficulty prediction from the prompt, and it is a feature limit rather than a tuning one. We swept regularisation across four orders of magnitude and the train-test gap never closed.
On your other half, though, the evidence I have seen points your way too. Showing the router the cheap model's own draft rather than only the question does better, AUC 0.640, which is what you would expect if the informative signal only exists after generation begins. And another developer in a thread I was following measured his own confidence scores against real misses and found 95.8 percent of the misses sat at confidence 0.9 or above. That is his data, not mine, but it matches the general shape: confidence is well calibrated to fluency and poorly calibrated to correctness.
The honest synthesis is probably that both are true in sequence. The question alone does not carry difficulty, and the model's stated confidence does not carry correctness, so the only place left to look is the draft itself and whether it survives a check.