If you serve a model cascade, escalation is your cost dial. Not your model choice, not your prompt, not your context window. The single number that moves your bill is what fraction of requests climb to the expensive tier.
So the obvious thing to build is a router: look at the incoming request, predict whether the cheap model will get it right, and only pay for the expensive one when the answer is no. We built that. It does not work, and the reason turned out to be more interesting than a working router would have been.
Publishing it because a negative result with a mechanism is worth more than a positive result with a story, and because the last two times we published numbers, readers found the flaws faster than we did.
What we built and what it scored
Our gateway already embeds every prompt, because the semantic cache needs a vector. Reusing that vector to predict difficulty is free. That is exactly why we did it, and that turned out to be the founding design flaw.
Trained on 539 real coding tasks labelled by actual execution (428 easy, 111 hard):
held-out AUC 0.594
5-fold CV 0.55 to 0.57 (one fold below chance)
best threshold scores exactly what "never escalate" scores
A classifier that cannot beat a constant policy is not a weak classifier. It is not a classifier.
It is a feature limit, not a tuning problem, and we made it prove that
The tempting read is underfitting or overfitting, so we swept L2 regularisation across four orders of magnitude, 0.01 to 100, with cross-validation at every step. Train AUC stayed at 0.94 to 0.98 while test AUC stayed at 0.58 to 0.59, at every single setting. The gap never closed. A gap that survives four orders of magnitude of regularisation is telling you the features do not contain the signal.
Then the result that actually explains it. We threw together 11 crude surface features, things like length and token counts, no semantics at all:
11 crude surface features AUC 0.610
1024-dimension prompt embedding AUC 0.552
both together AUC 0.609 (the embedding adds nothing)
Eleven numbers you could compute with a ruler beat a 1024-dimension semantic embedding. The reason is simple once you see it: the cache vector encodes topic, not difficulty. It is built to answer "have I seen a question like this before", and it is good at that. Two questions about sorting a list sit close together in that space whether one is trivial and the other is subtle. We reused it because it was free, and free was the whole problem.
Two measurement traps, which are worth more than the router
One. Scoring a cost-saving router on accuracy alone marks it failed by construction. A router that
saves money by sending easy work to a cheap model will, correctly, be slightly less accurate than
always escalating. Our first yardstick printed DO-NOT-SHIP on a router that actually had skill. The
control you need is random routing at the same cost, not the expensive model. If you cannot beat
a coin flip that spends what you spend, you have nothing. If you can, you have something, even if your
raw accuracy went down.
Two. The ceiling is not "always escalate". We assumed the expensive model was the upper bound and
a perfect router would approach it. It is not. On our 539 tasks, escalating everything to the top tier
rescues 39 answers the cheap model got wrong and breaks 23 it had already got right. Net +16.
always escalate 82.4%
a PERFECT router 86.6%
A perfect router beats always-escalate by 4.2 points, because it also knows when not to escalate. That headroom is real and no gate-local heuristic captures it. If you are benchmarking a cascade against "just use the big model", you are measuring against the wrong ceiling.
What is still alive
Showing the router the cheap model's draft rather than only the prompt does better: AUC 0.640, and 81.8% accuracy at 36% escalation against 80.5% for an equal-cost random control. That is the right shape. But permutation testing gives p=0.0375 unadjusted and roughly p=0.30 after Bonferroni, so the honest verdict is promising, not proven. We are not going to claim it until it survives a pre-registered run.
Which makes sense mechanically: the draft carries evidence of difficulty that the question alone does not. A model that is about to be wrong often looks different while being wrong.
The questions, and this is why I am posting
I would rather hear from people who have hit this than keep guessing.
What features actually carry difficulty before generation? We have shown that a topic embedding
does not, and that crude surface stats beat it. That is a low bar. What clears it?Is pre-generation difficulty prediction possible at all, or is the draft the earliest honest
signal? It is entirely plausible that "will this model fail" is not a property of the question,
only of the interaction. If someone has evidence either way I would like to see it.If you run a cascade in production, what do you actually route on? I suspect the honest answer
for most teams is a hand-written rule about request type, and I suspect those rules do better than
our classifier did. That would itself be a finding.
Numbers, labels and the negative result are ours to share. If you want the setup in more detail, ask and I will write it up.
Top comments (17)
This is a useful negative result. Routing sounds simple until you realize the cheap-versus-expensive decision depends on uncertainty, domain risk, and failure cost, not just prompt shape. The router needs to know when it is allowed to be wrong.
That reframes the negative result better than my own write-up did, and I think you are pointing at why it failed rather than just adding to it.
We tried to predict difficulty as a property of the prompt. But cost of being wrong is a property of the caller's situation, not of the text, so no amount of prompt features can carry it. The same request is cheap-is-fine inside an exploratory script and needs the expensive model inside a customer-facing job, and those two are byte-identical on the wire. Half the decision was never in the input we were modelling.
Which suggests the classifier was mis-specified rather than merely weak, and that is a more useful conclusion than "our features were bad".
What we do in practice is the boring version of your point: the caller declares it. A per-request mode says how far this one is allowed to climb, defaulting to a cascade that stops as soon as a check passes. It puts the risk tolerance with the person who knows it instead of asking a model to infer it from wording. It is less clever than a router and it has the advantage of being correct.
The part I still find open is your last sentence, and I would take an opinion on it. If the router must know when it is allowed to be wrong, that knowledge has to arrive as an explicit signal, at which point you are not really routing, you are honouring a policy. Do you think there is a version where the system infers permissible failure rather than being told it, or is being told it the actual answer?
That explains the failure really well. Prompt difficulty is not enough because risk belongs to the caller context. The same text can be disposable in a notebook and critical in production. A router probably needs policy/context inputs, not only linguistic features.
Risk living in the caller context is right, and I can tell you what happened when we tried to build the cheap version of that intuition, because the failure was more interesting than the idea.
We built a discriminator that reads the request and decides whether it needs the expensive path. It works. It is also unaffordable, and not for the reason I expected. To push false negatives down to a level anyone would accept in production, it had to route the large majority of traffic to the slow path anyway, which means it was buying almost nothing. Then we looked at what the discriminator was actually keying on, and it was the difference between same-parameter and different-parameter calls. That is argument extraction. It is the model's own job.
So the lesson we took is narrower and more annoying than "use policy features instead of linguistic ones": you cannot cheaply do part of the model's work in order to decide whether to call the model. Any signal good enough to predict whether the cheap path will succeed seems to require most of the understanding the cheap path was supposed to provide. I would be genuinely interested if you have seen a counterexample, because if one exists it is the whole ballgame and I could not find it.
What did work was not a better router. It was noticing we had one slot serving two task shapes with opposite binding constraints. A tool call is tens of output tokens, so the wall time is dominated by the round trip and what you want is accuracy. A code answer is hundreds to thousands of output tokens, so the wall time is dominated by generation rate and what you want is throughput. Those pull toward different models hard enough that on our candidate set nothing sat near the top of both. Splitting the slot by shape helped substantially where a year of swapping the single model had not.
I will flag the honest limit on that, since it is the part I would attack if you said it to me: it is an argument from a candidate set we happened to test, so "no model is good at both" is a statement about our sample, not a law. And I have wall-clock numbers for the whole task but not a decomposition proving the round-trip half, so treat that as the motivating story rather than a measured mechanism.
Your framing is the more general one, though, and I think it survives our result. The useful input was never a property of the prompt. It was a property of what the caller was doing with it, and task shape turned out to be the cheapest observable proxy for that we could find.
That is a strong result, and the “cheaply do part of the model’s work” line is the part I would keep. The shape split also feels more durable than a universal router: first decide what kind of work this is, then choose the tradeoff. Tool calls and long code answers really do optimize for different failure modes.
"First decide what kind of work this is, then choose the tradeoff" is the better statement of it, and I want to hand you something I found today that makes the original article read worse than it did when I wrote it.
I went through our own cascade line by line, checking every transition against the flags actually set in production rather than the defaults in source. The difficulty router that article is about is not merely ineffective. It is structurally dead. The weights are unset, so the router object is never constructed, so the branch that would serve a cheap answer on the router's say-so is unreachable code. We shipped a piece about a router that is not wired in.
That is embarrassing in a useful way, because it means the article's conclusion was right for a reason I had not identified. I concluded prediction does not work. What is actually running is not prediction at all.
The live mechanism is agreement. With no caller tests there is no oracle, so the gate asks a second cheap model the same question and serves the cheap answer only if the two agree above a similarity threshold. That is not a prediction made before the work, it is a measurement made after it, and it is a different animal with different failure modes. Prediction fails by being miscalibrated. Agreement fails by being expensive to obtain and by measuring the wrong thing.
Ours is currently measuring the wrong thing, which is the part I would flag as the transferable warning. The two models compared are different models, and the comparison is word overlap across the whole answer, prose included. So variable names and commentary dominate the signal, and a threshold that was tuned when both sides were the same model is now being applied to a cross-model comparison, where agreement is systematically lower for reasons that have nothing to do with correctness. The gate fires on dialect, not on disagreement.
So the shape I would now put on the whole thing: routing by predicted difficulty failed, routing by task shape works and is cheap because shape is an observable property of the request rather than an inference about it, and verifying by agreement is a third thing entirely that needs its threshold calibrated against the specific pair of models doing the agreeing. I have numbers on that last part but they are not through their own verification yet, so I will not quote them here.
This is the negative result I wish more router writeups kept. Accuracy against the expensive model is the wrong control if the thing you are buying is lower escalation. The useful baseline is cost-matched random routing, then verifier coverage on top of it. Otherwise the router gets blamed for not being an oracle.
You have put your finger on the exact thing that nearly made us bin a working router. Our first yardstick scored it on accuracy alone and printed DO-NOT-SHIP on a router that had real skill, because a router that correctly sends easy work to a cheap model will always look slightly worse than always escalating. Cost-matched random is the only control that answers the question you are actually asking.
On verifier coverage as the layer above it, one number that surprised us and is worth having before you lean on it. Coverage is not a single quantity, it is per workload family. Our agreement gate, meaning two different models answering and serving cheap when they concur, still lets through 1.7 to 3.5 percent wrong on code where executable tests are the ground truth. On faithfulness judgement, with nothing to execute, the same gate approves a wrong answer 27.5 percent of the time, 95 percent interval 16.1 to 42.8. Same mechanism, same code, an order of magnitude apart.
So the stack you are describing works, but the coverage term has to be measured on the traffic you actually serve rather than inherited from a benchmark. We published a 0.00 for that gate once and a reader took it apart in the comments, correctly, because it had been measured on arithmetic where the cheap model is essentially never wrong. A gate cannot be measured where the primary does not fail.
The other half of your point deserves stating plainly: the ceiling is not always-escalate either. On our 539 tasks the expensive model rescues 39 answers the cheap one got wrong and breaks 23 it already had right. A perfect router beats always-escalate by 4.2 points precisely because it also knows when not to escalate.
The surface features beating semantic embeddings is the important result here: difficulty is a relational property between the task and the model, not an intrinsic property of the task itself, so an embedding that describes the prompt content is measuring the wrong object. A router that could work would need to condition on the cheap model's known failure modes — where has it historically misfired, and does this prompt land in that neighborhood — rather than on the prompt's meaning in isolation. That's why crude surface features like length win: they correlate weakly with a model's token budget limits and attention degradation thresholds, which happen to be real failure boundaries. The negative result you'd want next is whether a model-error embedding (trained on cheap-model mistakes rather than task inputs) closes the gap.
This is the second time this week someone has reframed the failure better than the article did, and the two reframes turn out to be halves of one thing.
Alex Shev made the other half in this thread: the cost of being wrong is a property of the caller's situation, not of the prompt, so no prompt feature can carry it. You are making the same move on the other term: the probability of being wrong is relational between task and model, not intrinsic to the task, so no prompt embedding can carry that either.
Put together, the router was trying to predict an expected loss whose two factors were both outside its input. That explains the negative result better than anything I wrote. It was mis-specified rather than weak, and the surface features winning is the tell: length correlates with real failure boundaries (budget limits, attention degradation) while the semantic embedding describes the object we were not asking about.
On your proposed next negative result. The error labels exist, since every training row already carries whether the cheap model got it right, so a model-error embedding is buildable from data I have rather than data I would need to collect. I have not run it, and I would want the control specified in advance: compared against cost-matched random routing, not against accuracy, because a router that never routes anything cheap scores beautifully on accuracy.
The reason I would push that experiment down my own list is a constraint I ran into today and had not appreciated. An error-conditioned router is a claim about a specific model at a specific version. Our cheap tier changed three times in ten days, and every swap silently orphaned the measurements taken before it, so a router trained on model A's failure neighbourhood is describing a model we no longer serve. That is survivable for a stable deployment and close to fatal for one still choosing its models, which is where we are. So the honest ordering for us is: freeze the ladder first, then condition on the failure modes of the thing that is actually frozen.
Which is arguably a stronger version of your point. Difficulty is relational, and one side of the relation is a moving target unless you deliberately hold it still.
Fantastic analysis, Tom! The point about topic embeddings encoding meaning rather than failure modes is such a sharp takeaway. Spot-on technical breakdown.
Thank you. That distinction has held up, and it is the part I would keep if I had to throw the rest away: an embedding tells you what a prompt is about, and difficulty is not a property of the topic. It is a relation between the prompt and the particular model answering it.
The follow on we measured since: cheaply predicting the hard cases means doing part of the model's work in order to decide whether to call the model. Ours held false negatives to 0.5% only by sending about 70% of traffic down the slow path, which is not routing, it is a tax.
Spot on. That 'tax' description is too real—once 70% of traffic has to hit the slow path just to keep false negatives low, the optimization math completely breaks. Thanks for sharing the real-world numbers; it's a massive sanity check for anyone else tempted to build a custom router!
Thank you. One number I would add, because it arrived after that reply and it is the part that surprised me.
We finally metered our own cheap tier properly this week, and the first thing it caught was our benchmark harness overstating cost by about 3.3x. Not a rounding error, a whole different answer, and it had been wrong for weeks. The cause was two correct halves that could not talk: a rate fetcher keyed on the endpoint that actually served, and a cost lookup keyed on our internal slot name. Neither was broken. They simply never joined, and the lookup fell back to a default for a price tier we had retired.
The reason I mention it on a routing thread is that it is the same shape as the router failure. Both look like a hard modelling problem and turn out to be an instrument reading something other than what you believed. The router was measuring topic when we wanted difficulty. The cost table was measuring a slot name when we wanted an endpoint. In both cases the honest fix was to go and read what the thing actually returns instead of predicting it.
So the tax is real, and worth saying plainly: the cheaper move than a better router was a meter that does not lie.
Do you think the failure comes from predicting task difficulty, or from the fact that model confidence is a poor proxy for correctness?
Ours failed upstream of confidence, and I think the distinction matters because the two failures have different fixes.
We never got as far as using the model's confidence. The router was trained on the prompt embedding, on the theory that if the cache already computes a vector we may as well reuse it. That was the founding mistake: the vector encodes topic, not difficulty. Two questions about sorting a list sit close together in that space whether one is trivial and the other is subtle. The tell was brutal, and it is in the article: eleven crude surface features, things like lengths and counts with no semantics at all, scored AUC 0.610 while the 1024-dimension embedding scored 0.552, and combining them added nothing. When a ruler beats an embedding, the embedding is not measuring the thing.
So for us it was difficulty prediction from the prompt, and it is a feature limit rather than a tuning one. We swept regularisation across four orders of magnitude and the train-test gap never closed.
On your other half, though, the evidence I have seen points your way too. Showing the router the cheap model's own draft rather than only the question does better, AUC 0.640, which is what you would expect if the informative signal only exists after generation begins. And another developer in a thread I was following measured his own confidence scores against real misses and found 95.8 percent of the misses sat at confidence 0.9 or above. That is his data, not mine, but it matches the general shape: confidence is well calibrated to fluency and poorly calibrated to correctness.
The honest synthesis is probably that both are true in sequence. The question alone does not carry difficulty, and the model's stated confidence does not carry correctness, so the only place left to look is the draft itself and whether it survives a check.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.