If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do j...
For further actions, you may consider blocking this person and/or reporting abuse
The paragraph that saves this playbook is "Routing without measurement is just hope with extra steps" - most routing posts stop before it. Two questions from the trenches: (1) your signal.confidence > 0.9 gate - calibrated against what? A confidence threshold nobody has fed known-hard-disguised-as-mechanical inputs is itself an untested reviewer, and its failure mode (cheap answer, looks fine, quietly wrong) is invisible in pass rates by construction. Do you track the pre-classifier's confusion rate separately? (2) The 61/22/11/6 split is the most useful number in the post - is that one audited pipeline or a pattern across several? If several, that distribution deserves its own write-up.
Spot on — and the calibration point is the real gap, I'd say. The 0.9 gate was tuned on a held-out set of known-hard cases we'd labeled manually, but you're right that "known-hard-disguised-as-mechanical" never had its own pinned slice, so the gate's failure mode is exactly the invisible one you describe. We track per-tier error rates, but not the pre-classifier's confusion matrix on its own — that's a blind spot worth closing.
On the split: it's across several pipelines, not one audited run, and the shape holds within ~5 points pipeline-to-pipeline. You're right it deserves its own write-up — the useful part isn't the exact numbers, it's that the distribution is stable enough to route against. Happy to dig into the methodology if there's interest.
Explicit interest registered - please write the methodology up. Two requests for when you do: include the per-pipeline variance you mention (the ~5-point stability is the load-bearing claim, more than any single number), and the labeling rule for "known-hard" - that's the part others can reject or reuse. We went through the same exercise for retrieval failure classes, and publishing the definitions changed the conversation from vibes to reusable splits within a week. The write-up is worth more than the numbers in it.
On the labeling rule for known-hard — the trap we hit: known-hard is easy to over-include. Anything a reviewer struggled with gets tagged known-hard, but struggle ≠ genuinely hard. We now require two independent labelers and only model the intersection (cases both flagged); the disagreement band goes to "uncertain" rather than forced into a label. That's also why your ~5-point stability number holds — a single labeler's known-hard set swings more than that, so the variance claim only survives on the agreed core. Publishing the disagreement band next to the definition is what made it reusable for others: they adopt your core and bring their own edges.
"The curve tells you if the reviewer regressed; slice freshness tells you whether you can still trust the curve" - that's the sentence I'd steal back. It's the same failure class as a guard pinned to a fixed number: it keeps guarding the past faithfully. We had a watcher assert an exact count once; it stayed green for weeks while the world it was counting had moved on. Green, and measuring nothing.
Concrete question: what does slice freshness measure for you in practice - label age, divergence of the slice's input distribution against current production, or the share of slice cases that still occur in live traffic at all? The three drift at different speeds, and I suspect the third one is the one that bites first.
Glad that line earned its keep — and your "steal it back" is fair; it came out of your own pinned-slice point on the reviewer thread.
To the concrete question: we track all three as separate signals and never collapse them, because the remediation differs for each. On which bites first, your instinct is right for the steady-state regime, but I'd add a regime caveat:
Why three separate and not one score: the fix for each is different (re-label / re-sample / retire-or-replant). The moment you compress them you rebuild the exact failure you're describing — a metric that looks healthy while its components rot. Which is also why your sharpening from the other thread — calibration curve per release, not a raw average — is the right sibling signal: the curve tells you if the reviewer regressed, slice freshness tells you whether you can still trust the curve. Skip either and the discipline slowly rots.