I built the intent router that sits at the front of our support agent. It reads what a user typed and decides which pipeline handles it. I tested it on a nice balanced set, got 92 percent accuracy, and shipped it feeling good.
Two weeks later the on-call engineer noticed cancellation requests were landing in the billing queue. The fraud-report queue was empty while actual fraud complaints sat in product feedback. And a newer request type that did not exist when we launched was being silently filed as a billing question most of the time.
The model had not broken. My test set had simply hidden the failures that actually happen in production. The 92 percent was real and useless at the same time.
Here is what I learned pulling that apart, in plain terms, so you can skip the two weeks I spent finding out the hard way.
Why the headline accuracy number lied to me
Real support traffic is lopsided. Five or six common intents (product questions, pricing, account access, shipping, refunds) make up most of the volume, and they are easy. The rare stuff (cancel subscription, fraud report, policy dispute, a brand-new request type) is a small slice of traffic and it is exactly where the money, the on-call pages, and the trust-and-safety incidents live.
A balanced test set gives every intent equal weight. So the model learns to do fine on all of them evenly, and my one big accuracy number was dominated by the easy, high-volume classes. The rare classes could quietly collapse to 30 percent and the headline barely moved, because they were a rounding error in the average.
And every one of those rare classes matters more than its volume suggests. Whatever the router picks decides which knowledge base, which tools, and which escalation path runs next. Get the routing wrong and everything downstream is confidently wrong.
Two test sets, not one
The fix that changed everything for me was keeping two separate test sets:
- One that mirrors real traffic. Same lopsided mix as production. This is the number I report, because it is what users actually experience.
- One that oversamples every intent evenly, a hundred or so examples per class no matter how rare. This is the one I debug against, because it is the only way to see whether the model can even do the rare classes at all.
The gap between those two is the warning sign. If the model is great on the real-traffic set but weak on a rare class in the even set, that class is one traffic shift away from becoming my next incident.
Look at each intent, not the average
Once I stopped looking at one number and started looking at each intent separately, the problems were obvious. For every intent I now look at two things:
- How often is it right when it fires? If this is low, the model is over-using that intent as a dumping ground when it is unsure.
- How often does it catch the real cases? If this is low, the model is quietly sending those cases to some neighbouring intent instead.
The single most useful view is the grid of what got confused with what. That grid told me plainly that "cancel" requests were mostly being read as "billing." Once you can see the exact pair that is colliding, the fix is usually two or three edits to how you describe those two intents to the model, not some big model change.
When one intent is both rarely right and rarely caught, that is the danger zone: either the description is ambiguous, the label itself is badly defined, or the model genuinely cannot tell it apart from its neighbour. The grid tells you which.
The confidence score means nothing until you calibrate it
My router also spat out a confidence number, and I had wired the escalation logic to trust it: below some cutoff, hand off to a human. The problem is that the raw confidence was close to meaningless.
Two things surprised me. The scores clustered at the extremes, either very high or very low, with almost nothing in the middle to set a cutoff against. And the same number meant different things for different intents. A 0.7 on a common product question was right almost every time, while a 0.7 on a fraud report was barely better than a coin flip.
So one global cutoff was shipping a totally different level of caution per intent. What actually works:
- For irreversible actions like cancelling a subscription or flagging fraud, set a high bar and send anything below it to a clarifying question or a human.
- For catch-all safety intents, set a low bar so the system over-routes to the safe path instead of guessing.
And treat a change to those cutoffs like a change to the model itself. Version them together.
Give the model a "none of these" option
This was the one I had completely missed. If you do not give the model an explicit "none of these" class, it is forced to jam every input into some known label. Gibberish, off-topic questions, and outright jailbreak attempts all get filed as a real intent and routed into a real pipeline.
So I added an explicit "does not fit any intent" class and trained it on the messy stuff: adversarial prompts, vague multi-topic messages, plain nonsense. Two rules made it work:
- If nothing clears its confidence bar, fall through to this class instead of picking the best guess.
- Never let this class dead-end. Route it to a "which of these did you mean" question or a human, not into a pipeline.
I also watch this class's catch rate closely and never let it get worse. A drop there means real garbage is leaking into real flows again.
The taxonomy you ship goes stale
The list of intents I launched with was not the list production needed three months later. User language drifts, a product launch creates a brand-new request type, a policy change creates a new kind of complaint. That new request type being misfiled as billing was exactly this.
Two things I watch for now:
- An existing intent slipping. If its numbers move week over week, the model is fine but the incoming language shifted under it. Refresh its examples and re-baseline.
- A new intent forming. I group together the stuff that landed in the "none of these" bucket and the low-confidence misses, and look for a cluster that keeps showing up. If the same new pattern is there two weeks running, it has earned its own intent: label some examples, write a definition, add it.
Two things I wish I had known on day one
- Define intents by what the user wants, not what your system does. "User wants a refund" is an intent. "Run the refund flow" is a tool. I had mixed the two and it muddied the labels.
- Check that humans agree on the labels before blaming the model. Have a few people label the same hundred examples. If they cannot agree on an intent, the model has no chance, and that is a taxonomy problem to fix first, not a model problem.
The thing I keep coming back to is that the intent router is the first decision the whole agent makes, and I had been grading it with the one number least likely to show me its real failures. The moment I looked per intent, on real traffic, with a "none of these" escape hatch, the bugs that had hidden for two weeks were sitting right there in the grid.
If you run a router like this, I would love to hear which intent quietly fell apart on you. For me it will always be cancellations pretending to be billing questions.
Top comments (0)