An AI application can produce a clear, confident response after sending a request to the wrong workflow.
Consider these messages:
- Where is my order?
- What is your refund policy?
- I want to request a refund.
All three concern customer support, but each requires different handling.
For Day 15 of my AI engineering journey, I’m exploring deterministic intent routing and how to measure routing accuracy on labeled requests.
What makes a router deterministic?
A deterministic router follows explicit decision rules. Given the same input, application context, and rule version, it returns the same route.
That makes its behavior easier to reproduce and debug. When something goes wrong, there is a decision path to inspect.
However, reproducibility does not guarantee accuracy. A poorly defined rule can consistently send requests to the wrong handler.
Start with clear intent definitions
Before implementing a router, define the routes and their boundaries.
For a small support application, the specification might look like this:
| User request | Expected route |
|---|---|
| Track my order | order_status |
| Explain your refund policy | refund_policy |
| Request a refund | refund_request |
| Track my order and request a refund | clarify |
| Write a birthday poem | unsupported |
This example assumes one intent per workflow. A system that supports multiple intents could handle the fourth request differently.
The goal is to make the expected behavior clear enough that another developer could label the same requests consistently.
Why keyword matching needs careful evaluation
A rule that checks for the word “refund” might seem useful until it receives:
“Do not refund my order. I only want to understand the policy.”
The keyword is present, but the user explicitly rejects the action.
A useful evaluation set should therefore include:
- Different ways of expressing the same intent.
- Negation and overlapping vocabulary.
- Requests containing multiple intents.
- Short follow-ups that need conversation context.
- Requests outside the application’s capabilities.
These examples reveal weaknesses that straightforward test messages can miss.
What the research tells us
Anthropic’s Building Effective Agents describes routing as classifying an input and directing it to a specialized task. It presents routing as useful when request categories are distinct and classification can be performed accurately.
That supports separating workflows, while leaving the classification method open to the application’s needs.
Another useful reference is Larson et al.’s An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction.
The paper introduces a dataset covering 150 intents across 10 domains, including out-of-scope requests. Its evaluated classifiers performed well on supported intents but struggled to identify unsupported queries.
The practical lesson: recognizing a known intent and recognizing an unsupported request need separate attention.
Measuring routing accuracy
Create a labeled dataset containing the request, relevant context, and expected route.
The basic calculation is:
Routing accuracy = correctly routed requests / total labeled requests
For example, 90 correct decisions out of 100 requests would mean 90% accuracy.
This is an illustrative calculation, not a measured result from my project.
Overall accuracy is a starting point. It does not explain which routes fail.
Look beyond the overall score
| Measure | What it reveals |
|---|---|
| Per-intent precision and recall | How reliably each intent is recognized |
| Confusion matrix | Which intents are being mixed up |
| Unsupported-request rejection rate | Whether unsupported requests reach the fallback |
| Clarification rate | How frequently the router requests more information |
| Automatic-routing accuracy and coverage | How reliable direct decisions are and how many requests receive them |
Coverage matters because a router could achieve high accuracy on direct decisions by sending almost everything to clarification.
Keep development examples separate from a held-out test set. Otherwise, improving the rules may simply mean adapting them to requests you have already seen.
Connecting this to Day 14
Yesterday’s topic was tool permissions. Routing introduces another decision before tool execution.
Selecting refund_request should enter the refund workflow. Authentication, eligibility checks, and any required confirmation still happen before execution.
The route identifies what the user appears to need. The workflow determines whether and how the action can proceed.
What I would build first
For this challenge, I would start with a small router, explicit fallback behavior, and a labeled evaluation set.
Then I would inspect the mistakes:
- Are the intent definitions unclear?
- Are rules missing common paraphrases?
- Does the request require more context?
- Are unsupported requests being forced into supported routes?
Those findings provide a practical basis for deciding whether to improve the rules or evaluate a classifier.
Do you measure routing accuracy separately from the quality of your AI application’s final response?
Top comments (1)
References
Larson et al. (2019). An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. EMNLP-IJCNLP.
Anthropic (2024). Building Effective Agents.
CLINC150 dataset and evaluation repository.