DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Prompt routing: classify the message first, then dispatch it to a specialist that actually fits

#ai #promptengineering #llm #beginners

A single do-everything prompt that must answer billing questions, debug code, and make small talk is carrying every instruction and every tool at once — so it is bloated, pricey, and only mediocre at each job. Worse, it can never tell you a message is ambiguous: it just picks an interpretation and confidently runs with it. Prompt routing fixes both problems with two moves: classify the input, then dispatch it to the small specialist built for that case. Here is the pattern I built a demo around.

The thing you are replacing

The tempting design is one prompt for everything, because it is a single call:

// ❌ billing + code + chat + every tool, in one prompt. Jack of all trades.
async function doEverything(msg) {
  return llm(`You handle BILLING (refunds, charges, invoices),
             CODE bugs (errors, stack traces, APIs), AND general chat.
             You have every tool. Read the message and do whatever is needed.
             User: ${msg}`);
}
Enter fullscreen mode Exit fullscreen mode

It is bloated, and it cannot flag "which of these did you actually want?"

Routes are small, focused specialists

The heart of routing is a registry: one entry per route, each a sharp prompt with only its own instructions and tools. The billing specialist never carries debugging know-how; the debugger never sees the refund tool. Small prompts are reliable and cheap.

const ROUTES = {
  billing: { sys: "Billing specialist. Tools: ledger.lookup, refund.issue." },
  code:    { sys: "Senior engineer. Diagnose stack traces, give a concrete fix." },
  chat:    { sys: "Friendly general assistant for greetings and how-tos." },
};
const runSpecialist = (route, msg) => llm(ROUTES[route].sys + "\n\nUser: " + msg);
Enter fullscreen mode Exit fullscreen mode

The router is a tiny classifier

The router's only job is to pick a label. Constrain it to the fixed route set, make it return machine-readable JSON with a confidence and a short reason, and give it an explicit "fallback" option for when nothing fits. Use a small, cheap, fast model here — this call runs on every request.

async function classify(msg) {
  const out = await llm(
    `Classify the message into ONE of: billing | code | chat.
     Reply as JSON: {"route":"...","confidence":0-1,"reason":"..."}.
     If none fit or you're unsure, use route "fallback".
     Message: ${msg}`);
  return JSON.parse(out);   // { route, confidence, reason }
}
Enter fullscreen mode Exit fullscreen mode

The whole safety story: gate on confidence AND margin

Dispatch is a lookup — but gate it first. If the router returned "fallback", or its confidence is under your threshold, or the top two routes are too close, do not guess a specialist. Ask a clarifying question or escalate to a human. This gate is what stops a router from confidently sending an ambiguous message to the wrong expert.

const CONF_MIN = 0.6, MARGIN_MIN = 0.34;
async function dispatch({ route, confidence, margin }, msg) {
  if (route === "fallback" || confidence < CONF_MIN || margin < MARGIN_MIN)
    return clarifyOrEscalate(msg);   // the safety net — never guess
  return runSpecialist(route, msg);
}
Enter fullscreen mode Exit fullscreen mode

In the demo, the message "It's broken and I want my money back for this thing" is the interesting one. "broken" pulls toward code, "money back" pulls toward billing, the two tie at 50%, the margin is zero — so instead of bluffing, the router hands off to the fallback and asks which one the user meant. A single do-everything prompt, by contrast, silently opens a technical ticket and gets the outcome wrong.

Put it together — and keep the decision

The whole router is two calls: classify, then dispatch. Keep the decision around, because the route, confidence and reason make routing auditable — when an answer is off, you can see whether it was a misroute (wrong route chosen) or a bad specialist answer.

async function handle(msg) {
  const decision = await classify(msg);
  console.log(decision.route, decision.confidence, decision.reason); // audit trail
  return dispatch(decision, msg);
}
Enter fullscreen mode Exit fullscreen mode

A route's handler can itself be a whole chain, so routing and chaining compose cleanly: classify to the right pipeline, then run it.

Know when to route

Route when your inputs fall into distinct kinds that each need different handling — different instructions, tools, or a different chain. For one uniform kind of input, a router is just an extra hop of latency. The real costs are exactly that: the classify call adds a hop, and a misroute propagates into a confidently wrong answer. So keep the route set small, always keep a fallback, use a cheap model to classify, and log the routes to catch drift. That is the difference between a switchboard of specialists and one impossible prompt trying to be all of them.

Pick a query or type your own, then step through classify → gate → dispatch and watch the router choose (including the fallback):
https://dev48v.infy.uk/prompt/day36-routing.html

Top comments (0)