srinivas reddy gouru

Posted on Oct 22

How a Self-Learning Router Can Cut Your AI Bill

#ai #llm #machinelearning

Pick the right model for every request by learning from outcomes: scoring, Thompson sampling, a safe architecture, guardrails, and an honest list of pros and cons.

Most teams pick one LLM, hardcode its name, and pay for that choice on every single request. Some go further and write rules: "send 70 percent here, 30 percent there", "use the cheap model unless it is down". Then a vendor changes its prices, a model quietly gets worse after an update, or the traffic shifts from chat to code, and the rule keeps choosing exactly as before. Nobody is told.

This article explains a different approach: routing that learns from outcomes. The gateway in front of your models scores every response, keeps a belief about how well each model does for each kind of work, and lets those beliefs choose the next model. It can send most traffic to the cheapest model that is doing the job, and keep checking that it still is.

The idea is simple. Making it safe, honest and useful is most of the work, and that is what this piece walks through, including the parts that do not work as well as the idea suggests.

TL;DR

Score every response on what it cost (speed, price, completeness) and, when you can measure it, whether it was right.

Keep a belief per model, per customer and per type of work, not a single average.

Choose with Thompson sampling: draw a plausible score from each belief and pick the highest. Exploration comes for free.

Keep the learner off the request path, and never let it choose a model the caller is not allowed to use.

The catch: without a quality signal it optimises cost and speed, not correctness. A fast, cheap, confidently wrong model can win.

All numbers and model names in this article are illustrative, chosen to explain the mechanism.

The junction problem

On a railway, the points are the movable rails at a junction. Whoever sets them decides which track a train takes next. A gateway that sits in front of several language models does the same job thousands of times a minute: one request arrives, several models could answer it, and something has to throw the switch.

Most teams now have more than one model that can serve a request, and the candidates differ in ways that matter: price per token, speed, how often they time out, and how good they are at a particular kind of task. A model that is excellent at writing code may be mediocre at factual questions. A model that is cheap today may be repriced next month.

The usual answer is a fixed rule:

Priority: always use the first model and keep the others for failover.
Weighted split: send 70 percent here and 30 percent there.
Round robin: rotate evenly.
Fastest or cheapest: pick whichever scores best on one live measurement.

Each of these is a judgement frozen at the moment someone wrote it down. None of them hears back about how its choices turned out.

Adaptive routing closes the loop: score every response, update a belief, and let the next choice follow the beliefs. Everything below is about making that loop trustworthy.

Five words to know

The technique comes from a branch of statistics called multi-armed bandits, named after a row of slot machines ("one-armed bandits"), each paying out at an unknown rate. You learn the rates only by pulling the levers, and every pull spent learning is a pull not spent on the machine you currently think is best.

Arm: one thing the router can choose, usually a provider and model pair.
Reward: a score between 0 and 1 that says how well one response went.
Belief: a range of plausible values for an arm's typical reward, not a single number.
Policy: the rule that turns beliefs into a choice for the next request.
Exploration: choosing an arm you are less sure about, to learn whether it is better than it looks.

Always picking the arm that looks best right now means you never find out whether another arm is better. Picking at random learns a lot but wastes traffic. A good policy balances the two automatically.

Scoring a response

Before anything can be learned, the router needs to decide what a good response is worth. This is the most important design decision in the whole system, because the router will optimise exactly what the reward measures and nothing else.

Think of two sides of the same response:

The consumption side: what the answer cost to get. Latency against a target, cost compared with the other candidates that were available for this request, and a check that the response was complete rather than truncated or malformed.
The answer side: whether the answer was actually good. This comes from feedback, a validator, or a judge model.

The consumption side is always available, because the gateway measures it itself. The answer side is only available when someone tells the gateway, and that asymmetry shapes everything that follows.

Rules that keep the score honest

Score cost relative to what was on the table. "Expensive" should mean expensive compared with the other candidates at this request's actual size. A long prompt is costly on every model; that is not the model's fault.
Leave missing signals out. If a provider did not report token counts, that check is absent from the score, not counted as zero. Treating "unknown" as "bad" punishes requests for merely lacking a signal.
A failure scores zero, full stop. Otherwise a model that fails quickly would look attractive.
Learn cost only from trustworthy numbers. If the usage a provider reported is inconsistent, or had to be estimated, or the response came from a different model than the one you asked for, count it for reliability but do not learn anything about price from it. A router that learns from bad bills will happily prefer whichever model reports its usage incorrectly.

One knob: how much quality is worth

When both sides are present, a single preference decides how they combine. At a preference of 0.3, a rated response scores 0.3 times its answer side plus 0.7 times its consumption side. Low values favour efficiency; high values favour correctness.

Model A is fast and inexpensive with good accuracy. Model B is slower, costs far more, and is more accurate. Below a preference of about 0.84 the router prefers A, and that is correct: it was asked to weight efficiency, and it did. This is where the cost saving comes from.

Two consequences are easy to miss:

Unrated traffic is scored on consumption alone, whatever the preference. If nothing ever tells the router whether answers were right, a preference of 0.9 routes exactly like 0.1. Raising the knob without supplying a quality signal changes nothing, so watch the share of traffic that actually receives a rating.
Late feedback needs its own weight. A rating usually arrives after the response was already scored. It travels as a separate observation, weighted by p / (1 − p) where p is the preference: at 0.3 a rating counts as 0.43 of an observation, at 0.75 it counts as three. Multiplied by a feedback rate limit, the same weight caps how much any one client can steer the router.

Why default to efficiency? The two possible mistakes are not symmetric. A default that leans too cheap produces answers someone notices and dials up. A default that leans too expensive produces a bill nobody can undo.

Beliefs, not averages

An average hides how much evidence stands behind it. A model that scored 0.9 on five requests and a model that scored 0.9 on five thousand are not the same finding.

A common way to keep both pieces of information is the Beta distribution, described by two numbers:

α (alpha): roughly, the good outcomes seen so far, plus one.
β (beta): roughly, the bad outcomes seen so far, plus one.

Rewards are fractional, so a response that scored 0.9 adds 0.9 to α and 0.1 to β. The ratio α / (α + β) is the typical score. The sum α + β is the confidence: it grows with every observation, and the belief narrows as it grows. Every arm starts at α = 1, β = 1, a flat belief, because nothing has been seen.

Model C's centre looks poor, but six observations are close to a coin toss, and the width of its curve says so. Two arms can show the same mean and mean very different things: a narrow curve is a measurement, a broad one is a guess.

Thompson sampling: let uncertainty pay for exploration

The tempting policy is to pick the arm with the highest mean. It is wrong for a simple reason: an arm you stop choosing stops producing evidence, and its belief freezes wherever it happened to be. If it was unlucky early, you will never find out it was good.

Thompson sampling fixes this in one step. For each candidate, draw one random value from its belief. Route the request to whichever arm drew the highest value.

A narrow belief draws values close to its mean almost every time.
A wide belief sometimes draws high, so an uncertain arm occasionally wins on the strength of its uncertainty.
When it wins, it gets traffic and its belief narrows. If it really is better, it starts winning on its mean; if not, it fades.

10,000 simulated requests drawn from the three beliefs above. Model B still gets about one request in nine, because its wide belief often reaches past A's. Model C almost never wins: its belief is wide but sits mostly below A's. Uncertainty buys traffic only when the plausible range overlaps the leader's.

Exploration is not a separate mode switched on and off. It falls out of drawing from a width, and it shrinks as evidence accumulates. The draw costs a couple of random numbers per candidate, computed inside the gateway with no network call.

Log the probability each chosen arm had of being chosen. That small detail unlocks off-policy evaluation: replaying past traffic to estimate what a different policy or reward would have earned, before you ship it.

Learning about the right thing

A model that is excellent at code and average at trivia has no single score, and a belief that averages the two is wrong about both. So beliefs are kept per sampling key, typically the customer joined to a task type:

Per customer, so one tenant's traffic never teaches the router about another's.
Per task type (chat, code, summarise, extract, reasoning, factual knowledge), so models are compared on like-for-like work.

The task type can come from the caller, who usually knows what a request is, or from a cheap classifier. Keep the vocabulary closed: every distinct value becomes a set of beliefs the learner must store, so letting callers invent labels lets them grow its memory without bound.

What goes wrong when workloads are mixed. Imagine Model X is clearly best at factual recall, Model Y is clearly best at commonsense reasoning, and Model Z is decent at both and cheap. If both kinds of traffic share one belief, the averages wash out each specialist's strength and Z can take most of the traffic. Split the work into separate keys and X and Y each win where they are strong. Nothing about the models changed; only the key did. Telling the router what kind of work a request is may be the single most effective thing you can do for it.

Contextual bandits go further and condition the belief on features of the request itself, such as prompt length or the presence of code. A popular variant, linear Thompson sampling, keeps a belief over a small linear model of reward instead of one number per arm. It captures patterns that task labels miss, at the price of needing more data and being harder to explain.

Cold starts, curiosity and change

Starting cold. A new customer or task type has no evidence, so the first decisions are effectively coin flips. Make it visible: label each decision as learned, cold or exploration, so a dashboard can never present degraded routing as learned routing.

Starting warm. Beliefs can be seeded from an offline evaluation, as a prior mean worth a stated number of observations. A mean of 0.9 at a strength of 10 becomes α = 10, β = 2: a belief that leans, but that ten real observations can move. Keep the strength low, because a benchmark measures a different corpus at a different moment, and report seeded and live evidence separately.

Never fully giving up. An arm with a thousand bad observations will practically never draw high again, which is correct until the vendor fixes the model. A small exploration floor, say 5 percent of decisions picked at random, keeps a recovered arm findable. It is shared, though: with five arms, each losing arm gets about one percent, and every new arm thins every share.

When the world moves. A belief built on thousands of old observations moves slowly, so a router can keep avoiding an arm long after it recovered, or keep favouring one after it got worse. Gradual forgetting, change detection that resets a belief when the reward stream shifts, and the exploration floor all help. Each adds tuning: forget too quickly and the router chases noise; too slowly and it lives in the past.

An architecture that cannot fail traffic

The design choice that matters most is to separate the part that chooses from the part that learns, and let trust flow in one direction only.

Sampling happens locally. The gateway holds the latest published beliefs in memory and draws from them in-process. There is no synchronous "decide" call to fail or add latency.
The learner is polled, not called. If a poll fails, the gateway keeps what it had. If the learner stops entirely, routing continues on the last beliefs it published (or flat ones), and only learning pauses.
The gateway mints the decision id. The eventual reward carries the decision, the key and the arm that served, so attribution is structural and nothing can go stale on the learner side.
One writer for beliefs. Several learner workers holding beliefs in memory drift apart. Run a single writer, and refuse to start otherwise.
Two implementations must agree. If the gateway and the learner are written in different languages, test statistically that the gateway's sampler draws like the learner's. A sampler that draws wrongly explores wrongly, and invisibly.

Guardrails around the learner

The most important rule: the learner may choose, but it may never widen what is allowed.

Eligibility comes first. Before the sampler sees a candidate, everything that may not or cannot serve this request is removed: models the caller's key is not allowed to use, models missing a needed capability or with too small a context window, providers with an open circuit breaker or in a rate-limit cooldown, and anything that would break a budget. A belief, however confident, can never bring back a candidate a filter removed.

Failure handling comes after. If the chosen model fails, a fallback ladder re-checks live health at every step: the same model elsewhere, declared fallbacks, the same family or tier, then a tier up or down. The learner changes the order in which healthy candidates are tried, never the permissions.

Composing with everything else:

Conversation stickiness. Providers bill a repeated prompt prefix at a large discount, but only when consecutive turns land on the same provider. Pinning a conversation saves money, with an honest trade-off: it will not move to a newly preferred model until the pin lapses.
Deprecation as a weight. Multiply a retiring model's sampled score down at selection time instead of punishing its reward, so undoing it is instant.
Trust the evidence before learning from it. Check that the model that answered is the one you dispatched, validate reported usage, and keep unverified answers out of cost learning and out of any response cache.

A stricter sibling: proven quality

Learning on live traffic suits workloads where an occasional weaker answer during exploration is acceptable. Some cannot accept that. For those, quality-guarded routing serves a cheaper model only where there is statistical evidence it is good enough:

The evaluation is fully specified before anyone looks at results: task set, rubric, baseline, required quality and margin, confidence level.
Results are judged with exact confidence bounds, corrected for testing several models at once, so a model qualifies only if even the pessimistic end of its range clears the bar.
Evidence is bound to the exact model version and expires. If the model behind a name changes, qualification lapses.
Among qualified models the cheapest wins; with none qualified, a declared baseline serves. Never a random model.

	Learning router (bandit)	Quality-guarded router
Learns from	Every live response, continuously	A sealed offline evaluation
Optimises	A blend of cost, speed and quality	Price, among models proven good enough
Explores on live traffic	Yes, by design	No, or only among qualified models
Without evidence	Coin flips among eligible models	The declared baseline, or a clear refusal
Reacts to change	Automatically, with some lag	Only when a new evaluation runs
Best for	High volume, measurable outcomes	Regulated or contractual quality bars

Advantages

It adapts without anyone editing configuration. Repricing, a quiet regression, a new model in the group: the scores notice and traffic shifts.
It can cut cost where cost is safe to cut. At a low quality preference, the cheapest model that is measurably doing the job takes most of the traffic, and keeps being re-checked.
It learns from your traffic, not a leaderboard. Beliefs per customer and per task mean it learns which model works for your code and your summaries.
Trade-offs become explicit. One preference states how much quality is worth against cost and speed, per task if needed.
Exploration is built in and bounded. No separate A/B test to plan and dismantle.
It does not overreact to small samples. Five lucky responses do not crown a model, because beliefs carry their width.
It is safe by construction. Choosing is local and fast; a learner failure pauses learning and nothing else.
Decisions are explainable. Each choice records its beliefs, its probability, and whether it was learned, cold or exploratory.
It enables "what if" questions. Logged probabilities let you evaluate a new reward or policy offline before going live.
It gives feedback somewhere to go. A static router has nowhere to put a quality signal.

Disadvantages and limits

Without a quality signal, it optimises efficiency, not correctness. A fast, cheap, well-formed and confidently wrong model scores well on everything a gateway can measure by itself. This is the most dangerous failure mode, and it is invisible unless you watch the rated share of traffic.
Good quality signals are slow, sparse or costly. Machine-checkable outcomes are ideal but rare. A judge model costs money per judgement, and the bill grows with the number of models and how small a difference you want to detect, not with traffic volume.
Exploration costs real requests. Some users get answers from models the router already suspects are worse.
Cold start is a coin flip. Seeding helps only as far as offline data resembles real traffic.
Confident beliefs are slow to change. A recovered model can keep losing; a regressed favourite can keep winning.
The exploration floor thins as models are added. Each losing arm's share shrinks, and noticing a recovery takes longer.
Choosing the key is a balancing act. Too coarse averages away real differences; too fine starves each belief of evidence.
Routing is no longer deterministic. The same request can go to different models on different days, which complicates debugging and consistency.
The reward is a value judgement. The router optimises exactly what it is told to, including a badly chosen latency target or cost weight.
Feedback can be gamed. Weighting, rate limits and per-tenant keys have to be designed in.
It is more to operate. A second process, a belief store, a single-writer rule, parity tests and dashboards.
It competes with prompt caching. Moving a conversation to a newly preferred model forfeits the cached-prefix discount.
Honest evaluation is hard. Re-scoring old responses shows the direction of a change; a live run would collect different evidence.
Untrustworthy data must be thrown away. Inconsistent or unverified usage cannot teach cost, so the router learns from fewer observations than it serves.

When to use what

Situation	A sensible choice	Why
One approved model, or a mandate on which model serves	Fixed routing	Nothing to learn; predictability is the requirement
A canary or an A/B test	Weighted random split	The split you declare is the split you get
High volume with machine-checkable outcomes	Learning router with feedback	Free, accurate quality signals and plenty of evidence
Open-ended chat with no quality signal	Learning router at a low preference, or a latency/cost rule	It will optimise efficiency; say so
Very different kinds of work through one endpoint	Learning router with task types, or a contextual bandit	Specialists are not averaged away
A contractual or regulated quality bar	Quality-guarded routing with a baseline	Cheaper models only after statistical evidence

Adoption checklist

Start with comparable candidates. The router chooses; it should not have to rescue bad options.
Declare every price. A model with an unknown price must not be scored as free.
Decide your task types and send them. The cheapest large improvement to belief quality.
Start with a low quality preference. Raise it per task only where a quality signal actually arrives.
Wire up the cheapest honest quality signal. Parsers, schema checks, test runs and label matches beat judges wherever they apply.
Seed lightly, if at all.
Watch the right dashboards. Decision provenance, traffic share per model, belief width, and the rated fraction.
Plan for change. Choose an exploration floor and a forgetting policy deliberately.
Evaluate before you re-tune. Use logged probabilities to test a new reward offline.
Keep the guardrails non-negotiable. Eligibility before choice, fallback after it, and no path by which the learner can grant access.

The claim, without the marketing

Out of the box, a learning router is an efficiency-optimising router. For each customer and each kind of work it learns which model is fast, affordable and reliable, and it keeps relearning as those facts change. That alone beats a weight someone typed once and forgot, and it is where most of the savings come from.

Where you can say what a good answer is, and tell the router, it becomes an outcome-optimising router, and the quality preference states how much that definition is allowed to cost. Where a quality bar must be proven before anyone relies on it, a guarded router built on sealed evaluations is the better tool.

The switch at the junction was never the hard part. Knowing which track was right, and admitting how sure you are, is the whole job.

Are you routing between several LLMs today? I would like to hear how you decide, and what has burned you, in the comments.

DEV Community