DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

A Cost-Aware LLM Router That Cut a Real Batch's Bill by 78.5%

#ai #agents #llm #python

Not every question deserves your most expensive model. Sending every query to the frontier tier is simple and ruinously expensive — a trivial lookup and a rigorous proof cost the same. Project 7 of Agentic AI from Zero builds a router that spends the least money that still solves the task: it sends a trivial lookup to one cheap call, escalates a hard proof to best-of-3, exits early the moment a cheap answer is confident, and degrades rather than blow a budget. On a real six-query run against NVIDIA NIM it cost $0.039971 instead of $0.185790 — 78.5% saved, measured.

The price ladder

Route by effort, not by model name. Three tiers, each with a published $/1k rate, where escalate is ~60× the price of direct — which is exactly why routing pays off:

PRICE_TABLE = {                                       # cheapest → dearest
  "direct":   Tier("direct",   0.0005, 1, "one terse call — a cheap model"),
  "cot":      Tier("cot",      0.0030, 1, "one call that reasons step-by-step"),
  "escalate": Tier("escalate", 0.0300, 3, "best-of-3, priced like a frontier model"),
}
LADDER = ["direct", "cot", "escalate"]
EARLY_EXIT_CONFIDENCE = 0.75
Enter fullscreen mode Exit fullscreen mode

Classify for free, then route

A cheap keyword heuristic — no API call, $0 — scores each query trivial / medium / hard and maps it to an entry tier (direct / cot / escalate). The model never picks its own tier.

The route loop: classify → budget → run → gate

The whole policy is deterministic Python. Classify, step down the ladder if the budget can't afford the tier you wanted, run a real call, and check the confidence gate — early-exit if confident, escalate one step only if unsure and the budget allows:

comp = classify_complexity(query)                  # $0, deterministic
want = entry_tier(comp.label)
tier = budget.best_affordable_at_or_below(want, query)   # step DOWN if too dear
if tier is None: return refused                    # can't afford even `direct`
degraded = tier != want                            # forced cheaper than we wanted

while True:
    att = self._run_tier(query, tier)              # a REAL NIM call (usage → $)
    if att.confidence >= EARLY_EXIT_CONFIDENCE:    # confident enough?
        early_exit = tier != "escalate" and not degraded; break   # STOP — don't pay more
    nxt = next_tier(tier)
    if nxt is None: break                          # already at the top
    if not budget.affords(spent_c, spent_t, estimate_cost(query, nxt), ...):
        degraded = True; break                     # DEGRADE — budget says no
    tier = nxt                                      # escalate one step
Enter fullscreen mode Exit fullscreen mode

The split is the whole point: the model only answers, reasons, and self-rates its confidence. It does not pick the tier, set the budget, decide the early-exit, or compute a price. When escalate runs best-of-N, even the pick is deterministic — the model proposes N diverse samples and Python keeps the most confident one.

The four things one run demonstrates

  • Token budgeting — every query carries a hard $ ceiling; query 5's tight $0.0040 budget blocked the escalate it wanted, so it degraded to CoT rather than overspend.
  • Route by complexity + cost — the heuristic scored and mapped every query with no API call.
  • Early exit on confidence — 3 of 6 queries stopped at the cheap tier because self-rated confidence cleared 0.75; the router never paid to escalate them.
  • Cost-per-decision analytics — the ledger prices every query from real usage tokens and reports the saving against a measured baseline: the frontier tier was actually run on all six queries, so "78.5% saved" is a measurement, not a claim.

And an honest miss is left in: on the "17 sheep, all but 9 run away" riddle, the model answered 8 (correct is 9) and self-rated it 1.00, so the router early-exited on a confidently-wrong cheap answer. Confidence-based early-exit trades a small quality risk for a large cost saving, and it's only as good as the model's self-calibration — which is exactly why the budget and the audit ledger live in code the model can't move.

Walk the real routing pipeline stage by stage, see the cost ledger and the measured baseline, and grab the repo here: https://dev48v.infy.uk/agentic/project7-cost-router.html

Next up, Project 8: an event-triggered automation agent.

Top comments (0)