i built a tool that tracks what AI tasks actually cost. the real number surprised me.
you know how much your LLM costs per token. you probably don't know what it costs per task.
I didn't either. that's the embarrassing part. I've been routing LLMs across tiers for weeks — writing about it, presenting it — and I was tracking per-call cost like everyone else. a $0.001 call here, a $0.05 call there. neat numbers. wrong numbers.
here's the real math. you route a code task to your cheap workhorse model at $0.001 per call. it fails. you retry. it fails again. third try, still failing. you escalate to your architect model at $0.05. that one works. what did the task cost?
your dashboard says $0.001. or maybe $0.05. neither is right. the task cost $0.053 — three failed cheap calls plus one expensive call. that's the real number. and I couldn't find a single tool that tracks it.
so I built one. ai-tierforge. open source, MIT, two runtime deps. it tracks cost per completed task — retries, escalations, failed loops, all of it.
the field test
I didn't want to ship this without proving it works. so I ran it against real GitHub data — issues, PRs, and source code from FastAPI. not "write a fibonacci function." real bug reports with real labels. real pull requests with real version bumps. real source files that cut off mid-signature.
about 100 LLM calls across 4 test runs. the final run: 33 calls, 8/8 scenarios passed, $0.01 total. 44.8% savings vs sending everything to one model.
but the numbers aren't the interesting part. the interesting part is what broke.
it took 4 runs to get it right
this is the part nobody talks about. the first run: 5/8 passed, found 5 bugs. the second: 6/8, found 2 more. the third: 6/8, found 1 more. the fourth: 8/8, found nothing.
the first two runs used synthetic prompts — "write a fibonacci function." those passed fine. then I switched to real data and everything broke. rate limits I didn't expect. a DOWNGRADE cycling bug where the router kept switching tiers in a loop. a global retry budget that got exhausted by workhorse retries before the architect tier could even try. stale code routing that made the savings numbers lie.
I thought "write a fibonacci function" was a reasonable test prompt. it's not. real PRs have variable length. real issues have labels that confuse the model. real rate limits kick in after 10 calls. you need real data or you're lying to yourself about what works.
that was the biggest lesson. not the routing logic, not the cost tracking. the fact that synthetic tests validate code paths and real data tests validate the system. they're different things. I didn't appreciate how different until the third run.
the architect model found a real bug in a real PR
gpt-5-nano reviewed fastapi PR #16018 and caught something. the title said "bump mcp from 1.26.0 to 1.28.1" but the body quoted v1.28.0 release notes. version mismatch. real catch, real PR, $0.001.
I didn't expect that. I expected the cheap model to handle code review fine and the architect model to handle specs. I didn't expect the architect to find a real bug that a human reviewer might miss. and I definitely didn't expect it to cost less than a cent.
the cheap model surprised me too. when I fed DeepSeek a truncated FastAPI source file, it didn't hallucinate the missing code. it said "the code cuts off mid-signature" and suggested reviewing the full file. that's the behavior you want from a workhorse — not confidence, honesty. a model that says "I can't see the full picture" instead of making something up.
the budget downgrade
this is the feature I'm most proud of, and it's not the routing.
two calls, same scope. call 1 went to the architect model ($0.001). call 2 triggered a budget DOWNGRADE — accumulated spend exceeded the limit, so the router automatically dropped to the workhorse ($0.0004). both calls succeeded. call 2 cost 60% less because of the downgrade. no manual intervention.
route → architect / gpt-5-nano (matched task_type 'spec')
failover → workhorse / deepseek-v4-flash (budget: per_task limit exceeded)
route → workhorse / deepseek-v4-flash (task completed)
you set a limit, the router drops to a cheaper tier when you hit it. your agents don't need to know it happened. that should be boring and automatic. now it is.
the metric nobody tracks
per-call cost is easy to measure. it's also misleading. cost per completed task is harder to measure but actually correct.
but there's a third metric I don't see anyone tracking: escalation rate. what percentage of your tasks required the cheap model to fail over to the expensive one?
if 80% of your tasks escalate, you haven't built a router. you've built a system that pays for a cheap attempt before every expensive call. that's worse than just using the expensive model from the start.
FrugalGPT (Stanford) showed cascade routing can save up to 98%. but they also showed it only works if you monitor escalation rate. RouteLLM (UC Berkeley) achieved over 2× cost reduction — but tracked no cost-per-task data.
I track escalation rate as an SLO. if it's too high, your routing isn't saving money. that's the signal to rebalance.
what I'm still figuring out
the free tier hit 429 rate limits after about 10 rapid calls. "free" means $0 per call, not unlimited usage. I had to switch to the paid tier for reliable testing. for production, always configure a paid fallback.
cost-per-task only works if you can define "task." the router needs your agent code to label calls — "this is a code task" or "this is a spec task." if your labels are wrong, your cost data is wrong. this isn't a tool problem, it's a discipline problem. but it's real.
no dashboard. no async API. JSONL logs and CLI output work today. if you need either of those, this isn't your tool yet.
the number
$0.01 for 33 LLM calls on the final run. ~100 across all 4 runs. real FastAPI code reviewed. real PRs analyzed. real issues categorized. a real version mismatch found. for a penny.
the 44.8% savings is real and replicated — synthetic data showed 47.6%, real data showed 44.8%. that 2.8% gap is just real-world token variance. with a frontier model as the architect instead of gpt-5-nano, the savings would be 97%.
but the headline isn't the penny. it's the 44.8%. that's what tier routing buys you vs sending everything to one model.
repo: github.com/deghosal-2026/ai-tierforge. pip install ai-tierforge. MIT.
it's also the companion to ai-loopguard, my circuit breaker for agent loops. loopguard decides when to escalate. ai-tierforge decides where and tracks how much it cost.
discussion
I want to hear from people running multi-model setups in production, not just experimenting:
how are you tracking LLM costs today? per-call dashboards? per-token? has anyone tried per-task accounting? what does that look like in your stack? I couldn't find a single tool that does this, which either means I'm missing something or nobody's built it yet.
what's your escalation rate? do you know what percentage of your tasks escalate from cheap to expensive? if you don't know, how would you even measure it? this feels like the metric that tells you whether your routing strategy is working or just burning tokens.
has anyone else tested their routing against real data vs synthetic? I was surprised how much broke when I switched from "write a fibonacci function" to real GitHub issues. what did real data expose in your system that synthetic tests missed?
budget enforcement — auto-downgrade or hard stops? the DOWNGRADE feature (automatically switching to a cheaper tier when budget is exceeded) is the thing I'm most curious about. does anyone have experience with this in production? is it useful or just a nice idea?
drop a comment with your current setup. I'm especially interested in what's working and what's not.
Originally published on Hashnode.
Top comments (0)