Don't Burn Your AI Budget: I'm Routing AI Models Like Infrastructure

I had a bill problem.

Not a catastrophic one. Just the slow, annoying kind where you look at your monthly AI API spend and think, "Wait, why did updating three Jira tickets cost $40?"

The answer, obviously, is that I was sending everything to the same model. Architecture decisions. Boilerplate. Ticket updates. A one-line regex fix. All of it through a frontier model that charges like it's doing brain surgery every time.

That felt dumb.

So I started experimenting. And I want to share what I'm trying, because honestly I'm not sure how well it holds up and I'd rather find out now than six months in.

The Core Idea (Stolen From Infrastructure)

Here's the thing I keep thinking about: we already solve this problem in infrastructure.

Workload	Compute We Use	Why
Static files	CDN / edge	Cheap, fast, globally distributed
API server	App-tier instance	Balanced compute and cost
Batch ML training	GPU cluster	Expensive but necessary for the task
Cron cleanup job	Smallest available box	Low priority, no user waiting

We match compute to workload. Nobody questions this.

But with LLMs? We pick one model and blast everything through it.

AI Task	What I Was Doing	What It Probably Deserves
Architecture spec	Frontier model	Frontier model (actually worth it)
Unit test drafts	Frontier model	Cheaper model (probably fine)
Jira ticket update	Frontier model	Local model (literally free)
Regex fix	Frontier model	Cheaper model (overkill)
PR security review	Frontier model	Frontier model (I want this)
Status summary	Frontier model	Local model (why am I paying for this?)

When I wrote it out like that, the waste was obvious.

What I'm Testing Now

I split my AI usage into three tiers. Nothing revolutionary here, just borrowed from the LLM routing and cascade research that's been circulating [1][2][3].

Tier	Example Models I've Used	Cost Profile	What I Send There
Architect	GLM 5.2	Expensive per token	Specs, architecture, PR review, hard debugging
Workhorse	DeepSeek-V4-Flash	Cheap, fast	Code drafts, unit tests, refactors, repetitive work
Utility	Qwen, Llama, DeepSeek (self-hosted via OMLX)	Free (compute only)	Ticket updates, summaries, status changes, simple ops

A note on the Utility tier: OMLX is the harness I use to self-host and serve these local models. The actual models I've been running through it are Qwen, Llama, and DeepSeek. I swap between them depending on the task — Qwen tends to be a bit better at structured text, Llama is solid for summaries, and DeepSeek's local variant handles simple code-related admin well enough.

These aren't the only options. They're just what I've been testing so far. I'm curious what others are using in each tier.

The principle I'm operating on:

Send work to the cheapest tier that can do it without making me regret it later.

The "regret it later" part is where most of the engineering effort goes.

Where I'm Definitely Keeping the Expensive Model

Architecture and requirements. Period.

Early-Stage Task	Why I Won't Cheap Out
PRDs	Wrong assumptions here multiply downstream
Tech specs	Foundation for everything that follows
Architecture options	Reversing a bad call here costs weeks
Critical user journeys	Missing edge cases = wrong tests
Security-sensitive design	Not the place to experiment

I tried using a cheaper model for a tech spec once. It produced something that looked reasonable. Then I read it carefully and realized it had quietly skipped three integration points that mattered.

GLM 5.2 caught those. DeepSeek-V4-Flash didn't.

That was a $12 lesson that could've been a $12,000 lesson.

Does anyone else find that specs are where the model choice matters most? Or am I being too cautious here?

The Router Experiment

This is the part I'm least sure about.

Instead of manually picking a model for each task, I generate a work breakdown structure first, then ask GLM 5.2 to route it.

Given this WBS, for each task classify:
  - reasoning complexity (low / medium / high)
  - failure risk if done poorly
  - expected token volume
  - verification method (tests? review? both?)
  - recommended tier (architect / workhorse / utility)

The flow I'm testing:

Stage	What Happens	Which Model
Spec / PRD	Write and review	GLM 5.2 (Architect)
WBS generation	Draft task breakdown	DeepSeek-V4-Flash (Workhorse)
WBS routing	Review and classify tasks	GLM 5.2 (Architect)
Implementation	Write code + tests	DeepSeek-V4-Flash (Workhorse)
Stuck loop	Break the cycle	GLM 5.2 (escalation)
PR review	Final quality gate	GLM 5.2 (Architect)
Ops / admin	Tickets, summaries, status	Qwen / Llama / DeepSeek via OMLX (Utility)

Here's my question: should GLM 5.2 review every WBS, or only the ones flagged as high-risk?

Reviewing every WBS feels safer but costs more. Skipping reviews feels faster but risks routing important tasks to a model that can't handle them.

I genuinely don't know the right answer yet.

The Escalation Circuit Breaker (My Favorite Part So Far)

You've seen this loop. I know you have.

Attempt	What Happens
Try 1	Model changes code, test fails
Try 2	Model changes something else, different test fails
Try 3	Model reverts first change, original test fails again
Try 4	Model is now confused, starts hallucinating context
Try 5	Tokens are burning, nothing is improving

DeepSeek-V4-Flash is great until it spirals. And that spiral is where the "savings" evaporate.

So I built a circuit breaker:

DeepSeek-V4-Flash attempts fix
    -> run tests

Tests pass?
    -> yes: continue
    -> no: retry (up to 3 attempts)

Still failing after 3?
    -> package the context
    -> escalate to GLM 5.2
    -> prompt: "The workhorse is stuck in a loop. Here's what it tried. Break the cycle."

GLM 5.2 isn't the default engine anymore. It's the escalation path.

This maps to what the cascade routing literature describes: try cheap first, verify the result, escalate only when verification fails [1][2][4].

But here's something I'm wrestling with: is 3 the right number?

Too few retries and I escalate before DeepSeek-V4-Flash gets a fair shot. Too many and I'm burning tokens on a lost cause. What's your threshold?

The Metrics I'm Tracking

If this isn't measurable, it's just vibes.

Metric	Why It Matters	My Current Concern
Escalation rate	If everything escalates, I'm paying double	This is the one that keeps me up
Cost per completed task	The real number, not per-token price	Harder to calculate than I expected
Retry count before success	Tells me if the cheap model is effective	High retry = workhorse isn't fitting
Test pass rate after fix	Quality signal for the work	Need more automation here
Frontier usage by task type	Am I reserving GLM 5.2 for the right things?	Still manually reviewing this
Human correction rate	When do I have to step in?	Tracking but not yet patterned

The one I watch most closely is escalation rate.

If 80% of tasks escalate, I haven't built a router. I've built a system that pays for a DeepSeek-V4-Flash attempt before every GLM 5.2 call. That's the worst version of this architecture.

Production routing discussions make the same point: cascade economics live or die on how often the cheap tier resolves the task on its own [4][5].

What metrics are you all tracking? I feel like I'm missing something obvious.

Confidence Is a Liar

Here's a hill I'll die on: I don't trust models that say they're confident.

Signal Type	Example	Do I Trust It?
Self-reported confidence	"I'm 95% sure this is correct"	No
Test execution	pytest passes	Yes
Schema validation	Output parses against expected schema	Yes
Static analysis	Linter + type checker clean	Yes
Diff scope	Changes stay within expected files	Mostly
Human review	I actually read the diff	Yes, but expensive
Repeated failure	Same error twice	Strong escalation signal

The research backs this up. A study on front-door routing found that small models reported confidence on all 60 predictions, including wrong ones. Uncalibrated confidence is not a production signal [6].

For coding workflows, tests and static checks are far better escalation triggers than a model saying "I think this is right."

Has anyone tried using a judge model to verify cheaper model output? I'm curious if that's worth the extra call or if it just adds latency and cost.

Routing vs. Failover (Don't Mix These Up)

I learned this one the hard way.

Concept	Meaning	Example
Routing	I chose this model intentionally because it fits the task	"This is a CRUD scaffold, send it to DeepSeek-V4-Flash"
Failover	The model I wanted was unavailable, so I fell back	"GLM 5.2 API timed out, use the backup"

If you log these as the same thing, you can't tell whether your system is optimizing cost or just surviving outages.

I made this mistake for two weeks before I realized my "routing distribution" was actually a failure log.

The SDLC Map I'm Running

SDLC Phase	Task	Tier	Example Model	Open Questions
Conception	PRDs, CUJs, architecture	Architect	GLM 5.2	Non-negotiable for now
Planning	WBS generation	Workhorse -> Architect review	DeepSeek-V4-Flash -> GLM 5.2	Should Architect review every WBS or only risky ones?
Coding	Implementation + unit tests	Workhorse	DeepSeek-V4-Flash	Working well so far
QA	Test-fix loop	Workhorse -> escalation	DeepSeek-V4-Flash -> GLM 5.2	Testing N=3 retries, might change
Review	PR review, security, architecture	Architect	GLM 5.2	Worth the cost
Ops	Tickets, summaries, status	Utility	Qwen / Llama / DeepSeek via OMLX	Which local model is best for which admin task?

Is this the right split? I honestly don't know. It's working better than "everything to one model," but that's a low bar.

The Harness Matters More Than the Model

Here's the thing that surprised me most.

I expected the model choice to be the biggest cost lever. It wasn't.

The orchestration layer was.

Harness Decision	Cost Impact
How much context gets replayed each turn	Massive
When to retry vs. when to escalate	Massive
How much history to carry forward	Large
What tools are exposed to the model	Medium
How failures are detected and stopped	Large
What gets logged for later review	Small immediate, large long-term

A sloppy harness makes every model expensive. A disciplined harness makes every model cheaper.

There's research now showing that the orchestration layer can move cost per task more than switching between the cheapest and most expensive model does [7]. That matched my experience, even though I didn't expect it.

And for the Utility tier, OMLX is my harness for serving the local models. It handles the self-hosting, routing between Qwen/Llama/DeepSeek, and keeps everything off third-party APIs. The harness choice matters even at the free tier.

What I'm Still Figuring Out

Genuinely open questions, not rhetorical ones:

Question	Where I'm At
Should routing be static rules or learned from past runs?	Starting with rules. Not sure when to move to learned.
Should escalation happen after N failures or based on failure type?	Testing N=3. Failure-type routing sounds better but harder to build.
Are local models worth the ops overhead?	Mixed. Free is great. Slower is annoying. Privacy is a bonus.
Which local model handles admin tasks best?	Rotating Qwen, Llama, DeepSeek via OMLX. No clear winner yet.
Should I compress context before escalating to GLM 5.2?	Yes, probably. Haven't implemented it yet. Anyone doing this?
How do I measure "cost per completed task" honestly?	Still figuring out the accounting. Token price isn't real cost.
Where does cost optimization start hurting quality?	Haven't hit the line yet. Worried I will.
Would a different workhorse model change the escalation rate?	Haven't tested beyond DeepSeek-V4-Flash yet. Curious what others use.

My Current Take (Subject to Change)

I'm moving away from:

"Which model should power our workflow?"

And toward:

"Which model should handle this specific step, given what's at stake?"

Right now that means GLM 5.2 for judgment, DeepSeek-V4-Flash for volume, and Qwen/Llama/DeepSeek served through OMLX for routine admin. But I reserve the right to swap any of them out.

That feels like the right question. But I'm only a few months into this experiment and I could be completely wrong.

If you're doing something similar, I want to hear about it. Especially if you've hit problems I haven't encountered yet.

What models are you using in each tier? Has anyone found a better workhorse than DeepSeek-V4-Flash? Is anyone running local models for engineering admin, or is that just me? If you're self-hosting, what harness are you using — OMLX, Ollama, vLLM, something else?

References

[1] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
https://arxiv.org/abs/2305.05176

[2] Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
https://arxiv.org/abs/2603.04445

[3] RouteLLM: Learning to Route LLMs with Preference Data
https://arxiv.org/abs/2406.18665

[4] Intelligent LLM Routing: Cost & Quality-Aware Selection, TrueFoundry
https://www.truefoundry.com/blog/llm-routing-cost-quality-aware-model-selection

[5] LLMOps Guide 2026: Build Fast, Cost-Effective LLM Apps, Redis
https://redis.io/blog/large-language-model-operations-guide/

[6] Evaluating Small Language Models for Front-Door Routing
https://swarmsignal.net/small-model-routing-frontier-fallback/

[7] The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
https://arxiv.org/html/2607.06906