i was using a frontier model to update jira tickets. here's how i fixed it.
last month I looked at my AI API bill and realized I was using a frontier model to update Jira tickets. a one-line regex fix? frontier model. a status summary that nobody reads? frontier model. architecture spec that actually matters? also frontier model.
everything went through the same expensive pipe. it was the equivalent of running a cron cleanup job on a GPU cluster — total overkill, billed by the token.
so I started treating LLMs exactly like infrastructure: traffic routing, fallback logic, cost-tiering, circuit breakers. this post is what I'm testing, what's working, and where I'm still guessing.
the sticker shock moment
not a catastrophic bill. just the slow, annoying kind where you look at monthly AI API spend and think, "wait, why did updating three Jira tickets cost $40?"
the answer: I was sending everything to one model. architecture decisions. boilerplate. ticket updates. a one-line regex fix. all of it through a model that charges like it's doing brain surgery every time.
when I wrote it out, the waste was obvious. architecture specs — worth the frontier model. unit test drafts — probably fine on a cheaper model. Jira ticket updates — a local model could do that for free. regex fixes — overkill on a frontier model. PR security reviews — worth the frontier model. status summaries — why was I paying for this at all?
the three tiers I'm running
I split everything into three tiers. the architect tier is GLM 5.2 — expensive per token, used for specs, architecture, PR review, hard debugging. the workhorse tier is DeepSeek-V4-Flash — cheap and fast, used for code drafts, unit tests, refactors, repetitive work. the utility tier is Qwen, Llama, and DeepSeek self-hosted via OMLX — free beyond electricity, used for ticket updates, summaries, status changes, simple ops.
the principle is simple: send work to the cheapest tier that can do it without making me regret it later. the "regret it later" part is where most of the engineering effort goes.
a note on the utility tier: OMLX is the harness I use to self-host and serve local models. the actual models I run through it are Qwen, Llama, and DeepSeek. I swap between them depending on the task — Qwen tends to be better at structured text, Llama is solid for summaries, and DeepSeek's local variant handles simple code-related admin well enough.
why LLM routing is harder than load balancing
if you've run infrastructure, you know load balancing. the analogy gets you 80% of the way to LLM routing — then the last 20% breaks the metaphor.
traditional load balancers route by capacity — CPU load, memory, round-robin, latency. LLM routers route by capability — prompt token length, task complexity, expected output size. a round-robin LLM router is useless. you'd send a spec to a 4B model and a ticket update to a frontier model on alternating requests. the routing decision has to be semantic: what kind of work is this, and which tier can handle it?
the hardest part isn't the routing logic — it's the verification. in infra, a 503 tells you the instance is down. in LLM routing, the model returns a 200 with confidently wrong output. you need tests, schema validation, or human review to detect the failure.
the escalation circuit breaker
you've seen this loop. I know you have. try 1: model changes code, test fails. try 2: model changes something else, different test fails. try 3: model reverts first change, original test fails again. try 4: model is now confused, starts hallucinating context. try 5: tokens are burning, nothing is improving.
DeepSeek-V4-Flash is great until it spirals. and that spiral is where the "savings" evaporate.
so I built a circuit breaker. the workhorse attempts a fix and runs tests. if tests pass, continue. if not, retry up to 3 times. still failing after 3? package the context, escalate to the architect tier (GLM 5.2), and prompt: "the workhorse is stuck in a loop. here's what it tried. break the cycle."
GLM 5.2 isn't the default engine anymore. it's the escalation path.
routing vs failover — don't mix these up
I made this mistake for two weeks and I want to save you the time.
the escalation pattern handles quality failures — bad output, invalid JSON, test fails, hallucination. you retry, then escalate to a higher tier. you also need failover for availability failures — API down, rate limited, timeout. you failover to a backup provider at the same tier.
these are different. a 429 from DeepSeek shouldn't escalate to GLM 5.2 — that's 18x the cost for the same task. it should failover to another workhorse-tier provider or queue and retry with backoff. escalation is for "this task is too hard for this tier." failover is for "this tier is unavailable."
if you log routing and failover as the same thing, you can't tell whether your system is optimizing cost or just surviving outages. I did that for two weeks before I realized my "routing distribution" was actually a failure log.
where I'm definitely keeping the expensive model
architecture and requirements. period. PRDs — wrong assumptions here multiply downstream. tech specs — foundation for everything that follows. architecture options — reversing a bad call here costs weeks. critical user journeys — missing edge cases means wrong tests. security-sensitive design — not the place to experiment.
I tried using a cheaper model for a tech spec once. it produced something that looked reasonable. then I read it carefully and realized it had quietly skipped three integration points that mattered. GLM 5.2 caught those. DeepSeek-V4-Flash didn't. that was a $12 lesson that could've been a $12,000 lesson.
confidence is a liar
here's a hill I'll die on: I don't trust models that say they're confident.
self-reported confidence — "I'm 95% sure this is correct" — no. test execution — pytest passes — yes. schema validation — output parses against expected schema — yes. static analysis — linter + type checker clean — yes. human review — I actually read the diff — yes, but expensive. repeated failure — same error twice — strong escalation signal.
the research backs this up. a study on front-door routing found that small models reported confidence on all 60 predictions, including wrong ones. uncalibrated confidence is not a production signal. for coding workflows, tests and static checks are far better escalation triggers than a model saying "I think this is right."
the harness matters more than the model
here's the thing that surprised me most. I expected the model choice to be the biggest cost lever. it wasn't. the orchestration layer was.
how much context gets replayed each turn — massive cost impact. when to retry vs when to escalate — massive. how much history to carry forward — large. what tools are exposed to the model — medium. how failures are detected and stopped — large. what gets logged for later review — small immediate, large long-term.
a sloppy harness makes every model expensive. a disciplined harness makes every model cheaper. there's research now showing that the orchestration layer can move cost per task more than switching between the cheapest and most expensive model does. that matched my experience, even though I didn't expect it.
what I'm still figuring out
genuinely open questions, not rhetorical ones. should routing be static rules or learned from past runs? I'm starting with rules. not sure when to move to learned. should escalation happen after N failures or based on failure type? testing N=3. failure-type routing sounds better but harder to build. are local models worth the ops overhead? mixed — free is great, slower is annoying, privacy is a bonus. how do I measure "cost per completed task" honestly? still figuring out the accounting. token price isn't real cost. where does cost optimization start hurting quality? haven't hit the line yet. worried I will.
my current take (subject to change)
I'm moving away from "which model should power our workflow?" and toward "which model should handle this specific step, given what's at stake?"
right now that means GLM 5.2 for judgment, DeepSeek-V4-Flash for volume, and Qwen/Llama/DeepSeek served through OMLX for routine admin. but I reserve the right to swap any of them out.
discussion
I want to hear from people doing this in production, not just experimenting:
how are you handling LLM fallback strategies in production? are you using an open-source gateway like LiteLLM or Portkey, or did you build custom middleware? what pushed you one way or the other?
where do you draw the line between "simple" prompts for small models and "complex" prompts for frontier models? what's your heuristic? token length? task type? something you measure?
what's your escalation threshold? I'm testing N=3 retries before escalating. too few and the workhorse doesn't get a fair shot. too many and I'm burning tokens on a lost cause. what number are you using?
is anyone running local models for engineering admin, or is that just me? if you're self-hosting, what harness are you using — OMLX, Ollama, vLLM, something else? is the ops overhead worth the $0/token?
has anyone tried a judge model to verify cheaper model output? I'm curious if that's worth the extra call or if it just adds latency and cost.
if you're trying to optimize your team's AI spend without killing development velocity, drop a comment with your current stack and let's brainstorm solutions.
Top comments (0)