What building an LLM cost router, finding problems in my own benchmarks, and a corrected 100-query experiment taught me about the infrastructure layer between AI applications and model providers.
The expensive model problem
You ship a feature. It calls a large LLM. It works beautifully, so you move on.
Months later, someone looks at the bill and notices that a surprising amount of traffic is still going through that expensive model.
“What’s my account number?”
“Define photosynthesis.”
“What does NDA stand for?”
None of these questions needed the same model as a difficult reasoning task.
But without routing, they often get one anyway.
The model isn't necessarily the problem.
The lack of a routing layer is.
I built CARDIAC-PURR to address that problem.
I'm an attorney by training, not an engineer, so I made an early decision: I wasn't going to trust my intuition, and I wasn't going to trust somebody else's benchmark either.
I was going to measure it myself.
That turned out to be much harder than I expected.
What I built
CARDIAC-PURR sits between an application and its LLM providers.
The basic idea is simple:
Application
↓
CARDIAC-PURR
↓
Model tier
↓
LLM provider
↓
Response
Instead of sending every request to the most capable model, the router tries to determine which tier is appropriate for that request.
The design spans multiple providers and multiple model tiers.
The important economic idea is not “always use the cheapest model.”
That would just trade compute cost for answer quality.
The objective is:
Use a lower-cost model when the request can be handled there, and use a more capable tier when it cannot.
There is also a safety path for cases where the initially selected tier produces an inadequate or truncated response.
Some of the implementation is proprietary, so I'm not going to turn this article into a reverse-engineering document.
What I can publish is how the system behaved, what the benchmarks showed, and where the benchmarks themselves failed.
I wanted a benchmark I could challenge
The first version of the benchmark covered multiple providers and model tiers.
That was important because pricing is different across providers, and a routing strategy that looks impressive with one provider may look very different with another.
But benchmarking a router isn't just a matter of collecting dollar figures.
I wanted to answer two different questions:
- Did the router choose the intended model tier?
- What did those routing decisions cost?
Those are related, but they are not the same measurement.
A router can be very good at choosing tiers and still have poor economics.
It can also save money while making bad routing decisions.
That distinction became increasingly important as the investigation progressed.
Then the benchmark started teaching me things I didn't expect
During earlier testing, I saw unexpectedly large execution differences when the same underlying requests passed through different application frameworks.
One example involved an agent framework.
A simple question could arrive at the control plane wrapped in task instructions, output requirements, and other framework-generated context.
The user's task had not become harder.
The input seen by the router had.
That exposed a routing edge case.
We fixed it.
But fixing the router wasn't enough.
The investigation also led us to audit the benchmark itself.
And that's where things got uncomfortable.
A benchmark can be wrong and still look right
Our measurement harness had correct timestamps, but one part of the timing accounting was incomplete.
An interval between provider dispatch stages was not being attributed correctly to a named latency component.
In one example, the missing interval represented almost all of the observed time.
The timestamps were real.
The total request time was real.
The explanation of that time was wrong.
A benchmark doesn't become trustworthy merely because the numbers look plausible.
We corrected the accounting and added a validation layer for subsequent experiments.
The principle was simple:
The benchmark itself needs acceptance criteria.
The first 100-query economic benchmark failed
Once the routing and measurement issues had been addressed, I ran a paired benchmark using 100 randomly selected technical queries.
The same 100 queries went through both arms, with cache bypassed.
Arm A
CARDIAC-PURR routing enabled:
model=auto
Arm B
A LARGE-tier reference:
model=large
I expected the router to reduce cost.
The first result went the other way.
| Metric | CARDIAC-PURR | LARGE-tier reference |
|---|---|---|
| Total observed cost | $0.868122 | $0.802010 |
| Difference | 8.24% higher | — |
CARDIAC-PURR had lost.
I kept the result.
More importantly, I investigated it.
The problem was that the first run had left an output-budget variable uncontrolled. Different provider behavior interacted with the escalation path, causing some requests to be truncated and enter a more expensive recovery path.
That made the first run useful as a diagnostic artifact.
It did not make it a clean economic comparison.
So I did not use it as the headline result.
The corrected benchmark
I reran the same 100-query cohort.
The key change was simple:
max_tokens=2048
was explicitly applied to both arms.
The rest of the A/B structure and cache-bypass condition stayed the same.
The corrected result was:
| Metric | CARDIAC-PURR | LARGE-tier reference |
|---|---|---|
| Total observed model cost | $0.350698 | $3.997920 |
| Average cost/query | $0.003507 | $0.039979 |
| Cost difference | 91.2% lower | — |
| Cascades | 0 | — |
| Retries | 0 | — |
That 91.2% figure is the result I am comfortable publishing.
But it needs a precise definition:
On this 100-query technical workload, CARDIAC-PURR produced 91.2% lower observed model inference cost than the realized LARGE-tier reference arm.
It is not a claim about total application cost.
It is not a claim about TCO.
And it is not a promise that every workload will save 91.2%.
Where did the savings come from?
The tier distribution tells most of the story.
| Routed tier | Queries | CARDIAC-PURR cost | Matched LARGE-reference cost | Reduction |
|---|---|---|---|---|
| SMALL | 93 | $0.163543 | $3.715975 | 95.6% |
| MEDIUM | 4 | $0.065385 | $0.156775 | 58.3% |
| LARGE | 3 | $0.121770 | $0.125170 | 2.7% |
Ninety-three of the 100 requests stayed in SMALL.
On those requests, the matched observed cost was 95.6% lower than the LARGE-tier reference.
But the three requests that actually went to LARGE are just as interesting.
They cost almost the same in both arms.
That means the result wasn't simply:
"Avoid the expensive model."
The router selected LARGE when LARGE was needed.
The savings came primarily from not using LARGE when it wasn't needed.
That is the economic behavior I was trying to measure.
One detail I don't want to hide
The LARGE reference arm was not literally one identical provider for every request.
Of the 100 realized reference calls:
- 98 resolved to Gemini 2.5 Pro
- 1 resolved to DeepSeek V4 Pro
- 1 resolved to Command A
CARDIAC-PURR itself resolved to:
- 93 Gemini 2.5 Flash-Lite
- 4 Gemini 2.5 Flash
- 3 Gemini 2.5 Pro
So the 91.2% figure compares CARDIAC-PURR with the realized LARGE reference arm, rather than pretending that every reference request was physically served by one identical provider path.
I would rather disclose what actually happened than simplify the table until it tells a cleaner story.
There is another baseline, and it answers a different question
The benchmark also produced a synthetic always-LARGE baseline of $1.048460.
That is not a third live benchmark arm.
It is a per-request calculation estimating what each request would have cost if it had been sent to the LARGE-tier model at the provider selected for that request.
Against that baseline, CARDIAC-PURR was 66.6% lower.
So:
- 91.2% lower = realized LARGE-tier reference arm
- 66.6% lower = synthetic always-LARGE baseline
Those numbers should not be mixed.
They answer different questions.
What about quality?
This benchmark measured economics.
It did not measure answer quality on these same 100 requests.
That's important because cost savings without quality would be a bad trade.
We have separate prior evaluation results in the 96–100% range, depending on the evaluation set and methodology.
I am deliberately not attaching those results to the 91.2% number.
They are separate experiments.
The next benchmark needs to answer the question that matters most:
Does CARDIAC-PURR maintain answer quality while making these cheaper routing decisions on the same workload?
Until that is measured on the same cohort, I don't think it is responsible to claim otherwise.
And what about the failure path?
The corrected benchmark produced zero cascades and zero retries.
That's good for the cleanliness of the cost comparison.
But it also tells us what the experiment did not test.
We measured successful tier selection.
We did not exercise the economics of a request that starts in SMALL, fails or becomes inadequate, and then escalates.
So this benchmark does not prove that the recovery path behaves optimally under failure.
That is another experiment.
Similarly, routing overhead was not separately measured as part of this economic result.
That should be measured formally alongside inference cost.
The framework investigation changed how I think about the problem
The agent-framework episode ended up being useful for a reason that goes beyond that particular framework.
A router sitting underneath an application does not necessarily receive a clean user question.
It receives whatever the application sends.
That can include:
- framework instructions
- task context
- tool-related information
- output-format requirements
- generated memory or state
- other application-level scaffolding
The underlying user request may remain simple while the actual prompt structure becomes much more complicated.
That means a routing layer cannot be designed and calibrated only around direct API calls.
It has to work with the way real AI applications construct requests.
That's where the control plane idea becomes important.
Why routing belongs at the infrastructure layer
A lot of AI architecture still looks like this:
Application
↓
Model
That works when there is one model and one provider.
It gets less comfortable when an organization has multiple models, multiple providers, different cost policies, failure handling, and increasingly complex AI applications.
The architecture starts looking more like:
Application
↓
Agent / Framework
↓
AI Control Plane
↓
Model / Provider
↓
Execution
↓
Recovery / Observability / Cost
At that point, model routing is no longer just a convenience function inside an application.
It affects infrastructure economics.
It affects provider selection.
It affects failure behavior.
It affects observability.
And it has to understand the traffic arriving from the application layer.
That is the problem space CARDIAC-PURR is being built around.
What I got wrong
This is probably the part I value most in the whole exercise.
I was too quick to trust an early benchmark because its output looked reasonable.
I initially attributed a large latency anomaly too directly to routing behavior.
The benchmark's timing accounting contained a defect.
The first economic experiment omitted an output-budget control that mattered.
And the later investigation showed that fixing one routing bug did not explain every observed latency difference.
In other words, the system wasn't wrong in only one place.
The router could be wrong.
The measurement could be wrong.
And the interpretation of the measurement could be wrong.
That's why I became much more conservative about what I was willing to call a benchmark result.
The earlier multi-framework measurements remain useful as development and investigation artifacts, but I no longer treat all of their historical performance or savings numbers as authoritative measurements of the corrected system.
That is a much less exciting claim.
I think it is also a more honest one.
What I still need to test
There is a lot left to do.
The current result needs to be tested across different workloads, not just technical questions.
Quality and cost need to be measured together on the same requests.
Provider diversity needs to be tested deliberately.
Failure and escalation economics need a dedicated experiment.
The system needs to be tested on real multi-step agentic workflows rather than mostly single-turn requests.
Routing overhead needs to be measured formally alongside model inference cost.
The objective isn't to find another impressive percentage.
It is to find out where the system stops working.
That is a much harder problem.
It is also the more interesting one.
The bigger lesson
I started with a straightforward economic problem:
Why should every LLM request cost as much as the hardest request in the system?
That led to a router.
The router led to benchmarks.
The benchmarks exposed bugs.
The bugs forced a deeper look at the measurement process.
And the investigation eventually led to a broader conclusion:
Model routing is becoming an infrastructure problem.
Once applications use multiple models and providers, the decision about which model handles which request sits at the intersection of application behavior, model capability, provider economics, reliability, and operational policy.
That is bigger than a helper function.
It's a control-plane problem.
The corrected benchmark gives us one concrete result:
91.2% lower observed model inference cost on this 100-query technical workload versus the realized LARGE-tier reference.
That's meaningful.
But the number is only useful because the process around it is visible: the bad result, the investigation, the corrections, and the limitations are part of the result too.
I built CARDIAC-PURR because I believed LLM routing needed a better infrastructure layer.
I'm publishing the benchmark because I think that belief should be tested, not merely asserted.
Trust is something a benchmark has to earn.
Disclosure: I’m the founder of CARDIAC-PURR. This article describes our own engineering work and benchmark results. CARDIAC-PURR is a product in development. I’m intentionally not disclosing proprietary routing mechanisms or implementation details.
Emil Igidov
Kraków, Poland
Founder, CARDIAC-PURR · Patent-Pending Inventor
Learn more about CARDIAC-PURR: https://www.cardiac-purr.com
Top comments (0)