DEV Community: Emil

91.2% lower LLM inference cost — and the failed benchmark I almost buried

Emil — Tue, 15 Sep 2026 08:30:00 +0000

What building an LLM cost router, finding problems in my own benchmarks, and a corrected 100-query experiment taught me about the infrastructure layer between AI applications and model providers.

The expensive model problem

You ship a feature. It calls a large LLM. It works beautifully, so you move on.

Months later, someone looks at the bill and notices that a surprising amount of traffic is still going through that expensive model.

“What’s my account number?”

“Define photosynthesis.”

“What does NDA stand for?”

None of these questions needed the same model as a difficult reasoning task.

But without routing, they often get one anyway.

The model isn't necessarily the problem.

The lack of a routing layer is.

I built CARDIAC-PURR to address that problem.

I'm an attorney by training, not an engineer, so I made an early decision: I wasn't going to trust my intuition, and I wasn't going to trust somebody else's benchmark either.

I was going to measure it myself.

That turned out to be much harder than I expected.

What I built

CARDIAC-PURR sits between an application and its LLM providers.

The basic idea is simple:

Application
    ↓
CARDIAC-PURR
    ↓
Model tier
    ↓
LLM provider
    ↓
Response

Instead of sending every request to the most capable model, the router tries to determine which tier is appropriate for that request.

The design spans multiple providers and multiple model tiers.

The important economic idea is not “always use the cheapest model.”

That would just trade compute cost for answer quality.

The objective is:

Use a lower-cost model when the request can be handled there, and use a more capable tier when it cannot.

There is also a safety path for cases where the initially selected tier produces an inadequate or truncated response.

Some of the implementation is proprietary, so I'm not going to turn this article into a reverse-engineering document.

What I can publish is how the system behaved, what the benchmarks showed, and where the benchmarks themselves failed.

I wanted a benchmark I could challenge

The first version of the benchmark covered multiple providers and model tiers.

That was important because pricing is different across providers, and a routing strategy that looks impressive with one provider may look very different with another.

But benchmarking a router isn't just a matter of collecting dollar figures.

I wanted to answer two different questions:

Did the router choose the intended model tier?
What did those routing decisions cost?

Those are related, but they are not the same measurement.

A router can be very good at choosing tiers and still have poor economics.

It can also save money while making bad routing decisions.

That distinction became increasingly important as the investigation progressed.

Then the benchmark started teaching me things I didn't expect

During earlier testing, I saw unexpectedly large execution differences when the same underlying requests passed through different application frameworks.

One example involved an agent framework.

A simple question could arrive at the control plane wrapped in task instructions, output requirements, and other framework-generated context.

The user's task had not become harder.

The input seen by the router had.

That exposed a routing edge case.

We fixed it.

But fixing the router wasn't enough.

The investigation also led us to audit the benchmark itself.

And that's where things got uncomfortable.

A benchmark can be wrong and still look right

Our measurement harness had correct timestamps, but one part of the timing accounting was incomplete.

An interval between provider dispatch stages was not being attributed correctly to a named latency component.

In one example, the missing interval represented almost all of the observed time.

The timestamps were real.

The total request time was real.

The explanation of that time was wrong.

A benchmark doesn't become trustworthy merely because the numbers look plausible.

We corrected the accounting and added a validation layer for subsequent experiments.

The principle was simple:

The benchmark itself needs acceptance criteria.

The first 100-query economic benchmark failed

Once the routing and measurement issues had been addressed, I ran a paired benchmark using 100 randomly selected technical queries.

The same 100 queries went through both arms, with cache bypassed.

Arm A

CARDIAC-PURR routing enabled:

model=auto

Arm B

A LARGE-tier reference:

model=large

I expected the router to reduce cost.

The first result went the other way.

Metric	CARDIAC-PURR	LARGE-tier reference
Total observed cost	$0.868122	$0.802010
Difference	8.24% higher	—

CARDIAC-PURR had lost.

I kept the result.

More importantly, I investigated it.

The problem was that the first run had left an output-budget variable uncontrolled. Different provider behavior interacted with the escalation path, causing some requests to be truncated and enter a more expensive recovery path.

That made the first run useful as a diagnostic artifact.

It did not make it a clean economic comparison.

So I did not use it as the headline result.

The corrected benchmark

I reran the same 100-query cohort.

The key change was simple:

max_tokens=2048

was explicitly applied to both arms.

The rest of the A/B structure and cache-bypass condition stayed the same.

The corrected result was:

Metric	CARDIAC-PURR	LARGE-tier reference
Total observed model cost	$0.350698	$3.997920
Average cost/query	$0.003507	$0.039979
Cost difference	91.2% lower	—
Cascades	0	—
Retries	0	—

That 91.2% figure is the result I am comfortable publishing.

But it needs a precise definition:

On this 100-query technical workload, CARDIAC-PURR produced 91.2% lower observed model inference cost than the realized LARGE-tier reference arm.

It is not a claim about total application cost.

It is not a claim about TCO.

And it is not a promise that every workload will save 91.2%.

Where did the savings come from?

The tier distribution tells most of the story.

Routed tier	Queries	CARDIAC-PURR cost	Matched LARGE-reference cost	Reduction
SMALL	93	$0.163543	$3.715975	95.6%
MEDIUM	4	$0.065385	$0.156775	58.3%
LARGE	3	$0.121770	$0.125170	2.7%

Ninety-three of the 100 requests stayed in SMALL.

On those requests, the matched observed cost was 95.6% lower than the LARGE-tier reference.

But the three requests that actually went to LARGE are just as interesting.

They cost almost the same in both arms.

That means the result wasn't simply:

"Avoid the expensive model."

The router selected LARGE when LARGE was needed.

The savings came primarily from not using LARGE when it wasn't needed.

That is the economic behavior I was trying to measure.

One detail I don't want to hide

The LARGE reference arm was not literally one identical provider for every request.

Of the 100 realized reference calls:

98 resolved to Gemini 2.5 Pro
1 resolved to DeepSeek V4 Pro
1 resolved to Command A

CARDIAC-PURR itself resolved to:

93 Gemini 2.5 Flash-Lite
4 Gemini 2.5 Flash
3 Gemini 2.5 Pro

So the 91.2% figure compares CARDIAC-PURR with the realized LARGE reference arm, rather than pretending that every reference request was physically served by one identical provider path.

I would rather disclose what actually happened than simplify the table until it tells a cleaner story.

There is another baseline, and it answers a different question

The benchmark also produced a synthetic always-LARGE baseline of $1.048460.

That is not a third live benchmark arm.

It is a per-request calculation estimating what each request would have cost if it had been sent to the LARGE-tier model at the provider selected for that request.

Against that baseline, CARDIAC-PURR was 66.6% lower.

So:

91.2% lower = realized LARGE-tier reference arm
66.6% lower = synthetic always-LARGE baseline

Those numbers should not be mixed.

They answer different questions.

What about quality?

This benchmark measured economics.

It did not measure answer quality on these same 100 requests.

That's important because cost savings without quality would be a bad trade.

We have separate prior evaluation results in the 96–100% range, depending on the evaluation set and methodology.

I am deliberately not attaching those results to the 91.2% number.

They are separate experiments.

The next benchmark needs to answer the question that matters most:

Does CARDIAC-PURR maintain answer quality while making these cheaper routing decisions on the same workload?

Until that is measured on the same cohort, I don't think it is responsible to claim otherwise.

And what about the failure path?

The corrected benchmark produced zero cascades and zero retries.

That's good for the cleanliness of the cost comparison.

But it also tells us what the experiment did not test.

We measured successful tier selection.

We did not exercise the economics of a request that starts in SMALL, fails or becomes inadequate, and then escalates.

So this benchmark does not prove that the recovery path behaves optimally under failure.

That is another experiment.

Similarly, routing overhead was not separately measured as part of this economic result.

That should be measured formally alongside inference cost.

The framework investigation changed how I think about the problem

The agent-framework episode ended up being useful for a reason that goes beyond that particular framework.

A router sitting underneath an application does not necessarily receive a clean user question.

It receives whatever the application sends.

That can include:

framework instructions
task context
tool-related information
output-format requirements
generated memory or state
other application-level scaffolding

The underlying user request may remain simple while the actual prompt structure becomes much more complicated.

That means a routing layer cannot be designed and calibrated only around direct API calls.

It has to work with the way real AI applications construct requests.

That's where the control plane idea becomes important.

Why routing belongs at the infrastructure layer

A lot of AI architecture still looks like this:

Application
    ↓
Model

That works when there is one model and one provider.

It gets less comfortable when an organization has multiple models, multiple providers, different cost policies, failure handling, and increasingly complex AI applications.

The architecture starts looking more like:

Application
    ↓
Agent / Framework
    ↓
AI Control Plane
    ↓
Model / Provider
    ↓
Execution
    ↓
Recovery / Observability / Cost

At that point, model routing is no longer just a convenience function inside an application.

It affects infrastructure economics.

It affects provider selection.

It affects failure behavior.

It affects observability.

And it has to understand the traffic arriving from the application layer.

That is the problem space CARDIAC-PURR is being built around.

What I got wrong

This is probably the part I value most in the whole exercise.

I was too quick to trust an early benchmark because its output looked reasonable.

I initially attributed a large latency anomaly too directly to routing behavior.

The benchmark's timing accounting contained a defect.

The first economic experiment omitted an output-budget control that mattered.

And the later investigation showed that fixing one routing bug did not explain every observed latency difference.

In other words, the system wasn't wrong in only one place.

The router could be wrong.

The measurement could be wrong.

And the interpretation of the measurement could be wrong.

That's why I became much more conservative about what I was willing to call a benchmark result.

The earlier multi-framework measurements remain useful as development and investigation artifacts, but I no longer treat all of their historical performance or savings numbers as authoritative measurements of the corrected system.

That is a much less exciting claim.

I think it is also a more honest one.

What I still need to test

There is a lot left to do.

The current result needs to be tested across different workloads, not just technical questions.

Quality and cost need to be measured together on the same requests.

Provider diversity needs to be tested deliberately.

Failure and escalation economics need a dedicated experiment.

The system needs to be tested on real multi-step agentic workflows rather than mostly single-turn requests.

Routing overhead needs to be measured formally alongside model inference cost.

The objective isn't to find another impressive percentage.

It is to find out where the system stops working.

That is a much harder problem.

It is also the more interesting one.

The bigger lesson

I started with a straightforward economic problem:

Why should every LLM request cost as much as the hardest request in the system?

That led to a router.

The router led to benchmarks.

The benchmarks exposed bugs.

The bugs forced a deeper look at the measurement process.

And the investigation eventually led to a broader conclusion:

Model routing is becoming an infrastructure problem.

Once applications use multiple models and providers, the decision about which model handles which request sits at the intersection of application behavior, model capability, provider economics, reliability, and operational policy.

That is bigger than a helper function.

It's a control-plane problem.

The corrected benchmark gives us one concrete result:

91.2% lower observed model inference cost on this 100-query technical workload versus the realized LARGE-tier reference.

That's meaningful.

But the number is only useful because the process around it is visible: the bad result, the investigation, the corrections, and the limitations are part of the result too.

I built CARDIAC-PURR because I believed LLM routing needed a better infrastructure layer.

I'm publishing the benchmark because I think that belief should be tested, not merely asserted.

Trust is something a benchmark has to earn.

Disclosure: I’m the founder of CARDIAC-PURR. This article describes our own engineering work and benchmark results. CARDIAC-PURR is a product in development. I’m intentionally not disclosing proprietary routing mechanisms or implementation details.

Emil Igidov
Kraków, Poland
Founder, CARDIAC-PURR · Patent-Pending Inventor

Learn more about CARDIAC-PURR: https://www.cardiac-purr.com

I Built CARDIAC-PURR: What 900 Real API Calls Taught Me About LLM Cost Routing

Emil — Thu, 03 Sep 2026 04:26:52 +0000

You ship a feature, it calls GPT-4, it works beautifully, you move on. Six months later, someone checks the logs and — surprise — half your traffic is "what's my account number" going through a model smart enough to pass the bar exam. This isn't a one-off mistake, either — it's the default outcome of not routing at all. Support-queue traffic is mostly simple, but "mostly simple" tends to get shipped through whatever model handled the hard case that made you reach for a big model in the first place. The model's not the problem. The lack of a router is.

I built a router to fix that. I'm an attorney by training, not an engineer, so instead of trusting my own intuition (or published benchmarks), I decided to measure it properly.

What I Built

CARDIAC-PURR sits between your app and your LLM providers, monitoring each query and deciding whether it actually needs a large model or would do fine with a cheaper one. I tested it against 9 providers, each with its own small→medium→large split:

Provider	Small	Medium	Large
Anthropic	claude-haiku-4-5-20251001	claude-sonnet-4-6	claude-opus-4-6
OpenAI	gpt-4.1-nano	gpt-4.1-mini	gpt-4.1
Google	gemini-2.5-flash-lite	gemini-2.5-flash	gemini-2.5-pro
Azure OpenAI	gpt-4.1-nano	gpt-4.1-mini	gpt-4.1
Mistral	mistral-small-latest	mistral-medium-latest	mistral-large-latest
DeepSeek	deepseek-v4-flash	deepseek-v4-flash†	deepseek-v4-pro
Cohere	command-r7b-12-2024	command-r-plus-08-2024	command-a-03-2025
Grok	grok-4.3§	grok-4.3§	grok-4.3
Qwen	qwen-turbo	qwen-plus	qwen-max

† DeepSeek's small and medium tiers point at the identical model (deepseek-v4-flash) — there's no true 3-way split for DeepSeek, just flash vs. pro. That sounds like a limitation, and I originally treated it as one, but it turns out to be the reason DeepSeek posts the highest savings of any provider tested: flash is dramatically cheaper than pro, and 92% of traffic never touches pro at all. Full explanation, including why an earlier 46.0% estimate for DeepSeek was wrong, is below.

§ This resolves an open flag from an earlier draft — I'd noted that xAI retired the standalone grok-4-1-fast endpoints on May 15, 2026 and worried the small/medium/large split might secretly be hitting one model. Turns out that's exactly what's happening, confirmed directly from the router's own config: all three Grok tiers call grok-4.3. The only lever available is a reasoning_effort parameter (none/low/high) — a compute-budget knob on one model, not a routing decision between differently-priced models. Details and what this does to Grok's savings ceiling are below.

Tier selection happens before inference, based on query complexity — no waiting for a model response, no trained classifier, no embedding model.

Every query gets a complexity score against calibrated thresholds (self.c_target, currently 0.600 for the LARGE-tier gate). Most queries score nowhere near a boundary — the 100-question set was intentionally realistic, not adversarial, and the accuracy numbers below reflect that. Queries that land close to a threshold are exactly where I'd expect the router to be least confident, and it's also where the escalation safety net matters most: if a borderline SMALL-tier pick produces a weak or truncated response, that's caught and escalated rather than silently returned. I don't yet have a clean breakdown of accuracy specifically on near-boundary queries in this run — that's on my list for the next pass.

There's also a safety net: if the tier it picked gives a truncated or weak-looking response, the router escalates to a bigger model. That's a fallback, not the main path.

Under the hood, there are 15 distinct cost-saving mechanisms and 5 layers of failure/quality prevention (traffic protection, security, cost control, data durability, observability) working together. Patent-pending, so I won't break them down in detail here — but the numbers below are what all of them add up to in production.

How I Tested It

I've been running this in production since March. For this post, I wanted a clean, controlled comparison: same 100 questions, asked to nine providers, so the numbers below are apples-to-apples.

I ran 100 queries per provider — Anthropic, OpenAI, Google, Azure OpenAI, Mistral, DeepSeek, Cohere, Grok, and Qwen. Real money spent. Real API calls. Same 100 questions across all nine providers.

The queries were intentionally unglamorous: 75% factual definitions ("What does NDA stand for?"), 17% explanatory ("Describe the main steps in..."), 8% complex reasoning. This is what actual support queues look like. The full 100-question set, labeled by expected tier and vertical, is linked at the bottom of this post — if you think my distribution is wrong for your workload, don't take my word for it; run it yourself.

I measured two things separately because mixing them is how benchmarks lie:

Did the router pick the right tier? (routing accuracy)
Did accuracy hold up across domains, not just in aggregate? (vertical accuracy — legal, healthcare, finance, IT)

The Numbers

A note before the table: with n=100 per provider, 100% accuracy doesn't mean zero future misclassifications — it means the 95% confidence interval is [96.4%, 100%]. I'm expanding sample size (see "What Comes Next"), and in the meantime the full methodology below is reproducible — run it on your own data and see where your numbers land.

Run date: July 12, 2026 — router commit 57175b7, which fixed a warmup-accounting bug that had previously forced the first ~47 post-warmup queries to the LARGE tier regardless of actual complexity. This is a clean rerun; the numbers below supersede any earlier pass.

Provider	Accuracy	Savings	Fail %	Errors	Vertical (Legal/Health/Finance/IT)
DeepSeek	100%	88.3%†	0.0%	0/100	100/100/100/100
Qwen	100%	86.3%	0.0%	0/100	100/100/100/100
OpenAI	100%	84.9%	0.0%	0/100	100/100/100/100
Azure OpenAI	100%	84.9%	0.0%	0/100	100/100/100/100
Google	100%	84.4%	0.0%	0/100	100/100/100/100
Anthropic	100%	78.8%	0.0%	0/100	100/100/100/100
Mistral	100%	77.7%	0.0%	0/100	100/100/100/100
Cohere⚠	98.0%	73.9%‡	2.0%	2/100	96/100/96/100
Grok	100%	30.9%§	0.0%	0/100	100/100/100/100

† DeepSeek is the best performer in this benchmark, and the reason flips an earlier, wrong estimate: a prior pass reported 46.0% from a coarse price-ratio table, not real dollars. The actual number, computed from real per-query costs against DeepSeek's LARGE-tier price, is 1 - $0.0059/$0.0505 = 88.3%. Full mechanism explained in the DeepSeek/Grok section below.

‡ Cohere's medium and large tiers are priced identically ($12.50/MTok combined), so there's no discount available between those two tiers by design — that structural fact caps Cohere's best-case savings below the other providers' ceiling regardless of errors. Separately, and unrelated to pricing: Cohere hit 98.0% accuracy (95% CI: 93.0–99.4%) because of two transient read timeouts — one on a legal MEDIUM-tier query, one on a financial LARGE-tier query. In both cases the router picked the correct tier and the underlying API call timed out before returning a response; these are provider/network-side failures, not routing misclassifications.

§ Grok is the lowest-savings provider in this run — structural, not a routing weakness. Full explanation below.

⚠ Cohere is the only provider with a nonzero error rate in this run (2/100 — real timed-out API calls, distinct from cascades, which were zero for every provider this run). Every other provider, including Anthropic, came back completely clean: 100% accuracy, 0 errors, no vertical dips.

*Savings % isn't comparable across providers as a ranking — it reflects each provider's own small-vs-large price ratio, not routing quality. Routing accuracy is the fair cross-provider comparison, and on that metric 8 of 9 providers hit 100%.

All queries were pre-labeled into expected tiers (small/medium/large) based on response complexity. Routing accuracy compares the router's decision against these labels.

Methodology: labeling, baseline, and confidence intervals. Given the actual query mix (75% small / 17% medium / 8% large), a naive router that always guesses "small" gets 75% accuracy for free, with zero routing logic — that's the honest baseline to beat. (Random guessing among three tiers gives 33%, the more commonly cited comparison, but it's the weaker one given this distribution.)

Per-provider 95% confidence intervals (Clopper-Pearson): providers at 100/100 sit at [96.38%, 100.00%]; Cohere at 98/100 sits at [93.0%, 99.4%].

8 of 9 providers hit 100% routing accuracy against my 96% target; only Cohere (98%) came in under it, and that's down to two transient timeouts, not a misclassification — detailed above and again under Reliability. Anthropic's savings (78.8%) are the lowest among the clean-run providers, though, and that's worth explaining on its own:

Claude's small tier (Haiku) has a 256-token output cap. When you ask medical or legal questions that need longer answers, Haiku hits that cap. The router detects this on the response and escalates to a larger tier. This is not a bug — it's the router correctly identifying when a cheap tier won't work.

But here's exactly what "escalation" does and doesn't guarantee. If the small-tier answer scores as weak or truncated, the router retries once against the large-tier model. If that retry also fails — a provider timeout, rate limit, or outage — the error isn't propagated as a hard failure. It's caught and logged, and the router falls back to returning the pre-escalation answer rather than a 500, with the event tagged in response metadata (escalation_failed: true).

That's graceful degradation, not a guarantee that escalation always produces a better answer. An earlier draft of this post said "100% cascade recovery," which overstated it — the accurate claim is: escalation is a single retry, fully observable, that never turns into a hard failure on our end, but can't promise the retry itself succeeds when it depends on an upstream provider.

Why Grok saves 30.9%, not 78%+ like Anthropic

This one's worth a full explanation rather than a footnote, because the root cause surprised me. Every other provider in this benchmark gets its savings from calling a genuinely smaller, cheaper-per-token model for easy queries. Grok doesn't have that option right now: all three of its tiers — small, medium, and large — call the same model, grok-4.3. The only lever the router has is a reasoning_effort parameter (none/low/high), which is a compute-budget knob on one model, not a routing decision between differently-priced models.

The measured per-tier cost ratios (relative to LARGE = 1.0) make this concrete:

Provider	SMALL ratio	MEDIUM ratio	LARGE ratio
Anthropic	0.052	0.55	1.0
Grok	0.605	0.924	1.0

Anthropic's small tier (Haiku) costs 5.2% of the large (Opus) because it's a fundamentally smaller model. Grok's small tier costs 60.5% of the large because it's the same model just told to think less — fewer reasoning tokens burned before answering, same per-token price. That gap in ratios is the entire explanation for why Grok tops out at 30.9% savings while Anthropic gets 78.8%+. It's an honest, structurally-limited ceiling given how xAI currently exposes Grok, not a routing weakness on my end. Getting Grok into Anthropic's range would need xAI to expose an actual smaller, cheaper model for the tier — not just an effort knob on grok-4.3.

While investigating this, I also found and fixed two real bugs. Neither explains the 30.9% ceiling itself — they were quietly suppressing accuracy and dollar-separation before the fix, independent of the structural issue above:

_GROK_EFFORT_MAP was defined at function scope instead of module scope in universal_http_client_v20.py — it was intermittently unreachable.
The router's LARGE-tier gate used a hardcoded c_current < 0.65 threshold instead of the calibrated self.c_target (0.600), which made LARGE unreachable for queries scoring between 0.600 and 0.649.

Both are fixed and spot-checked — 4/4 verification queries now route correctly with real dollar separation between tiers ($0.00037 → $0.00175 → $0.0045). The numbers in this post reflect the fixed router.

The same identical-model pattern exists on DeepSeek (see the model table above), which is what prompted checking it too — but DeepSeek's story turned out to be the opposite of Grok's. Full explanation is in the main results table footnote above: DeepSeek's shared small/medium model is so much cheaper than the large tier that it posts 88.3% savings, the highest of any provider in this benchmark, once measured from real per-query dollars rather than an earlier coarse price-ratio estimate.

Latency (Where I Lost Sleep)

Routing decision time (pre-inference): measured in isolation over 3,000 samples — p50 0.42ms, p95 0.68ms, p99 0.97ms, max 1.85ms. Pure in-process computation, no I/O, so it stays sub-millisecond in the typical case.

But end-to-end latency (routing + provider response), same run as the results above:

Provider	P50	P95	P99	Router Overhead
Qwen	880ms	4.4s	5.0s	14.9ms
OpenAI	835ms	2.4s	3.7s	15.0ms
Azure OpenAI	1.2s	2.3s	3.5s	16.7ms
Google	717ms	4.7s	5.4s	15.0ms
Anthropic	1.8s	21.2s	46.2s	16.7ms
Mistral	730ms	4.4s	5.5s	15.6ms
Grok	1.0s	5.5s	7.2s	15.5ms
Cohere	3.7s	11.7s	16.8s	622.7ms
DeepSeek	1.6s	9.2s	22.9s	15.9ms

Router overhead itself is a tight 14.9–16.7ms across 8 of 9 providers — genuinely small and consistent, which is what I'd hoped for. Cohere's 622.7ms is the outlier, and this run I can point to exactly why: it's not steady-state routing cost; it's the retry overhead from the same two timeout events described above. Two calls that had to time out and retry pull the aggregate proxy-overhead number up dramatically even though 98 of Cohere's 100 calls behaved normally.

Anthropic's P99 is the ugliest number in the table — 46.2 seconds. Worth explaining rather than hand-waving: router overhead for Anthropic was 16.7ms this run, same as every other clean provider, so the router itself isn't the delay. Anthropic's IT-vertical queries also had the longest average output in the run (309.9 tokens vs. 65–100 for most other providers on the same vertical), and the LARGE-tier prompts in the query set are genuinely demanding — think cross-border data-sharing risk under Schrems II, or designing zero-trust architecture for a 2,000-person org. That's Opus doing real reasoning work, not an infrastructure stall — though I don't have per-query timing saved from this run to point to the single slowest call directly, which is a gap in the benchmark script I'm fixing next.

Reliability (The Part I Care About)

2 query errors out of 900 total calls (100 per provider × 9 providers, 0.22%) — both on Cohere, both read timeouts (one MEDIUM-tier legal query, one LARGE-tier financial query), 0 across the other 8 providers
Memory footprint: baseline RSS 36.8–45.4 MB, peak RSS 38.0–46.6 MB across providers — stable, no leak pattern across the run
Cascade rate: 0.00% for all 9 providers this run — no query required escalation beyond its initially assigned tier

Separately, an earlier QA pass (May 2026) ran 751 automated tests across unit, integration, and prompt-injection suites — 751/751 passing at the time. Happy to share the suite breakdown if useful; I didn't want to pad this post with test-framework details.

What Happens When Things Break

Clean benchmark numbers are half the story — what matters is what happens when a provider goes down mid-write. Three decisions I'd stand behind: Redis calls fail open (semantic cache and rate limiter degrade gracefully instead of taking the router down with them); every billing write that can't reach Postgres falls back to an fsync'd on-disk log and rehydrates automatically once the database is back, because "the database hiccuped" should never quietly become "we lost a customer's usage record"; and the provider circuit breaker is shared across worker processes, so one outage gets detected once, not four times. All of it is chaos-tested, not just unit-tested — killing Redis and Postgres mid-run, SIGTERM under load, injection attempts while infrastructure is already degraded. Happy to go deeper on any of this in the comments.

Where the Router Struggles (Honestly)

In this 9-provider run, the cascade rate was 0.0% for every provider, across all four verticals (legal, healthcare, finance, IT). That's not a gap in instrumentation; the field is populated and genuinely zero this time. It doesn't mean the mechanism is broken or unused, though. Archived test records show the cascade logic firing exactly as designed elsewhere: small-tier Anthropic queries truncated at the 256-token ceiling, correctly flagged as structurally weak, escalated to the large-tier model, with the escalation reflected in both the response metadata (cascaded: true) and a real, higher cost for that specific query. The mechanism works. This particular 9-provider run's query/tier mix just didn't happen to trigger it.

For real cascade examples, I'm pulling from an earlier, separate 4-provider deep-dive run, where cascading did occur:

Anthropic, on "What does the HIPAA Privacy Rule require from covered entities?" — the small-tier response was insufficient and the router cascaded to a larger tier, producing a 384-token response at $0.00625 (vs. an average $0.00055 for a small-tier query on Anthropic).
Azure OpenAI, on "What does contraindicated mean?" — same pattern, cascading to an 80-token response at $0.000436.

Both fell into the same category: clinical or regulatory terminology needing a qualifying clause to answer correctly. In that run, only Anthropic cascaded on the HIPAA question and only Azure cascaded on "contraindicated" — Google and OpenAI both had zero cascades on the same two queries. The failure mode is systematic (a query category, not noise); which provider hits it on a given day is not — and in the newer, larger run above, it didn't happen at all.

The Economic Reality

CARDIAC-PURR runs on a savings-share model: no per-request fee, no platform fee, and nothing owed in any month where verified savings are $0.

How "verified savings" actually works, and why this is different from the benchmark numbers above. Every month, we measure two things: your baseline (your average LLM/API spend over the trailing 90 days on the same workloads) and your current spend with the router turned on. Verified savings = baseline − current, based on your actual provider invoices plus our usage analytics, and agreed with you before anything is billed. That's the only number we price against.

One distinction matters here: this uses a different baseline than the benchmark section above. The benchmark's "Baseline Cost (Always-LARGE)" is a synthetic comparison — what the same query set would have cost if every query hit the large-tier model — used to measure routing quality in a controlled test. Your actual billing baseline is your own real, historical spend, not a synthetic always-large figure. The two will generally differ, sometimes substantially, depending on how much oversized-model usage you already have in production.

Pricing is a savings-share model that scales automatically with verified monthly savings: free up to 15K requests/month, then 75–82% of verified savings stays with you across all paid tiers, with a monthly cap so there's never an open-ended bill regardless of scale. Full tier breakdown is on the site — it aligns our incentives with yours (if we don't reduce your spend, we don't earn) and gives finance teams a predictable ceiling.

Actual payback depends on your real trailing-90-day baseline, provider mix, and volume — not the benchmark numbers above. Calculate it against your own invoices. (There's also an async batch mode for latency-tolerant queries, roughly +5–6 points on Anthropic and Azure in earlier, not-yet-reverified testing.)

How This Compares to Other Routers, Mechanically

People will ask about OpenRouter and Martian, so here's the technical distinction rather than a pricing pitch. OpenRouter's default Auto Router is a meta-model call: your prompt goes to a model that decides which of dozens of downstream models should handle it, then forwards the request — an extra inference step, with the meta-model's own latency and judgment in the loop, tuned for output quality across providers rather than a specific cost target. Martian's router works differently again — it's built on what they call Model Mapping, converting model internals into a more interpretable form so it can predict a candidate model's expected quality and cost for a given prompt without necessarily running it.

CARDIAC-PURR's routing decision happens before any inference call, using a calibrated complexity score against fixed thresholds (c_target) rather than a second model call or an internals-based prediction. No meta-model latency, no black-box quality prediction — a deterministic score, a tier decision, then one inference call to the chosen tier. The tradeoff is the mirror image of both approaches: no meta-model means faster, more predictable routing decisions, but the classifier is calibrated against my own benchmark distribution, not a general-purpose prompt-quality predictor — which is exactly why the per-provider tier-pairing table above matters more here than a claimed "best model" headline would.

I haven't run a head-to-head benchmark against either of them, and I'd rather say that plainly than imply a comparison I haven't done.

What the Benchmark Doesn't Measure

Production at scale (I tested 100 questions per provider, not 1M)
Provider errors and outages at higher volume — 0.22% here, two timed-out calls on 1 of 9 providers, isn't the same as what a real outage looks like under sustained load
Your specific query distribution (mine was mostly 75/17/8; yours might be different)
What happens when models change (testing was conducted over several months, with the most recent full benchmark run completed today; new models will behave differently)
Correctness of the underlying model's answer, at any tier. The router decides which model answers a query — it doesn't verify the answer itself. For high-stakes domains (legal, medical, financial), that verification step is still on you; a "correctly routed" query and a "correct answer" are different things, and I'd treat the vertical-accuracy numbers above as a tier-selection metric, not an answer-quality guarantee.

How to Use This

The router is available as:

Managed platform: https://www.cardiac-purr.com/PLATFORM/pricing.html — click through, get an API key immediately, free tier needs no credit card
Self-hosted Docker: runs in your own VPC/on-prem, your own Postgres, your own API keys

Source code isn't open (patent-pending) — more on why at the end.

Compliance & Production Status

0.22% overall error rate in testing (2 of 900 total calls, 100 per provider — both Cohere read timeouts), but caveat emptor in production at higher volume
HIPAA BAA available for self-hosted
GDPR Data Processing Agreement ready
U.S. Provisional Patent 64/005,834 (May 24, 2026)

Why I'm Publishing This

I'm tired of reading benchmark posts that hide the methodology. "We tested on our internal queries" tells you nothing about whether it applies to your traffic. "75% simple, 17% explanatory, 8% complex across 4 domains, same 100 questions per provider, across 9 providers" — that, you can actually check your own workload against.

The code runs in Docker. The methodology is reproducible. If you disagree with my query distribution, run your own benchmark.

What Comes Next

I'm expanding the per-provider query count beyond 100 for tighter confidence intervals — Cohere's CI is the widest right now, so that's first. I'm waiting for customer feedback on what breaks in production.

If you want to try it, the Free tier is $0/month forever, up to 15K requests/month, no credit card required — not a trial, a structural zero under that threshold. Use your own API keys (your data, your keys, your infrastructure).

Questions about "why not open-source" are fair — short answer: solo founder, patent-pending, I need the runway before I can afford to give away the moat. That changes once the patent grants or the moat shifts elsewhere. Source access for compliance audits is available under NDA in the meantime.

Emil Igidov
Kraków, Poland
Founder, CARDIAC-PURR · Patent-Pending Inventor.
Learn more about CARDIAC-PURR: https://www.cardiac-purr.com