CARDIAC-PURR — I built an LLM cost-router. Here's what 100 questions per provider, across 9 providers, taught me.

You ship a feature, it calls GPT-4, it works beautifully, you move on. Six months later, someone checks the logs and — surprise — half your traffic is "what's my account number" going through a model smart enough to pass the bar exam. This isn't a one-off mistake, either — it's the default outcome of not routing at all. Support-queue traffic is mostly simple, but "mostly simple" tends to get shipped through whatever model handled the hard case that made you reach for a big model in the first place. The model's not the problem. The lack of a router is.

I built a router to fix that. I'm an attorney by training, not an engineer, so instead of trusting my own intuition (or published benchmarks), I decided to measure it properly.

What I Built

CARDIAC-PURR sits between your app and your LLM providers, monitoring each query and deciding whether it actually needs a large model or would do fine with a cheaper one. I tested it against 9 providers, each with its own small→medium→large split:

Provider	Small	Medium	Large
Anthropic	claude-haiku-4-5-20251001	claude-sonnet-4-6	claude-opus-4-6
OpenAI	gpt-4.1-nano	gpt-4.1-mini	gpt-4.1
Google	gemini-2.5-flash-lite	gemini-2.5-flash	gemini-2.5-pro
Azure OpenAI	gpt-4.1-nano	gpt-4.1-mini	gpt-4.1
Mistral	mistral-small-latest	mistral-medium-latest	mistral-large-latest
DeepSeek	deepseek-v4-flash	deepseek-v4-flash†	deepseek-v4-pro
Cohere	command-r7b-12-2024	command-r-plus-08-2024	command-a-03-2025
Grok	grok-4.3§	grok-4.3§	grok-4.3
Qwen	qwen-turbo	qwen-plus	qwen-max

† DeepSeek's small and medium tiers point at the identical model (deepseek-v4-flash) — there's no true 3-way split for DeepSeek, just flash vs. pro. That sounds like a limitation, and I originally treated it as one, but it turns out to be the reason DeepSeek posts the highest savings of any provider tested: flash is dramatically cheaper than pro, and 92% of traffic never touches pro at all. Full explanation, including why an earlier 46.0% estimate for DeepSeek was wrong, is below.

§ This resolves an open flag from an earlier draft — I'd noted that xAI retired the standalone grok-4-1-fast endpoints on May 15, 2026 and worried the small/medium/large split might secretly be hitting one model. Turns out that's exactly what's happening, confirmed directly from the router's own config: all three Grok tiers call grok-4.3. The only lever available is a reasoning_effort parameter (none/low/high) — a compute-budget knob on one model, not a routing decision between differently-priced models. Details and what this does to Grok's savings ceiling are below.

Tier selection happens before inference, based on query complexity — no waiting for a model response, no trained classifier, no embedding model.

Every query gets a complexity score against calibrated thresholds (self.c_target, currently 0.600 for the LARGE-tier gate). Most queries score nowhere near a boundary — the 100-question set was intentionally realistic, not adversarial, and the accuracy numbers below reflect that. Queries that land close to a threshold are exactly where I'd expect the router to be least confident, and it's also where the escalation safety net matters most: if a borderline SMALL-tier pick produces a weak or truncated response, that's caught and escalated rather than silently returned. I don't yet have a clean breakdown of accuracy specifically on near-boundary queries in this run — that's on my list for the next pass.

There's also a safety net: if the tier it picked gives a truncated or weak-looking response, the router escalates to a bigger model. That's a fallback, not the main path.

Under the hood, there are 15 distinct cost-saving mechanisms and 5 layers of failure/quality prevention (traffic protection, security, cost control, data durability, observability) working together. Patent-pending, so I won't break them down in detail here — but the numbers below are what all of them add up to in production.

How I Tested It

I've been running this in production since March. For this post, I wanted a clean, controlled comparison: same 100 questions, asked to nine providers, so the numbers below are apples-to-apples.

I ran 100 queries per provider — Anthropic, OpenAI, Google, Azure OpenAI, Mistral, DeepSeek, Cohere, Grok, and Qwen. Real money spent. Real API calls. Same 100 questions across all nine providers.

The queries were intentionally unglamorous: 75% factual definitions ("What does NDA stand for?"), 17% explanatory ("Describe the main steps in..."), 8% complex reasoning. This is what actual support queues look like. The full 100-question set, labeled by expected tier and vertical, is linked at the bottom of this post — if you think my distribution is wrong for your workload, don't take my word for it; run it yourself.

I measured two things separately because mixing them is how benchmarks lie:

1. Did the router pick the right tier? (routing accuracy)
2. Did accuracy hold up across domains, not just in aggregate? (vertical accuracy — legal, healthcare, finance, IT)

The Numbers

A note before the table: with n=100 per provider, 100% accuracy doesn't mean zero future misclassifications — it means the 95% confidence interval is [96.4%, 100%]. I'm expanding sample size (see "What Comes Next"), and in the meantime the full methodology below is reproducible — run it on your own data and see where your numbers land.

Run date: July 12, 2026 — router commit 57175b7, which fixed a warmup-accounting bug that had previously forced the first ~47 post-warmup queries to the LARGE tier regardless of actual complexity. This is a clean rerun; the numbers below supersede any earlier pass.

Provider	Accuracy	Savings	Fail %	Errors	Vertical (Legal/Health/Finance/IT)
DeepSeek	100%	88.3%†	0.0%	0/100	100/100/100/100
Qwen	100%	86.3%	0.0%	0/100	100/100/100/100
OpenAI	100%	84.9%	0.0%	0/100	100/100/100/100
Azure OpenAI	100%	84.9%	0.0%	0/100	100/100/100/100
Google	100%	84.4%	0.0%	0/100	100/100/100/100
Anthropic	100%	78.8%	0.0%	0/100	100/100/100/100
Mistral	100%	77.7%	0.0%	0/100	100/100/100/100
Cohere⚠	98.0%	73.9%‡	2.0%	2/100	96/100/96/100
Grok	100%	30.9%§	0.0%	0/100	100/100/100/100

† DeepSeek is the best performer in this benchmark, and the reason flips an earlier, wrong estimate: a prior pass reported 46.0% from a coarse price-ratio table, not real dollars. The actual number, computed from real per-query costs against DeepSeek's LARGE-tier price, is 1 - $0.0059/$0.0505 = 88.3%. Full mechanism explained in the DeepSeek/Grok section below.

‡ Cohere's medium and large tiers are priced identically ($12.50/MTok combined), so there's no discount available between those two tiers by design — that structural fact caps Cohere's best-case savings below the other providers' ceiling regardless of errors. Separately, and unrelated to pricing: Cohere hit 98.0% accuracy (95% CI: 93.0–99.4%) because of two transient read timeouts — one on a legal MEDIUM-tier query, one on a financial LARGE-tier query. In both cases the router picked the correct tier and the underlying API call timed out before returning a response; these are provider/network-side failures, not routing misclassifications.

§ Grok is the lowest-savings provider in this run — structural, not a routing weakness. Full explanation below.

⚠ Cohere is the only provider with a nonzero error rate in this run (2/100 — real timed-out API calls, distinct from cascades, which were zero for every provider this run). Every other provider, including Anthropic, came back completely clean: 100% accuracy, 0 errors, no vertical dips.

*Savings % isn't comparable across providers as a ranking — it reflects each provider's own small-vs-large price ratio, not routing quality. Routing accuracy is the fair cross-provider comparison, and on that metric 8 of 9 providers hit 100%.

All queries were pre-labeled into expected tiers (small/medium/large) based on response complexity. Routing accuracy compares the router's decision against these labels.

Methodology: labeling, baseline, and confidence intervals. Given the actual query mix (75% small / 17% medium / 8% large), a naive router that always guesses "small" gets 75% accuracy for free, with zero routing logic — that's the honest baseline to beat. (Random guessing among three tiers gives 33%, the more commonly cited comparison, but it's the weaker one given this distribution.)

Per-provider 95% confidence intervals (Clopper-Pearson): providers at 100/100 sit at [96.38%, 100.00%]; Cohere at 98/100 sits at [93.0%, 99.4%].

8 of 9 providers hit 100% routing accuracy against my 96% target; only Cohere (98%) came in under it, and that's down to two transient timeouts, not a misclassification — detailed above and again under Reliability. Anthropic's savings (78.8%) are the lowest among the clean-run providers, though, and that's worth explaining on its own:

Claude's small tier (Haiku) has a 256-token output cap. When you ask medical or legal questions that need longer answers, Haiku hits that cap. The router detects this on the response and escalates to a larger tier. This is not a bug — it's the router correctly identifying when a cheap tier won't work.

But here's exactly what "escalation" does and doesn't guarantee. If the small-tier answer scores as weak or truncated, the router retries once against the large-tier model. If that retry also fails — a provider timeout, rate limit, or outage — the error isn't propagated as a hard failure. It's caught and logged, and the router falls back to returning the pre-escalation answer rather than a 500, with the event tagged in response metadata (escalation_failed: true).

That's graceful degradation, not a guarantee that escalation always produces a better answer. An earlier draft of this post said "100% cascade recovery," which overstated it — the accurate claim is: escalation is a single retry, fully observable, that never turns into a hard failure on our end, but can't promise the retry itself succeeds when it depends on an upstream provider.

Why Grok saves 30.9%, not 78%+ like Anthropic

This one's worth a full explanation rather than a footnote, because the root cause surprised me. Every other provider in this benchmark gets its savings from calling a genuinely smaller, cheaper-per-token model for easy queries. Grok doesn't have that option right now: all three of its tiers — small, medium, and large — call the same model, grok-4.3. The only lever the router has is a reasoning_effort parameter (none/low/high), which is a compute-budget knob on one model, not a routing decision between differently-priced models.

The measured per-tier cost ratios (relative to LARGE = 1.0) make this concrete:

Provider	SMALL ratio	MEDIUM ratio	LARGE ratio
Anthropic	0.052	0.55	1.0
Grok	0.605	0.924	1.0

Anthropic's small tier (Haiku) costs 5.2% of the large (Opus) because it's a fundamentally smaller model. Grok's small tier costs 60.5% of the large because it's the same model just told to think less — fewer reasoning tokens burned before answering, same per-token price. That gap in ratios is the entire explanation for why Grok tops out at 30.9% savings while Anthropic gets 78.8%+. It's an honest, structurally-limited ceiling given how xAI currently exposes Grok, not a routing weakness on my end. Getting Grok into Anthropic's range would need xAI to expose an actual smaller, cheaper model for the tier — not just an effort knob on grok-4.3.

While investigating this, I also found and fixed two real bugs. Neither explains the 30.9% ceiling itself — they were quietly suppressing accuracy and dollar-separation before the fix, independent of the structural issue above:

1. _GROK_EFFORT_MAP was defined at function scope instead of module scope in universal_http_client_v20.py — it was intermittently unreachable.
2. The router's LARGE-tier gate used a hardcoded c_current < 0.65 threshold instead of the calibrated self.c_target (0.600), which made LARGE unreachable for queries scoring between 0.600 and 0.649.

Both are fixed and spot-checked — 4/4 verification queries now route correctly with real dollar separation between tiers ($0.00037 → $0.00175 → $0.0045). The numbers in this post reflect the fixed router.

The same identical-model pattern exists on DeepSeek (see the model table above), which is what prompted checking it too — but DeepSeek's story turned out to be the opposite of Grok's. Full explanation is in the main results table footnote above: DeepSeek's shared small/medium model is so much cheaper than the large tier that it posts 88.3% savings, the highest of any provider in this benchmark, once measured from real per-query dollars rather than an earlier coarse price-ratio estimate.

Latency (Where I Lost Sleep)

Routing decision time (pre-inference): measured in isolation over 3,000 samples — p50 0.42ms, p95 0.68ms, p99 0.97ms, max 1.85ms. Pure in-process computation, no I/O, so it stays sub-millisecond in the typical case.

But end-to-end latency (routing + provider response), same run as the results above:

Provider	P50	P95	P99	Router Overhead
Qwen	880ms	4.4s	5.0s	14.9ms
OpenAI	835ms	2.4s	3.7s	15.0ms
Azure OpenAI	1.2s	2.3s	3.5s	16.7ms
Google	717ms	4.7s	5.4s	15.0ms
Anthropic	1.8s	21.2s	46.2s	16.7ms
Mistral	730ms	4.4s	5.5s	15.6ms
Grok	1.0s	5.5s	7.2s	15.5ms
Cohere	3.7s	11.7s	16.8s	622.7ms
DeepSeek	1.6s	9.2s	22.9s	15.9ms

Router overhead itself is a tight 14.9–16.7ms across 8 of 9 providers — genuinely small and consistent, which is what I'd hoped for. Cohere's 622.7ms is the outlier, and this run I can point to exactly why: it's not steady-state routing cost; it's the retry overhead from the same two timeout events described above. Two calls that had to time out and retry pull the aggregate proxy-overhead number up dramatically even though 98 of Cohere's 100 calls behaved normally.

Anthropic's P99 is the ugliest number in the table — 46.2 seconds. Worth explaining rather than hand-waving: router overhead for Anthropic was 16.7ms this run, same as every other clean provider, so the router itself isn't the delay. Anthropic's IT-vertical queries also had the longest average output in the run (309.9 tokens vs. 65–100 for most other providers on the same vertical), and the LARGE-tier prompts in the query set are genuinely demanding — think cross-border data-sharing risk under Schrems II, or designing zero-trust architecture for a 2,000-person org. That's Opus doing real reasoning work, not an infrastructure stall — though I don't have per-query timing saved from this run to point to the single slowest call directly, which is a gap in the benchmark script I'm fixing next.

Reliability (The Part I Care About)

• 2 query errors out of 900 total calls (100 per provider × 9 providers, 0.22%) — both on Cohere, both read timeouts (one MEDIUM-tier legal query, one LARGE-tier financial query), 0 across the other 8 providers
• Memory footprint: baseline RSS 36.8–45.4 MB, peak RSS 38.0–46.6 MB across providers — stable, no leak pattern across the run
• Cascade rate: 0.00% for all 9 providers this run — no query required escalation beyond its initially assigned tier

Separately, an earlier QA pass (May 2026) ran 751 automated tests across unit, integration, and prompt-injection suites — 751/751 passing at the time. Happy to share the suite breakdown if useful; I didn't want to pad this post with test-framework details.

What Happens When Things Break

Clean benchmark numbers are half the story — what matters is what happens when a provider goes down mid-write. Three decisions I'd stand behind: Redis calls fail open (semantic cache and rate limiter degrade gracefully instead of taking the router down with them); every billing write that can't reach Postgres falls back to an fsync'd on-disk log and rehydrates automatically once the database is back, because "the database hiccuped" should never quietly become "we lost a customer's usage record"; and the provider circuit breaker is shared across worker processes, so one outage gets detected once, not four times. All of it is chaos-tested, not just unit-tested — killing Redis and Postgres mid-run, SIGTERM under load, injection attempts while infrastructure is already degraded. Happy to go deeper on any of this in the comments.

Where the Router Struggles (Honestly)

In this 9-provider run, the cascade rate was 0.0% for every provider, across all four verticals (legal, healthcare, finance, IT). That's not a gap in instrumentation; the field is populated and genuinely zero this time. It doesn't mean the mechanism is broken or unused, though. Archived test records show the cascade logic firing exactly as designed elsewhere: small-tier Anthropic queries truncated at the 256-token ceiling, correctly flagged as structurally weak, escalated to the large-tier model, with the escalation reflected in both the response metadata (cascaded: true) and a real, higher cost for that specific query. The mechanism works. This particular 9-provider run's query/tier mix just didn't happen to trigger it.

For real cascade examples, I'm pulling from an earlier, separate 4-provider deep-dive run, where cascading did occur:

1. Anthropic, on "What does the HIPAA Privacy Rule require from covered entities?" — the small-tier response was insufficient and the router cascaded to a larger tier, producing a 384-token response at $0.00625 (vs. an average $0.00055 for a small-tier query on Anthropic).
2. Azure OpenAI, on "What does contraindicated mean?" — same pattern, cascading to an 80-token response at $0.000436.

Both fell into the same category: clinical or regulatory terminology needing a qualifying clause to answer correctly. In that run, only Anthropic cascaded on the HIPAA question and only Azure cascaded on "contraindicated" — Google and OpenAI both had zero cascades on the same two queries. The failure mode is systematic (a query category, not noise); which provider hits it on a given day is not — and in the newer, larger run above, it didn't happen at all.

The Economic Reality

CARDIAC-PURR runs on a savings-share model: no per-request fee, no platform fee, and nothing owed in any month where verified savings are $0.

How "verified savings" actually works, and why this is different from the benchmark numbers above. Every month, we measure two things: your baseline (your average LLM/API spend over the trailing 90 days on the same workloads) and your current spend with the router turned on. Verified savings = baseline − current, based on your actual provider invoices plus our usage analytics, and agreed with you before anything is billed. That's the only number we price against.

One distinction matters here: this uses a different baseline than the benchmark section above. The benchmark's "Baseline Cost (Always-LARGE)" is a synthetic comparison — what the same query set would have cost if every query hit the large-tier model — used to measure routing quality in a controlled test. Your actual billing baseline is your own real, historical spend, not a synthetic always-large figure. The two will generally differ, sometimes substantially, depending on how much oversized-model usage you already have in production.

Pricing is a savings-share model that scales automatically with verified monthly savings: free up to 15K requests/month, then 75–82% of verified savings stays with you across all paid tiers, with a monthly cap so there's never an open-ended bill regardless of scale. Full tier breakdown is on the site — it aligns our incentives with yours (if we don't reduce your spend, we don't earn) and gives finance teams a predictable ceiling.

Actual payback depends on your real trailing-90-day baseline, provider mix, and volume — not the benchmark numbers above. Calculate it against your own invoices. (There's also an async batch mode for latency-tolerant queries, roughly +5–6 points on Anthropic and Azure in earlier, not-yet-reverified testing.)

How This Compares to Other Routers, Mechanically

People will ask about OpenRouter and Martian, so here's the technical distinction rather than a pricing pitch. OpenRouter's default Auto Router is a meta-model call: your prompt goes to a model that decides which of dozens of downstream models should handle it, then forwards the request — an extra inference step, with the meta-model's own latency and judgment in the loop, tuned for output quality across providers rather than a specific cost target. Martian's router works differently again — it's built on what they call Model Mapping, converting model internals into a more interpretable form so it can predict a candidate model's expected quality and cost for a given prompt without necessarily running it.

CARDIAC-PURR's routing decision happens before any inference call, using a calibrated complexity score against fixed thresholds (c_target) rather than a second model call or an internals-based prediction. No meta-model latency, no black-box quality prediction — a deterministic score, a tier decision, then one inference call to the chosen tier. The tradeoff is the mirror image of both approaches: no meta-model means faster, more predictable routing decisions, but the classifier is calibrated against my own benchmark distribution, not a general-purpose prompt-quality predictor — which is exactly why the per-provider tier-pairing table above matters more here than a claimed "best model" headline would.

I haven't run a head-to-head benchmark against either of them, and I'd rather say that plainly than imply a comparison I haven't done.

What the Benchmark Doesn't Measure

• Production at scale (I tested 100 questions per provider, not 1M)
• Provider errors and outages at higher volume — 0.22% here, two timed-out calls on 1 of 9 providers, isn't the same as what a real outage looks like under sustained load
• Your specific query distribution (mine was mostly 75/17/8; yours might be different)
• What happens when models change (testing was conducted over several months, with the most recent full benchmark run completed today; new models will behave differently)
• Correctness of the underlying model's answer, at any tier. The router decides which model answers a query — it doesn't verify the answer itself. For high-stakes domains (legal, medical, financial), that verification step is still on you; a "correctly routed" query and a "correct answer" are different things, and I'd treat the vertical-accuracy numbers above as a tier-selection metric, not an answer-quality guarantee.

How to Use This

The router is available as:

1. Managed platform: https://www.cardiac-purr.com/PLATFORM/pricing.html — click through, get an API key immediately, free tier needs no credit card
2. Self-hosted Docker: runs in your own VPC/on-prem, your own Postgres, your own API keys

Source code isn't open (patent-pending) — more on why at the end.

Compliance & Production Status

• 0.22% overall error rate in testing (2 of 900 total calls, 100 per provider — both Cohere read timeouts), but caveat emptor in production at higher volume
• HIPAA BAA available for self-hosted
• GDPR Data Processing Agreement ready
• U.S. Provisional Patent 64/005,834 (May 24, 2026)

Why I'm Publishing This

I'm tired of reading benchmark posts that hide the methodology. "We tested on our internal queries" tells you nothing about whether it applies to your traffic. "75% simple, 17% explanatory, 8% complex across 4 domains, same 100 questions per provider, across 9 providers" — that, you can actually check your own workload against.

The code runs in Docker. The methodology is reproducible. If you disagree with my query distribution, run your own benchmark.

What Comes Next

I'm expanding the per-provider query count beyond 100 for tighter confidence intervals — Cohere's CI is the widest right now, so that's first. I'm waiting for customer feedback on what breaks in production.

If you want to try it, the Free tier is $0/month forever, up to 15K requests/month, no credit card required — not a trial, a structural zero under that threshold. Use your own API keys (your data, your keys, your infrastructure).

Questions about "why not open-source" are fair — short answer: solo founder, patent-pending, I need the runway before I can afford to give away the moat. That changes once the patent grants or the moat shifts elsewhere. Source access for compliance audits is available under NDA in the meantime.

---

Emil Igidov
Kraków, Poland
Founder, CARDIAC-PURR · Patent-Pending Inventor.
Learn more about CARDIAC-PURR: https://www.cardiac-purr.com