If you're running an LLM-powered product in production, your monthly AI bill is almost certainly higher than it needs to be, often by a factor of two.
This isn't a pricing problem. Frontier models from OpenAI, Anthropic, Google, and the open-weight ecosystem have never been cheaper per token. The problem is architectural. Most teams send every request to a single high-end model, pay full retail through whatever SDK they wired up first, and then absorb a hidden gateway markup on top of that, without realizing any of it is optional.
This article breaks down where LLM costs actually come from in 2026, why single-provider strategies are leaving 30–50% on the table, and how a multi-model routing approach (plus an honest look at gateway economics) gets that money back.
If you're new to the gateway pattern itself, start with this article. This article assumes you understand the concept and want to focus on the cost layer.
The four hidden cost drivers in LLM APIs
When teams audit their AI spend for the first time, they usually find four cost drivers stacking on top of each other. Most of them are invisible until you go looking.
1. Model overspecification
The single biggest source of waste. A team picks GPT-4-class or Claude Opus-class as their default during prototyping, because it "just works", and then routes every production request through it. Classification, summarization, intent detection, formatting cleanup, simple Q&A, all of it goes through the same flagship model that costs 10–30x more than a mid-tier alternative that would handle the task identically.
In most production traffic mixes, fewer than 20% of requests genuinely require a frontier model. The other 80% could run on Haiku, Gemini Flash, GPT-4o-mini, or a quantized open-weight model with no measurable quality loss. Teams know this in theory but rarely act on it because routing logic is a pain to build.
2. Provider lock-in tax
Single-provider strategies look operationally clean but cost more in three ways:
- No price arbitrage. When a cheaper model ships that meets your quality bar, you can't capture the savings without an SDK migration.
- No fallback options. When your provider has a regional outage, latency spike, or rate-limit event, you either degrade or go down. Both have measurable revenue cost.
- No leverage at renewal. Enterprise customers especially overpay because they have no credible alternative to walk away to.
The operational pain of running multiple SDKs is real, see this for the full story. But it's a one-time cost. The lock-in tax is a recurring one.
3. Gateway markup (the invisible one)
This is the cost driver almost no one audits. Most multi-provider gateways and routing services charge a percentage on top of provider rates (typically 5–15%). They don't always call it "markup." Sometimes it's bundled as "platform fees," "credit conversion," or simply baked into a higher per-token rate than the underlying provider charges directly.
For a team spending $20K/month on LLMs, a 10% gateway markup is $24K/year gone, with no corresponding service improvement. The gateway exists to save you money through routing and reliability; paying it a percentage of your token spend creates a perverse incentive where the gateway makes more when you spend more.
A zero-markup gateway eliminates this entirely. You pay the underlying provider rate, full stop, and the gateway monetizes through other means (subscription, enterprise tier, etc.).
4. Reliability cost
Outages, timeouts, and rate-limit errors aren't "free", every failed request is either a retry (extra tokens) or a lost user interaction (extra churn). Teams without proper failover routing end up paying for both the original request and whatever recovery logic kicks in, often through the same expensive provider.
For a deeper dive on the reliability layer, check article.
Why "switching to a cheaper provider" doesn't actually work
The intuitive response to high LLM costs is "switch to whoever is cheapest." This rarely holds up in practice for three reasons:
Quality varies by task, not by provider. Anthropic models are stronger at certain reasoning tasks; Google's Gemini family is competitive on long-context; OpenAI leads on certain coding benchmarks; open-weight models like Llama and Qwen are excellent for structured extraction. There is no single "cheapest provider". There's only the cheapest provider for a given task.
Cheapest at the model tier ≠ cheapest at the request level. A request that needs three retries on a cheaper model often costs more end-to-end than a single successful call to a more expensive one. Token economics are deceiving without per-task quality measurement.
Switching costs compound. SDK migration, prompt re-tuning, eval re-runs, monitoring updates. The engineering hours required to fully migrate from one provider to another typically exceed the first 6–12 months of savings.
The real cost win isn't switching providers. It's not committing to a single one in the first place.
Multi-model routing: the architecture that actually saves money
Multi-model routing means classifying each incoming request and dispatching it to the model best suited for that request — by quality, latency, cost, or any combination. Done well, it routinely produces 30–50% cost reductions in production traffic without measurable quality regression.
The core decision points:
Tier classification. Split your traffic into tiers based on task complexity. A reasonable starting taxonomy:
· Tier 0 — trivial: keyword extraction, formatting, simple classification. Route to the cheapest capable model.
· Tier 1 — standard: summarization, structured Q&A, content generation under 500 tokens. Route to mid-tier (Haiku, Flash, GPT-4o-mini).
· Tier 2 — complex: multi-step reasoning, long-context analysis, code generation. Route to flagship models.
· Tier 3 — adversarial or high-stakes: anything user-facing where errors are expensive. Route to flagship + add a verification pass.
Routing signals. Tier classification can be done by request metadata (which endpoint was called), prompt analysis (token count, presence of certain keywords or structures), explicit user-tier flags, or a small classifier model running in front of the main routing decision.
**Fallback chains. **Every primary route should have an ordered fallback list. If the primary model is rate-limited or returns a timeout, the gateway transparently retries on the next model in the chain. This is the reliability layer doing double duty as a cost layer, fewer failed requests means fewer wasted tokens.
Quality monitoring. Routing without measurement is gambling. You need per-route quality metrics (eval scores, human spot-checks, downstream conversion data) so you can confidently move traffic from a more expensive model to a cheaper one when the data supports it.
For a full evaluation framework on routing logic, see my last post. It covers the criteria most teams miss when designing their first routing rules.
The cost math, worked out
Here's a representative example from a team running ~10M requests/month before optimization:
Before: All traffic routed through flagship models via a single SDK.
10M requests × ~800 avg tokens × ~$7/1M tokens (blended in/out) = ~$56,000/month
After multi-model routing:
· 60% of traffic (Tier 0–1) routed to Haiku/Flash-tier at ~$0.50/1M → ~$2,400
· 30% of traffic (Tier 2) routed to flagship → ~$16,800
· 10% of traffic (Tier 3) routed to flagship + verification → ~$8,400
· Total: ~$27,600/month — a ~51% reduction
The gateway markup factor:
- On a 5% markup gateway, you'd pay an additional ~$1,920/month (~$23K/year)
- On a 10% markup gateway, ~$3,840/month (~$46K/year)
- On a zero-markup gateway, $0
The markup decision alone is worth more than most teams' entire devops budget for AI infrastructure. (Numbers above are illustrative — your traffic mix and provider rates will shift the math, but the structural ratio holds across most production workloads we've seen.)
Implementing this without rebuilding your stack
You have three viable paths:
1. Build it yourself.
Wire up SDKs for each provider, write your own router and fallback logic, build observability from scratch. Realistic for a senior engineering team with 4–8 weeks of runway. Gives maximum control. Becomes a meaningful maintenance burden as the model landscape shifts (which it does, monthly).
2. Use a marked-up gateway.
Fastest to integrate. Hidden recurring cost. The tradeoff makes sense for very small workloads where the markup is rounding error, but breaks down badly at scale.
3. Use a zero-markup unified API.
You get the unified-API benefit (one SDK, one auth, transparent routing, built-in failover) without paying a percentage of your token spend.
This is the architecture AllToken is built on: a unified API, the performance engine for all tokens, with zero markup as a structural commitment rather than a launch promotion.
If you want to see what the integration looks like end-to-end, the five-minute quickstart walks through a working multi-model setup with fallback routing.
The routing configuration guide covers the tier-based setup described above
And the provider coverage page lists every model accessible through a single endpoint.
What to do this week
If you're auditing LLM costs for the first time, three concrete moves will surface most of the savings:
1. Pull last month's request logs and tag them by complexity.
You'll likely find 50–70% are Tier 0–1 traffic running on flagship models. That's your immediate routing target.
2. Audit your gateway's pricing page for markup language.
Look for "platform fee," "credit multiplier," or per-token rates that don't match the underlying provider's published price. If you can't tell from their docs whether they mark up, assume they do.
3. Run a 1-week shadow test with a routing layer in front of 10% of traffic.
Compare quality (eval scores), latency, and cost per request. Most teams see the cost delta clearly within 5 days.
The core insight is simple: LLM costs are a routing problem and a markup problem, not a pricing problem.
Solve the routing layer and pick a gateway that doesn't tax your token spend, and 30–50% of your bill comes back automatically, no provider switch required.
Building on AllToken?
_The quickstart gets you running on a unified API across all frontier models in under five minutes. _
Top comments (0)