DEV Community

dubleCC
dubleCC

Posted on • Originally published at heycc.cn

How to Reduce LLM API Costs in 2026: Prompt Caching, Batch APIs, and Model Routing

#llmapi #promptcaching #batchapi #modelrouting

Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

How to Reduce LLM API Costs in 2026: Prompt Caching, Batch APIs, and Model Routing

Your LLM bill reduces to one equation: cost = tokens × price-per-token, summed across every request and every model. There are only three levers that actually move it. You can pay less per token (caching and batch discounts), you can route cheaper work to cheaper models, or you can send fewer tokens. The first two carry the most weight in 2026, because that is where the biggest, lowest-effort savings live — a cache hit on a frontier model now costs 90% less than a fresh read, and the Batch API knocks 50% off both input and output. Stack them and a workload can run at a fraction of what the naive rate implies.

The dollar figures below are snapshots from the official pricing pages on 2026-06-28; the mechanics shift more slowly than the prices, so each section explains how a lever works first and treats the specific numbers as something to re-check against the live table before committing a workload.

The three levers, ranked by effort vs. payoff

Lever Typical savings Code effort Latency impact Best for
Prompt / context caching Up to 90% on repeated input tokens Low (one field, or automatic) Faster (cache reads are quicker) Long, stable system prompts, RAG context, multi-turn chat
Batch API Flat 50% on input and output Low (different endpoint) Slower (async, up to 24h) Evals, backfills, summarization, classification at scale
Model routing 60–95% on routed requests Medium (a router + heuristics) Neutral or faster Mixed workloads with easy and hard requests

The punchline: caching and batch are nearly free to adopt and stack multiplicatively. Routing takes more engineering but is the only lever that helps when you're already caching and batching. Do them in that order.

Lever 1: Prompt caching — stop paying full price for the same tokens

If you send the same large block of text on many requests — a system prompt, a tool schema, a retrieved document, a conversation history — you are re-billing for tokens the model has already processed. Caching fixes that.

Anthropic (Claude) uses explicit cache breakpoints. You mark content with cache_control and pay a write premium once, then read cheaply:

  • 5-minute cache write: 1.25× base input price
  • 1-hour cache write: 2× base input price
  • Cache read (hit): 0.1× base input price (a 90% discount)

The break-even is fast. A 5-minute cache pays for itself after a single read (you spent 1.25× to write, but each read costs 0.1× instead of 1×). A 1-hour cache pays off after two reads. So cache anything you'll reuse within the window.

OpenAI caching is automatic — no code changes — and kicks in for prompts longer than 1,024 tokens. The original 2024 launch advertised a flat 50% discount on cached input, but the 2026 pricing tables show it has deepened and is now model-specific: gpt-5.5, for example, lists standard input at $5.00/MTok versus cached input at $0.50/MTok — a 90% cached-input discount. Don't assume 50%; read the live table.

Google Gemini has two modes. Implicit caching is on by default for Gemini 2.5+ models ("There is nothing you need to do" — savings are passed through automatically on cache hits). Explicit caching lets you declare a cache with a custom TTL for a guaranteed discount. The cached-token discount is 90% on Gemini 2.5+ (75% on 2.0). In both modes you pay standard rate for the initial cache-creation tokens; explicit caching adds a storage cost for the duration you keep the cache alive.

Caching gotchas

  • Cache writes cost more than a normal read. If your "reusable" prefix is only ever read once, caching loses money on Anthropic. Cache only what you'll reuse inside the TTL.
  • The cached prefix must be byte-identical. A timestamp or user name injected before the stable block busts the cache. Put volatile content after the cached prefix.
  • Storage isn't free on explicit Gemini caches. Long TTLs on rarely-hit caches can cost more than they save.

Lever 2: Batch APIs — 50% off for work that can wait

If a workload doesn't need an answer right now — nightly evals, document classification, embedding a corpus, generating product descriptions — submit it as a batch. All three major providers give the same deal:

  • Anthropic Message Batches: 50% off input and output; 24-hour target, most finish in under an hour.
  • OpenAI Batch API: flat 50% off input and output; up to 24h, usually faster.
  • Gemini Batch Mode: 50% of standard cost; 24-hour target, higher rate limits.

The trade is latency, not quality — same models, same outputs. The only real cost is restructuring synchronous code into a submit-poll-retrieve flow, and handling partial failures inside a batch.

Discounts stack — and most cost estimates ignore it

Batch and caching are not either/or. They multiply. OpenAI's own pricing makes this explicit for gpt-5.5: standard input $5.00 → batch input $2.50 → batch cached input $0.25. That's caching applied on top of the already-halved batch rate — a 95% reduction from the standard input price. Anthropic states the same: "Batch API and prompt caching discounts can be combined," and the caching multipliers "stack with the Batch API discount."

So the cheapest possible input token on a frontier model in 2026 is: batch rate × cache-read multiplier. For a workload that is both deferrable and repetitive, that's the configuration to target.

Three stacking levers for LLM cost reduction: caching, batch, and routing, with a worked cost waterfall

Lever 3: Model routing — send cheap work to cheap models

Not every request needs your flagship model. Routing classifies each request and dispatches it to the cheapest model that can handle it, reserving the expensive model for genuinely hard work. The price gaps are large enough that even crude routing pays off.

Verified Anthropic base prices (per MTok, 2026-06-28):

Model Input Output Cache read Batch input Batch output
Claude Opus 4.8 $5.00 $25.00 $0.50 $2.50 $12.50
Claude Sonnet 4.6 $3.00 $15.00 $0.30 $1.50 $7.50
Claude Haiku 4.5 $1.00 $5.00 $0.10 $0.50 $2.50

Moving a high-volume classification task from Opus 4.8 to Haiku 4.5 cuts the per-token rate by 5× on input and output before any other discount. To put your own token counts against every model in this table — caching and batch toggles included — use the LLM API cost calculator. A practical router uses cheap heuristics first — input length, presence of code, a small classifier model — and only escalates to the frontier model when a confidence check fails or the task is flagged as complex. The general rule: Haiku-class for extraction/classification/routing, Sonnet-class for most production reasoning, Opus-class for the hardest multi-step problems.

Watching the levers compound: a worked example

Anthropic's own worked example: 10,000 support-ticket conversations averaging ~3,700 tokens each, on Haiku 4.5 ($1/MTok in, $5/MTok out), total ~$37.00. Now apply the levers to a similar 10,000-request job where each request shares an 8,000-token instruction-and-policy prefix:

  • Naive (Sonnet 4.6, no optimization): the shared prefix alone is 8,000 × 10,000 = 80M input tokens × $3/MTok = $240 just for the repeated prefix.
  • Add caching: write once, read 9,999 times at 0.1× → roughly 80M × $0.30/MTok ≈ $24. ~90% off the prefix.
  • Add batch on top: halve it again → ~$12.
  • Route the easy 70% to Haiku: the prefix cost on routed requests drops a further ~3× for that slice.

The exact figure depends on output length and routing accuracy, but the structure holds: each lever compounds on the last.

Don't forget the guardrail: spend limits aren't all hard caps

Optimizing per-token cost won't save you from a runaway loop. Know your provider's controls:

  • OpenAI's project "Monthly budget" is a soft, notification-only threshold. When usage exceeds it, requests keep being processed and you keep being billed — it emails/alerts you (default alert at 100%) but does not enforce a hard stop. OpenAI replaced its old hard monthly cap with this notification system.
  • Anthropic's Claude Console supports per-workspace and per-API-key spend limits plus "Add notification" alerts, and a Cost page with per-model, monthly, and daily breakdowns. Spend limits are configured in the Console UI, not via a programmatic API.

Treat budgets as alarms, not brakes, unless your provider explicitly enforces a hard cap. Add your own application-side kill switch for true protection. For the broader picture on choosing between providers, see our Claude vs GPT vs Gemini API comparison.

Sequencing the levers per workload

Run each workload through this in order:

  1. Is the input repetitive within minutes/hours? → Enable caching. Put volatile tokens after the stable prefix.
  2. Can it tolerate up to 24h latency? → Submit via the Batch API. Stack it with caching.
  3. Are many requests easy? → Add a router; default to the cheapest capable model, escalate on low confidence.
  4. Could a loop blow the budget? → Set Console spend limits and an app-side cap, since some "budgets" are alerts only.

Sourcing the pricing math

The cache multipliers, batch rates, and per-MTok prices trace back to the pages in Sources: Anthropic's pricing, prompt-caching, and batch-processing docs for the Claude figures and the "discounts combine" language; OpenAI's API pricing and Batch guide for the gpt-5.5 standard/batch/cached-input ladder; and Google's Gemini context-caching and Batch API docs for the implicit/explicit modes and the 2.5-vs-2.0 discount tiers. The OpenAI and Anthropic spend-control behaviour comes from their respective Console/projects help pages, also linked below. One caveat is worth carrying forward specifically: cached-input discounts are model-specific and have only deepened over time — OpenAI's cached input started at a flat 50% and now reaches 90% on gpt-5.5 — so the per-row prices here are worth a glance at the live table before you size a budget against them.

Sources

Top comments (0)