Cutting your LLM API bill means paying only for the tokens that change the answer. Most teams pay full price for the same system prompt thousands of times a day, run flagship models on trivial tasks, and let reasoning tokens run unchecked. Five techniques, prompt caching, batch APIs, model routing, semantic caching and output discipline, can cut a typical bill by half or more. The maths below uses one real workload throughout, so you can see exactly where the money goes.
Key takeaways
Prompt caching is the biggest win for repeated context: Google discounts cached input by 95% on Gemini 4 Argon, turning a $4,800 monthly input bill into $525.
Batch APIs cost half the standard rate at Google and Anthropic, so anything that can wait a few hours should never run synchronously.
Routing 70% of simple requests to Claude Haiku 5.5 cuts the bill by about two thirds on its own.
Combined, the five techniques take our workload from $6,600 a month to about $580, a 91% reduction.
Every introductory price here doubles on 1 December 2026, so budget on the December rates.
Why do LLM bills explode?
LLM APIs charge per million tokens, and the two sides of the bill are priced very differently. Output tokens typically cost about five times input tokens: Gemini 4 Argon lists $2 per million input tokens and $10 per million output tokens through 30 November 2026, according to Google's Gemini API pricing page. Output pricing includes thinking tokens, so reasoning you never see is billed at the highest rate.
The second problem is repetition. A support assistant that re-sends the same 7,500-token product manual with every question pays to read the manual ten thousand times a day.
The workload we will price everything against
Percentages without a workload are marketing, so here is the one every number here is computed from. A documentation question-answering service handles 10,000 requests a day, 300,000 a month. Each request sends an 8,000-token prompt, 7,500 tokens of shared system prompt and documentation plus 500 tokens of user question, and receives a 600-token answer.
At baseline the monthly bill is $6,600: input is 2.4 billion tokens at $2 per million, or $4,800, and output is 180 million tokens at $10 per million, or $1,800. From 1 December the same workload costs $13,200. Every figure below is arithmetic from published list prices, not a vendor benchmark.
How much does prompt caching actually save?
Prompt caching is the cheapest saving here because it needs no model change and no quality trade-off. When the same prefix of a prompt repeats, the provider charges a fraction of the normal input price for it. Google's implicit caching on Gemini 4 Argon discounts cached input by 95%, charging $0.10 per million tokens instead of $2, and switches on automatically for prompts of at least 4,096 tokens.
On our workload, 7,500 of the 8,000 input tokens are identical every time. Cached, those cost $0.00075 per request; the 500 fresh tokens cost $0.001. Input per request falls from $0.016 to $0.00175, and the monthly input bill falls from $4,800 to $525. Output is untouched at $1,800, so the total bill becomes $2,325, a 65% saving from a change that is mostly about prompt structure.
The cache keys on the exact prefix, so stable content must come first and changing content last; putting the question before the documentation defeats it. Prompts under 4,096 tokens do not qualify, so small, chatty workloads gain little here.
# Cache-friendly prompts: stable content first, changing content last.
SYSTEM_PROMPT = open("system.md").read()
PRODUCT_DOCS = open("docs.md").read()
def build_prompt(user_question: str) -> str:
return SYSTEM_PROMPT + "\n\n" + PRODUCT_DOCS + "\n\n" + user_question
# Anti-pattern: a timestamp or the question before the docs breaks
# the cache prefix on every single request.
When should you use batch APIs?
Batch APIs trade latency for price: you submit a file of requests, collect the results hours later, and the provider charges half the standard rate. Google lists Gemini 4 Argon batch rates at $1 per million input tokens and $5 per million output tokens through November, and Anthropic offers the same 50% discount. On our workload, batching everything cuts input from $4,800 to $2,400 and output from $1,800 to $900, for a total of $3,300, exactly half the baseline.
Batching only fits work that can wait. Nightly reports, bulk classification, evaluation runs and data enrichment are ideal; a live chat reply is not. Split traffic by deadline: anything with a service level above a few hours goes to batch by default. Batching also combines with caching: the cached-input rate on Argon's batch tier is $0.05 per million tokens, so caching plus batching brings the bill to roughly $1,162 a month, an 82% reduction.
# Batch request file (JSONL): one request per line, custom_id for matching.
{"custom_id": "doc-0001", "method": "POST", "url": "/v1/chat/completions",
"body": {"model": "gemini-4-argon",
"messages": [{"role": "user", "content": "Summarise section 4."}]}}
Is model routing worth the complexity?
Model routing means sending each request to the cheapest model that can handle it, instead of sending everything to the flagship. The price gaps are enormous: Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens, against $2 and $10 for Gemini 4 Argon at introductory rates, according to Anthropic's and Google's pricing pages as of 24 October 2026.
Suppose 70% of requests are simple lookups, short summaries and classification, and 30% genuinely need the flagship model. The simple 70% on Haiku costs $168 of input plus $63 of output, or $231. The remaining 30% on Argon costs $1,980. The routed total is $2,211 against the $6,600 baseline, a 67% saving with no caching and no batching at all.
The risk is real and manageable. Route by task type, not by guessing difficulty per request: FAQ lookup, extraction and short summarisation go to the small model, and open-ended reasoning stays on the flagship. Then measure: sample 100 routed answers a week, grade them against the flagship's answers, and move a category back up if quality drops. A router without that feedback loop is a cost cut pretending to be engineering.
# Minimal router: route by task type, fall back to the flagship.
from litellm import completion
SMALL = "claude-haiku-5.5" # $0.10 / $0.50 per 1M tokens
FLAGSHIP = "gemini/gemini-4-argon"
SIMPLE_TASKS = {"faq", "extract", "summarise_short", "classify"}
def answer(task: str, messages: list) -> str:
model = SMALL if task in SIMPLE_TASKS else FLAGSHIP
return completion(model=model, messages=messages).choices[0].message.content
Can semantic caching cut repeat questions?
Semantic caching stores answers you have already given and returns them when a new question means the same thing, even if the wording differs. Where prompt caching saves you from re-reading the same documents, semantic caching saves you from re-generating the same answer. It pays off in support and FAQ workloads, where the same questions arrive in different words all day.
The saving depends entirely on your repeat rate, so measure before building. Log a week of questions, embed them, cluster near-duplicates, and count the share a cache would have served. If fewer than 10% repeat, skip it. If a third repeat, a cache can remove a third of your generation bill outright. A small table keyed by embedding, with a similarity threshold around 0.9 and a time-to-live, is enough to prove the value before investing in a vector store.
How do you stop paying for tokens you never needed?
The fifth technique is output discipline, and it attacks the most expensive tokens on the bill. First, cap reasoning: on models with thinking levels, such as Argon's low, medium and high settings, start at the lowest level that passes your quality checks, because thinking tokens are billed as output. Second, set a real max_tokens limit and a stop sequence instead of letting the model ramble. Third, stop retrying blindly: a failed call retried three times triples the cost of one failure, so retry only on transient errors, with backoff.
Structured output helps too: asking for a JSON object with exactly the fields you need produces shorter, parseable answers than a free-form essay you then summarise. On our workload, trimming the average answer from 600 tokens to 400 tokens saves $600 a month at Argon output prices, with no model change at all.
What does the combined bill look like?
Putting the techniques together: caching alone takes the bill from $6,600 to $2,325; batching alone to $3,300; routing alone to $2,211; caching plus batching to about $1,162. Routing 70% of traffic to Haiku 5.5 while the remaining 30% runs on Argon in batch with its cache hit comes to roughly $580 a month, a 91% reduction from baseline.
That last figure assumes a 90% cache hit rate and that the 70% routed to the small model passes your quality checks. Change those assumptions and the number moves: the techniques stack, but the exact total is yours to compute from your own traffic mix, not to copy from this table.
What is still unknown?
Three honest limits apply. First, cache hit rates are workload-dependent: a 95% discount is worth little if your prompts share almost nothing. Second, vendor benchmark claims are unverified here: Google's launch claim that Argon ranked first on one index had slipped to third within a week, according to the leaderboard itself, so test models on your own prompts. Third, prices move: every rate here is an introductory price, and the December rates, $4 input and $20 output for Argon, are the ones to budget against.
Start with the free wins this week: reorder prompts so the stable content comes first, move overnight work to batch, and route the simple tasks to a small model with a weekly quality sample. Those three changes need no new infrastructure and cover most of the saving. Your December invoice will be decided by what you change before 1 December, not by which model you pick.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)