Six Levers That Cut an LLM Bill Without Changing Model

Every claim in this page is arithmetic you do with your own numbers, not a percentage from somebody else’s workload. That is deliberate: the share each lever contributes depends entirely on the shape of your traffic, and a borrowed percentage is worse than useless because it sets the wrong priority.

Find the shape of the bill first

Four numbers decide which levers can possibly matter. Nothing below is worth reading until you have them.

Per prompt template, over a representative week:

  N        requests
  P        mean prompt tokens
  O        mean completion tokens
  p_in     price per input token
  p_out    price per output token
  r        attempts per served request (retries + failovers), >= 1.0

  spend  =  N * r * (P * p_in  +  O * p_out)

Two ratios decide everything:

  output_share = (O * p_out) / (P * p_in + O * p_out)
  repeat_share = (r - 1) / r

If output_share is high, levers 2 and 3 dominate.
If P is large and repeated across requests, lever 1 dominates.
If r is meaningfully above 1.0, lever 4 is free money.

Output tokens are typically priced several times higher than input tokens, so a workload with a long prompt and a short answer and one with a short prompt and a long answer need opposite work. Compute output_share before choosing.

Lever 1: stop paying for context you resend

Any system with a stable preamble — a system prompt, a tool schema, a few-shot block, a retrieved corpus that rarely changes — resends the same tokens on every request and pays full price each time. Prompt caching prices those repeated tokens lower after the first call.

Let  P = P_fixed + P_variable
     c = cache price multiplier on the fixed part (provider-specific,
         commonly a large discount on a cache hit)
     h = cache hit rate

  before = P * p_in
  after  = P_fixed * p_in * (1 - h + h*c)  +  P_variable * p_in

  saving_fraction_on_input = (P_fixed / P) * h * (1 - c)

WORKED (label your own): P_fixed 6,000, P_variable 400,
h 0.9, c 0.1
  P_fixed / P = 6000 / 6400 = 0.9375
  saving on input = 0.9375 * 0.9 * 0.9 = 0.759  -> 76% off the input line

Three conditions decide whether you get that. The fixed part must come first in the message array, byte-identical, every time; the cache must not be invalidated by a per-request field slipped into the preamble; and the hit rate must be measured rather than assumed. Ordering is the one people get wrong — cache ordering and what caching actually saves cover both.

Lever 2: cap output, because output is the expensive half

Output is priced higher per token and is usually the larger share of the bill, and it is the line most inflated by things nobody asked for: restated questions, apologetic preambles, summaries of what was just said.

  saving = N * r * (O_before - O_after) * p_out

WORKED: N 200,000/month, r 1.0, p_out 0.000010 per token
  O_before 700, O_after 420  (a 40% reduction from format discipline)
  saving = 200,000 * 280 * 0.000010 = $560/month

Where O_after comes from, in rough order of size:
  - a hard max_tokens that reflects the real answer length
  - a schema that has no field for preamble
  - "answer only with X" plus an example, which works far better
    than an instruction alone
  - removing "explain your reasoning" where nothing reads it

Setting a maximum token limit truncates rather than compresses. If the model was going to write 700 tokens, a cap at 420 gives you 420 tokens of a 700-token answer, which is a different failure. The cap is a backstop; the schema and the instruction are what actually shorten it. Controlling output length covers the distinction.

Lever 3: route the easy majority to a cheaper model

Most workloads have a large easy majority and a small hard tail, and send both to the same model. A cascade sends everything to a cheap model first and escalates only what fails a check.

Let  f = fraction escalated to the expensive model
     C_cheap, C_exp = cost per request on each

  cascade = C_cheap + f * C_exp
  break-even f* where cascade = C_exp:
     f* = 1 - (C_cheap / C_exp)

WORKED: C_cheap $0.0004, C_exp $0.0090
  f* = 1 - 0.044 = 0.956
  -> the cascade is cheaper unless you escalate more than 96% of traffic
  At f = 0.20:  cascade = 0.0004 + 0.0018 = $0.0022  (76% saving)
  At f = 0.50:  cascade = 0.0004 + 0.0045 = $0.0049  (46% saving)

The break-even is far more forgiving than most people expect, which is why cascades are usually worth trying. Two costs are not in that arithmetic and must be added: the latency of the failed cheap attempt is paid on escalated requests, and the escalation check itself costs something if it is a model call. A deterministic check — schema validation, a confidence threshold, a regular expression — keeps the arithmetic honest. Model cascading has the full treatment.

Lever 4: stop paying twice for retries

This is the only lever on the page that is pure waste rather than a trade-off, and it is the one most often missing from cost work because it is invisible in a per-request cost figure.

  waste = N * (r - 1) * cost_per_attempt

WORKED: N 200,000, r 1.18, cost_per_attempt $0.0031
  waste = 200,000 * 0.18 * 0.0031 = $112/month, buying nothing

Where r above 1.0 comes from:
  - retrying errors that are deterministic (a bad schema will fail
    identically on every attempt)
  - retrying a provider whose account is out of credit, which some
    providers signal with a status code that looks transient
  - a client library default of several retries stacked on top of
    your own retry loop, multiplying rather than adding
  - failover to a dearer route triggered by a fault that a cheaper
    route would also have had

Two fixes, both cheap. Classify errors before retrying, so that permanently-failing requests are not retried at all — see production LLM error classes for the classifier. And use an idempotency key so a retry that races a slow success cannot buy the same completion twice; safe retries covers the shape.

Lever 5: batch what is not interactive

Providers commonly offer a substantially discounted asynchronous tier with a long completion window. Any workload where no human is waiting qualifies: back-fills, nightly classification, enrichment, evaluation runs.

  saving = N_async * cost_per_request * discount

The question is not the discount, it is N_async. Audit by asking
of each call site: is a person waiting for this specific response?

  user-facing chat            -> no
  search result summarisation -> no
  nightly re-classification   -> YES
  ingest-time enrichment      -> YES, if ingest is not user-visible
  evaluation suite runs       -> YES, and this is often large

The evaluation suite is the one teams forget. A regression set run nightly is pure batch workload and frequently a non-trivial share of total spend. The batch discount explains the trade-offs.

Lever 6: delete the calls nobody reads

The largest single saving available in many systems is not an optimisation at all. Instrument by call site — not by model, not by endpoint — and then ask what consumes each result.

• Reasoning nobody displays. A prompt asking the model to explain itself, whose explanation is discarded, is paying output price for tokens that reach no one.
• Pre-emptive generation. Summaries, suggestions and titles generated on page load for content the user may never open. Generate on demand and cache the result.
• Duplicate work across a pipeline. Two stages each extracting the same entities from the same document, because they were built by different people.
• A feature nobody uses. Its inference is usually a small share of its total cost, but the inference is the part you can stop today. The full accounting is in what an unused AI feature actually costs.

Per-call-site attribution is the part that is awkward to build and is the prerequisite for everything on this page. Multigrid records cost per request with your own tags, so “what does this call site cost” is a query rather than a project.

The order to do them in

1. Lever 4 first. It is pure waste, it requires no quality trade-off, and the work is error classification you should have anyway.
2. Then lever 6. Deleting a call is the only change with no ongoing cost and no risk of regression on work that still happens.
3. Then lever 1 if P_fixed / P is high, or lever 2 if output_share is high. The two ratios you computed at the top decide which.
4. Then lever 5, which is configuration and scheduling rather than model work.
5. Lever 3 last, because it is the only one that can change output quality, and it needs an evaluation set to do safely.

Verifying the saving actually landed

Cost work has an unusually high rate of changes that were shipped, believed, and did nothing. Four checks, each of which has caught a non-saving:

1. Compare cost per unit of work, not total spend. Total spend moves with traffic, so a 20% saving during a 25% traffic increase looks like a regression and a change that did nothing looks like a win in a quiet week. Divide by the number of completed tasks.
2. Check the cache hit rate directly, not the bill. A caching change that fails silently — a per-request field slipped into the preamble, an ordering change, a prompt edit that shifted the fixed block — produces exactly the same responses at exactly the old price. If your provider reports cached token counts, that number is the verification; if it does not, compare the input-token price realised against the list price.
3. Watch attempts per served request after every change. Several savings are paid for by a rise in retries. A cheaper model that fails schema validation more often, or a smaller token cap that truncates and triggers a repair pass, can cost more in total while looking cheaper per call.
4. Re-run the evaluation set. Every lever except 4 and 5 can move quality, and a saving that degrades output is a price change, not an optimisation. Record the score alongside the cost so the two are comparable later.

One structural point that makes all four easy: attribute cost at the call site rather than at the model. Model-level totals cannot answer whether a change worked, because a single model serves many call sites with different prompt shapes, and the one you changed is averaged away.

Whether these total 80% on your workload is not something this page can tell you, and any figure it gave would be invented. The arithmetic above takes an afternoon with your own numbers and will give you a real total before you implement anything. Reducing LLM costs has the broader catalogue of techniques.

Related

• What an Unused AI Feature Actually Costs
• Production LLM Error Classes, and How to Categorise Your Own