Most LLM features get a cost estimate the same way. Someone multiplies one prompt's token count by the price on the pricing page, gets a number that rounds to zero, and moves on. Then the first monthly bill arrives and it's several times what anyone expected. Nobody did the math wrong. They just did only part of it.
Normal features cost almost nothing to serve one more user. LLM features have a variable unit cost: every request costs money, the amount depends on how much text goes in and out, and you're guessing at usage before launch. So "what does this cost per user?" is a question you'll get from your manager or finance early, and it's worth answering with something better than a napkin.
This is the cost model we use, adapted for engineers. The running example is a feature that summarizes customer support threads for an agent who's picking up the ticket.
About the numbers: every price and usage figure below is a made-up example chosen to keep the arithmetic easy. None of them are any provider's real prices. Pricing changes often, so plug in your provider's current pricing page or your contract rates, and write down the date you checked.
1. The core formula
Most LLM APIs bill per token, with separate prices for input (what you send) and output (what the model generates), usually quoted per million tokens.
cost_per_request = (input_tokens * input_price + output_tokens * output_price) / 1_000_000
cost_per_user = cost_per_request * requests_per_user_per_month
Everything else is about getting those inputs right and adding the multipliers the happy path leaves out.
2. Measure the inputs, don't guess them
Before you open a spreadsheet, get real token counts. Take 20 to 50 scrubbed, representative inputs, run them through the actual prompt template, and log the token usage the API returns for each call. Most provider responses include input and output token counts, and that's the number you're billed on.
"Input tokens" means everything you send: the system prompt, few-shot examples, retrieved context, conversation history and the user's message. The user's message is usually the smallest part. For our ticket summarizer, the illustrative measured numbers look like this:
| Input | Example value |
|---|---|
| Input tokens per request (instructions + thread) | 2,000 |
| Output tokens per request (the summary) | 300 |
| Input price per 1M tokens | $1.00 (example) |
| Cached input price per 1M tokens | $0.25 (example) |
| Output price per 1M tokens | $4.00 (example) |
| Requests per active user per month | 40 |
Look at the median and the p90 of your measurements, not only the mean. Long threads are where the cost goes.
3. Put it in code
A spreadsheet works fine. But if the estimate lives next to the feature, in a small script, it's easy to rerun when the prompt or model changes, and it's reviewed like any other change.
# cost_model.py
from dataclasses import dataclass
# EXAMPLE prices, not any provider's real rates. Replace with your
# provider's current pricing page or contract, and note the date.
PRICES = { # USD per 1M tokens
"input": 1.00,
"cached_input": 0.25,
"output": 4.00,
}
@dataclass
class Feature:
input_tokens: int # per request: system prompt + context + user input
output_tokens: int # per request
requests_per_user: float # per active user per month
cached_share: float = 0.0 # fraction of input tokens served from the prompt cache
retry_rate: float = 0.0 # extra calls from timeouts, schema failures, "regenerate"
overhead: float = 0.0 # evals, judges, fallbacks, as a fraction of production spend
def cost_per_request(f: Feature, p=PRICES) -> float:
cached = f.input_tokens * f.cached_share
fresh = f.input_tokens - cached
usd = fresh * p["input"] + cached * p["cached_input"] + f.output_tokens * p["output"]
return usd / 1_000_000
def cost_per_user(f: Feature) -> float:
multiplier = (1 + f.retry_rate) * (1 + f.overhead)
return cost_per_request(f) * f.requests_per_user * multiplier
if __name__ == "__main__":
naive = Feature(input_tokens=2_000, output_tokens=300, requests_per_user=40)
real = Feature(input_tokens=2_000, output_tokens=300, requests_per_user=40,
cached_share=0.5, retry_rate=0.05, overhead=0.10)
grown = Feature(input_tokens=4_000, output_tokens=300, requests_per_user=40,
cached_share=0.5, retry_rate=0.05, overhead=0.10)
for name, f in [("naive", naive), ("adjusted", real), ("prompt grew 2x", grown)]:
print(f"{name:15} ${cost_per_request(f):.5f}/req ${cost_per_user(f):.3f}/user/mo "
f"${cost_per_user(f) * 10_000:,.0f}/mo @10k users")
Output:
naive $0.00320/req $0.128/user/mo $1,280/mo @10k users
adjusted $0.00245/req $0.113/user/mo $1,132/mo @10k users
prompt grew 2x $0.00370/req $0.171/user/mo $1,709/mo @10k users
Notice where the money goes. In the naive case, output costs four times as much per token, but input is still about 60% of the bill ($0.0020 of $0.0032), because there's so much more of it. That's typical for summarization, RAG and anything that stuffs context into the prompt. It's also why "make the summary shorter" usually saves less than "send less context."
4. The multipliers
Each field with a default of zero in Feature is something the naive estimate leaves out. Here's what each one stands for and how to get a real value.
Caching (cached_share). Many providers discount input tokens that repeat a recently seen prefix, such as a long, stable system prompt. That's the one multiplier that usually works in your favor. In the example, caching half the input more than offsets the retries and overhead. Two caveats. Caching usually only applies to an identical prefix, so put stable content (instructions, examples) first and variable content (the thread) last. And don't assume a cache hit rate. Measure it from the usage fields in the API response once you have traffic, and check your provider's caching rules and prices.
Retries (retry_rate). Timeouts, rate-limit errors your client retries, outputs that fail schema validation, and users clicking "regenerate" all produce billed calls that aren't on the happy path. Get a starting rate from dogfooding and count every call, not every success. If you retry on validation failure, cap the retries. An uncapped loop on a bad input is the classic way to get a surprise bill.
Overhead (overhead). Evals cost tokens too, especially LLM-as-judge. Say a golden set of 300 cases is scored on 4 dimensions with one judge call each, at about 3,000 input and 200 output tokens per call. At the example prices that's roughly $4.56 per full run. Run it on every PR that touches the prompt plus nightly, and it becomes a line item. Fallback routing belongs here as well. If you send hard cases or outage traffic to a larger model, the blended cost depends on the mix, so model each path with the share of traffic you expect it to take.
Prompt growth. Your v1 prompt is the shortest it will ever be. Instructions, examples, guardrail text and retrieved context all accumulate. The third scenario above doubles input tokens and cost per user goes up about 50%, with no change in usage. Always ship the estimate with at least one growth scenario.
5. Model the distribution, not just the average
An average of 40 requests per user can hide very uneven usage. If 90% of users make 20 requests a month and 10% make 220, the average is still 40. But each heavy user costs about $0.70 a month at the naive rate (220 × $0.0032), more than five times the average.
That matters if you charge per seat, or if heavy users cluster in a few big accounts. Decide on per-user and per-tenant rate limits or spend caps before launch, not after the first strange invoice. A quick way to check exposure is to compute cost per tenant from your usage assumptions and sort descending. The top few rows are your risk.
6. Sanity-check the estimate
A model is only useful if it matches reality. Do three checks, in this order.
Before launch, check the arithmetic against a real run. Replay your sample inputs through the real pipeline, retries and all, and sum the token usage the API reports. Compare it to what cost_per_request predicts for the same inputs. If they differ by more than about 20%, one of your inputs is wrong. Usually it's input tokens (a context builder that adds more than you thought) or the retry rate.
# sanity_check.py: compare the model to logged usage from a replay or dogfood week
import json
from cost_model import PRICES
calls = [json.loads(line) for line in open("usage_log.jsonl")] # one line per billed API call
actual = sum(
(c["input_tokens"] - c.get("cached_tokens", 0)) * PRICES["input"]
+ c.get("cached_tokens", 0) * PRICES["cached_input"]
+ c["output_tokens"] * PRICES["output"]
for c in calls
) / 1_000_000
requests = len({c["request_id"] for c in calls}) # user-facing requests, not API calls
print(f"API calls per request: {len(calls) / requests:.2f}") # your real retry multiplier
print(f"Actual cost per request: ${actual / requests:.5f}")
After launch, reconcile with the bill. Divide the provider invoice for the feature's API key or project by the number of user-facing requests you logged. If that's well above your modeled cost per request, something is calling the model that isn't in your model: a background job, an eval run on the production key, or a retry loop. Separate API keys or project tags per feature make this possible, so set them up before launch.
Gut-check against value. Compare cost per user to what the feature earns or saves per user: the seat price, the add-on fee, the agent time saved. If it costs more to serve than it plausibly returns, it's far better to find out in a script than on an invoice.
7. Don't forget latency
Cost and latency trade off, so record them together. A rough estimate:
latency ≈ time_to_first_token + output_tokens / output_tokens_per_second + your_overhead
your_overhead covers retrieval, post-processing and network. Longer outputs cost more and take longer. If you stream, time to first token matters more to how fast it feels than total time. Put a p95 latency budget next to the cost budget, so model comparisons are judged on both.
When you compare models, run each candidate through the same script with the same assumptions, and make the decision on eval scores, not price. A cheaper model that fails your quality bar isn't cheaper. It's a different feature.
8. Ship the estimate as a table, not a number
One cost-per-user figure invites false precision. Share a small scenario table instead:
| Scenario | Users | Requests/user | Input tokens | Monthly cost | Cost/user |
|---|---|---|---|---|---|
| Launch (conservative) | |||||
| Target scale | |||||
| Heavy usage + prompt growth |
Under it, write three sentences: the biggest assumption, how you'll validate it (the replay and the first invoice), and the spend alert thresholds and who owns the budget. Before you share it, check:
- Prices come from the provider's current page or your contract, with the date noted
- Token counts were measured on real, scrubbed inputs
- Caching, retries, eval runs and fallback paths are all included
- Prompt-growth and heavy-user scenarios are in the table
- Per-user and per-tenant limits and spend alerts are defined
- There's a separate API key or project tag for the feature, so you can reconcile with the bill
If you want a head start
We packaged a spreadsheet version of this model into the AI Feature Spec & Eval Kit. It has a cost and latency calculator (Google Sheets and Excel) with scale scenarios, a token-growth sensitivity table, and a side-by-side comparison of up to five models against your budgets. It ships with clearly labeled example prices that you replace with your own, plus the PRD, eval plan and launch checklist templates the estimate usually ends up in: AI Feature Spec & Eval Kit.
Disclosure: we make and sell that kit. You don't need it for anything in this article. The Python above and a usage log will do.
A PM-focused version of this guide first appeared on the AI Spec Kit Substack.
What has surprised you most on an LLM bill? We'd like to hear which multiplier bit you, and which ones turned out not to matter.
This article was written with the help of AI tools, then reviewed and edited before publishing.
Top comments (0)