Building with a large language model (LLM) can look inexpensive at first: send a prompt, receive an answer, pay a fraction of a cent. The surprise arrives when that single call becomes thousands of prompts, long documents, retrieval, retries, monitoring, and users expecting fast answers.
Building reliable, cost-aware applications is a core part of AI Engineering.
The right question is not, “What does one model call cost?” It is, “What does one successful user outcome cost, end to end?” This guide shows how to estimate that number before launch and keep it under control after launch.
Start with the basic LLM cost formula
Most hosted LLM providers charge separately for input and output tokens. A token is a small unit of text; it is not the same as a word. The exact token count varies by language and formatting.
Use this calculation for one request:
Cost per request =
(input tokens / 1,000,000 × input price per million tokens)
- (output tokens / 1,000,000 × output price per million tokens) For example, assume an application sends 2,500 input tokens and generates 500 output tokens. If the model costs $2 per million input tokens and $8 per million output tokens: Input: 2,500 / 1,000,000 × $2 = $0.005 Output: 500 / 1,000,000 × $8 = $0.004 Total model cost per request = $0.009 That looks small. At 100,000 requests per month, it becomes about $900 in model inference alone. This is why usage assumptions matter more than the price of an individual prompt.
Measure every token source, not only the user message
The user’s question is usually only part of the input. A production LLM request may include:
• System instructions and safety rules
• Conversation history
• Retrieved RAG documents
• Tool definitions or function schemas
• Structured output instructions
• Few-shot examples
• The user message itself
If your user writes a 30-token question but your application sends a 7,000-token prompt, your cost is driven by application design—not user behaviour.
Create a token budget for each request type. For a customer-support RAG bot, it could look like this:
Prompt component Typical tokens
System prompt and policies 800
Conversation summary 500
Retrieved knowledge chunks 3,000
Tool instructions 700
User question 100
Expected answer 400
Total 5,500
This table is the foundation of a useful cost forecast.
Forecast monthly cost using user behaviour
Use an activity-based model rather than a vague monthly estimate:
Monthly cost =
active users × requests per user × cost per request
- ingestion cost
- infrastructure cost
- evaluation and operations cost Imagine a knowledge assistant with 5,000 monthly active users. Each user asks 12 questions each month. The end-to-end cost per answer—including model, retrieval, and guardrails—is $0.018.
5,000 × 12 × $0.018 = $1,080 per month
Add $250 for vector storage and search, $400 for application infrastructure, and $270 for scheduled evaluations and monitoring. The expected monthly operating cost becomes $2,000.
Now calculate unit economics:
Cost per active user = $2,000 / 5,000 = $0.40 per month
This is the number product, finance, and sales teams can use to decide pricing, limits, and margins.
Plan for the costs that spike unexpectedly
The average request is not the dangerous request. Costs commonly jump because of long conversation history, large file uploads, repeated retries, tool loops, broad RAG retrieval, or a sudden usage increase.
Build three scenarios:
• Expected: normal user behaviour and target adoption.
• High usage: more requests, longer prompts, and greater retrieval volume.
• Failure mode: retries, fallback-model calls, and runaway agent loops.
For agentic applications, set hard limits on tool calls, maximum turns, maximum output tokens, and total tokens per task. An agent without budgets is a small financial experiment with a keyboard.
Reduce cost without damaging the user experience
Cost optimisation should improve the product, not merely shrink outputs. Start with these high-impact controls:
- Route requests by complexity. Use a smaller, cheaper model for classification, extraction, and simple answers. Reserve the strongest model for reasoning-heavy work.
- Keep prompts lean. Remove duplicate instructions, summarize history, and pass only relevant context.
- Improve retrieval. Retrieve fewer, more relevant chunks; apply metadata filters before semantic search.
- Cache repeatable results. Cache embeddings, common answers, and deterministic tool outputs where appropriate.
Set output limits. Ask for the shortest useful answer and cap maximum completion tokens.
Evaluate before switching models. A cheaper model that increases escalations or wrong answers may cost more overall.
Monitor cost by feature. Track tokens, latency, errors, and quality by customer, workflow, model, and prompt version.
Create a cost dashboard before launch
At minimum, your dashboard should show daily and monthly spend, cost per request, input versus output tokens, cost per active user, cost per successful task, retrieval volume, error/retry rate, and spend by model. Set alerts for sudden increases in prompt size, request volume, or cost per task.
Also attach business context. If an AI support assistant costs $0.03 per resolved case but avoids a $4 human-handled case, the decision is different from a feature that costs $0.03 and creates no measurable value.
Final takeaway
An LLM application is a system, not a single API call. Calculate its cost by measuring model tokens, retrieval, infrastructure, evaluations, and operational overhead. Then translate that total into cost per request, cost per successful task, and cost per active user.
The teams that manage LLM spend well do not guess. They set budgets, measure real usage, test quality, and design the application so that every extra token earns its place.
Top comments (0)