DEV Community: Andrea Susic

How to estimate your LLM inference bill before you ship (with Python)

Andrea Susic — Thu, 01 Oct 2026 08:26:29 +0000

Most unexpected LLM bills start with a misleadingly cheap prototype.

Someone tests a feature with a handful of short prompts, checks the usage dashboard and concludes that inference costs almost nothing. Then the application reaches production. Prompts get longer, retrieval adds context, conversations stretch across multiple turns and thousands of users start making requests.

A month later, the invoice looks very different.

Fortunately, inference cost is relatively straightforward to model. If your provider charges by token, you can estimate the cost of a realistic request before launch and then model what happens as traffic grows.
The arithmetic is simple. Getting realistic inputs is the important part.

How LLM inference is usually billed

Many hosted LLM APIs charge separately for input and output tokens.
Input tokens include everything sent to the model: the system prompt, retrieved context, conversation history, tool information and the user's current message.
Output tokens are the tokens generated by the model.
Providers commonly quote both prices per million tokens, and output tokens are often more expensive than input tokens.
At its simplest, the calculation is:

cost = (input_tokens × input_price + output_tokens × output_price) / 1,000,000

The formula is rarely the source of forecasting errors.
The problem is estimating input_tokens.
A production request is usually much larger than the text the user sees in the chat box.

Step 1: Count a realistic request

Start with something that resembles the request your application will actually send.
If you are building a retrieval-augmented support assistant, for example, include the system instructions, retrieved documents and user message rather than measuring the user message alone.

Install tiktoken:

pip install tiktoken

Then create a representative request:

import tiktoken

# o200k_base is a modern BPE tokenizer.
# Different model families can use different tokenizers, so use the
# target model's tokenizer when you need model-specific counts.
enc = tiktoken.get_encoding("o200k_base")

def count(text: str) -> int:
    return len(enc.encode(text))

SYSTEM_PROMPT = """You are a support assistant for an online store.
Answer only from the provided context. If the answer is not in the
context, say you don't know and offer to connect a human agent.
Keep answers under 120 words and use a friendly, plain tone."""

RETRIEVED_CONTEXT = """Returns: items can be returned within 30 days of delivery.
Refunds go back to the original payment method within 5 business days.
Shipping: standard delivery takes 2-4 business days. Express is next day if ordered before 14:00. Damaged items: send a photo to support and we replace the item at no cost."""

USER_MESSAGE = """Hi, my blender arrived with a cracked jug.
Can I get a new one, and how long will it take?"""

SAMPLE_ANSWER = """Sorry to hear the jug arrived cracked. Please reply with a photo
of the damage and your order number, and we'll send a replacement at no cost. Replacements ship with standard delivery, which takes 2-4 business days. If you'd prefer a refund instead, that's possible too, and it reaches your original payment method within 5 business days."""

input_tokens = (
    count(SYSTEM_PROMPT)
    + count(RETRIEVED_CONTEXT)
    + count(USER_MESSAGE)
)

output_tokens = count(SAMPLE_ANSWER)

This is still an estimate. Chat APIs can add formatting and message-structure overhead, and different model families use different tokenizers.
For an open-weight model, use the tokenizer associated with the model when possible. With models distributed through Hugging Face, that might look like this:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("<model-repo-id>")

def count_model_tokens(text: str) -> int:
    return len(tok.encode(text))

The closer your test request is to production traffic, the more useful the estimate becomes.

Step 2: Turn tokens into money

Once you have representative token counts, pricing different models is straightforward.

The rates below are deliberately illustrative. Replace them with the current input and output prices from the providers you are evaluating.

# Illustrative prices per 1M tokens.
# Replace these with your provider's current rates.
PRICES = {
    "small model": {"input": 0.20, "output": 0.80},
    "mid model":   {"input": 1.00, "output": 4.00},
    "large model": {"input": 3.00, "output": 15.00},
}

REQUESTS_PER_DAY = 2_000
DAYS_PER_MONTH = 30

# Allow for retries, longer-than-average responses,
# prompt changes and other production variance.
SAFETY_MARGIN = 1.3

def cost_per_request(price):
    return (
        input_tokens * price["input"]
        + output_tokens * price["output"]
    ) / 1_000_000

print(f"Input tokens per request:  {input_tokens}")
print(f"Output tokens per request: {output_tokens}\n")
print(f"{'model':<12} {'per request':>12} {'per month':>12}")

for name, price in PRICES.items():
    per_req = cost_per_request(price)
    monthly = (
        per_req
        * REQUESTS_PER_DAY
        * DAYS_PER_MONTH
        * SAFETY_MARGIN
    )
    print(f"{name:<12} {per_req:>12.6f} {monthly:>12.2f}")

Using the example token counts, you might get something like:

Input tokens per request:  149
Output tokens per request: 74

model         per request    per month
small model      0.000089         6.94
mid model        0.000445        34.71
large model      0.001557       121.45

The exact figures are not important because both tokenisation and provider pricing will vary.
The comparison is what matters.

The same application and traffic assumptions can produce substantially different monthly costs depending on model choice. At the same time, the calculation may reveal that inference is a relatively small part of the application's total operating cost.
That is useful information too. If the projected token bill is €30 per month, spending weeks engineering around it probably makes little sense.

Step 3: Model the multipliers your prototype missed

The simple calculation assumes every request looks roughly the same.
Production applications rarely behave that way.

Three factors in particular can make real token consumption substantially higher than the first estimate.

1. Conversation history grows with every turn

Many chat implementations send previous messages back to the model so that it has the context required to answer the next message.
That means later requests can contain much more input than earlier ones.
Consider this simplified example:

def conversation_input_tokens(
    system_tokens,
    user_tokens,
    answer_tokens,
    turns
):
    """
    Estimate total input tokens billed across a conversation
    when previous messages are included on each new turn.
    """
    total = 0
    history = system_tokens

    for _ in range(turns):
        history += user_tokens
        total += history
        history += answer_tokens

    return total

for turns in (1, 3, 5, 10):
    print(
        turns,
        conversation_input_tokens(
            400,
            40,
            80,
            turns
        )
    )

Output:

Ten turns therefore do not necessarily cost ten times as much input as the first turn. In this simplified example, total input consumption across the conversation is more than 22 times the first request's input.
Why?

Because previous messages are repeatedly included as context.
Real implementations vary. Some APIs and providers offer prompt caching, and applications can summarise, truncate or selectively retrieve previous conversation history rather than continually resending everything.

The important lesson is to estimate chat costs per conversation, not simply by multiplying the first message by the expected number of turns.

2. Language can materially change token counts

Do not assume that a 100-word prompt has the same token count in every language.
Tokenisation efficiency varies by language, writing system, vocabulary and tokenizer. The same underlying meaning can therefore require noticeably different numbers of tokens depending on both the language being used and the model's tokenizer.
That matters commercially.

If you benchmark an application using English prompts but most production users communicate in Serbian, Polish, German or another language, the English benchmark may not accurately represent production token consumption.
The safest approach is not to apply a generic "non-English multiplier."

Build a representative sample of actual user prompts in the languages your application supports and run each sample through the tokenizer of the models you are considering.
For example:

samples = {
    "english": "My order arrived damaged. How can I replace it?",
    "serbian": "Moja porudžbina je stigla oštećena. Kako mogu da je zamenim?",
}

for language, text in samples.items():
    print(language, count(text))

For a multilingual product, repeat this with dozens or hundreds of representative prompts rather than relying on a single sentence.
The resulting numbers are much more useful than assuming one language will always cost a fixed percentage more than another.

3. Context tends to grow after launch

Prompt growth is the easiest of the three to overlook.
The first system prompt contains five rules. Then a production problem appears and someone adds three more.
Retrieval initially returns three document chunks. Later it returns five because answer quality improves.

A few examples are added to handle a difficult edge case. Tool descriptions become longer. More metadata is inserted into every request.

Each change looks small in isolation.
But repeated input is multiplied by every request the application makes.

Suppose you add 500 tokens of permanent instructions to an application handling 100,000 requests per month. That change alone creates:

500 × 100,000 = 50,000,000 additional input tokens per month

At scale, prompt design is therefore also infrastructure design.
This is one reason the example calculator includes a safety margin. More importantly, token measurement should become part of the development process rather than a calculation performed once before launch.

Step 4: Turn the estimate into an engineering decision

Once you know the approximate cost per request and per month, the numbers can guide architecture choices.

Start with the smallest model that meets your quality requirements. Run the same evaluation set against several model tiers. There is little reason to pay for a larger model on every request if a smaller one reliably handles the workload.

Route requests by complexity. Straightforward classification, extraction or support questions may work well on a smaller model, while difficult requests can be escalated to a more capable one.
Reduce unnecessary input. System prompts, retrieval results and conversation history are all recurring costs. Removing irrelevant context can improve both economics and, in some cases, model performance.

Test caching where the workload allows it. Repeated prefixes, instructions and context may qualify for discounted or cached processing depending on the provider.

Measure actual production distributions. An average request is useful for planning, but averages hide expensive outliers. Track median and high-percentile token consumption as well as the mean.

Compare metered inference with dedicated infrastructure only when demand is sufficiently predictable. Usage-based APIs are attractive when traffic is uncertain or variable. Dedicated GPU capacity becomes easier to evaluate once you know how much compute the application consistently consumes.

Build a small pricing table

You do not need a sophisticated cost platform to make the initial decision.
A simple table with model name, input price, output price, average input tokens, average output tokens and expected request volume is enough to compare several scenarios.
Populate the PRICES dictionary with current provider rates rather than estimates. Providers serving open-weight models commonly publish separate rates for input and output tokens; for example, current per-token prices for open-weight models can be used to populate the same calculator for models such as DeepSeek, GLM and Kimi.

Then run several scenarios rather than one:

TRAFFIC_SCENARIOS = {
    "pilot": 200,
    "expected": 2_000,
    "high growth": 10_000,
}

for scenario, requests_per_day in TRAFFIC_SCENARIOS.items():
    print(f"\n{scenario.upper()}")

    for name, price in PRICES.items():
        per_req = cost_per_request(price)
        monthly = (
            per_req
            * requests_per_day
            * DAYS_PER_MONTH
            * SAFETY_MARGIN
        )
        print(f"{name:<12} {monthly:>10.2f}")

That gives you something more useful than a single forecast: a range showing what happens if adoption is much lower or much higher than expected.
Wrapping up
Estimating an LLM inference bill does not require sophisticated financial modelling.

You need realistic traffic, realistic prompts and current token prices.
The basic process is:

Build representative production requests, including system instructions, retrieved context and conversation history.
Count input and output tokens using the tokenizer appropriate for the model.
Apply the current input and output prices for each model you are considering.
Model conversations, supported languages, retries and likely prompt growth.
Run multiple traffic scenarios instead of relying on a single forecast.
Measure actual token consumption after launch and update the model with production data.

The biggest forecasting mistake is usually not getting the price of a token wrong.

It is counting only the tokens that are obvious.

A user's ten-word question may arrive at the model wrapped in thousands of tokens of instructions, retrieved documents, tool definitions and conversation history. Once you measure the complete request rather than the visible message, LLM inference costs become much easier to predict, and much harder to be surprised by.

I work with AI infrastructure providers, including the one linked above.

The hidden cost of cloud GPU training: egress, idle time, and lock-in

Andrea Susic — Thu, 28 May 2026 20:20:53 +0000

The GPU hourly rate is the number everyone compares. It is also the number that tells you the least about what a training run actually costs.

The sticker price, say $2 to $3.50 an hour for an H100 on a specialized cloud, is the visible tip. The real bill is built from three things almost nobody puts on the comparison spreadsheet: the GPU sitting idle, the data you have to move, and the cost of ever leaving. This post breaks down each one, with 2026 numbers, and what you can actually do about it.

1. Idle time: paying full price for nothing

The most expensive line item in most setups is not compute. It is compute you pay for but never use.

The 5 percent problem

A 2026 Cast AI report found average GPU utilization across Kubernetes clusters on major clouds sits around 5 percent. Other analyses are kinder, Anyscale puts sustained production utilization below 50 percent, FinOps studies land at 20 to 30 percent, but the conclusion holds: most of every GPU-hour you pay for produces no useful work.

Why it happens

It is structural, not laziness. Workloads bounce between CPU preprocessing, GPU training, and CPU postprocessing. Python dataloaders on the GPU node starve the accelerator. Teams overprovision to dodge out-of-memory errors and default to the biggest instance "just in case."

Why it hurts more than CPU waste

An idle CPU costs cents per hour. An idle GPU costs dollars per hour. A single AWS p4d.24xlarge left idle over one weekend burns about $1,573 for nothing. A month of overnight and weekend idling typically wastes $3,000 to $8,000 per instance.

What to do

Add idle detection. A script watching nvidia-smi that scales down an instance after utilization stays below ~5 percent for 30 minutes is the highest-ROI thing most teams can ship. Commonly cuts 20 to 35 percent off GPU spend.
Right-size the hardware. Not every job needs an H100. Running on the biggest card when a smaller one delivers the same result is pure burn.
Fix the pipeline first. If dataloaders are starving the GPU, a bigger GPU does not help. Profile the input pipeline before upgrading hardware.

2. Egress: the cost of moving your own data

Uploading data is free everywhere. Moving it out is not, and the asymmetry is deliberate.

The 2026 rates

Verified across multiple pricing surveys: AWS charges about $0.09 per GB outbound, roughly $90 per TB. Google Cloud is higher at $0.12 per GB. Azure sits in the same range. Hetzner includes large free allowances and charges on the order of $1 per TB beyond them. Some object-storage options are zero egress entirely.

Why training amplifies it

The volumes are large and recurring. Datasets, checkpoints, and exported weights all move. On a workload pulling 10 TB out per month, the gap between a hyperscaler and a zero-egress provider is the difference between a four-figure line item and almost nothing.

What to do

Co-locate compute and storage. Keep training data in the same region and provider as the GPUs. Intra-zone transfer is usually free; internet egress is not.
Compress before transfer. gzip or zstd cuts checkpoint and dataset volume 30 to 60 percent, and egress is billed per byte.
Price egress into provider comparisons. A cheap GPU hour on a provider with expensive egress can lose to a pricier hour on one with none.

3. Lock-in: the bill you pay to leave

The third cost is the one you do not see until you try to escape it. It is the same mechanism as egress, viewed over a longer horizon.

Data gravity is the anchor

Once terabytes of data and checkpoints accumulate in one region, moving them is slow and expensive. The egress fee is not just a per-transfer cost, it is an exit tax that grows with every gigabyte you store. By design, the more your data piles up, the less likely you are to leave.

It is not only data

Proprietary services, custom tooling, and provider-specific orchestration all raise the cost of moving. But for training workloads, raw data gravity is the heaviest anchor.

What to do

Prefer portable formats and open tooling. Standard container images, open checkpoint formats, and provider-agnostic orchestration keep your options open.
Model the exit cost up front. Calculate what it would cost to move everything out at a year's expected data volume. If the number is alarming, factor it in now, not later.
Favor zero or low egress for data-heavy work. When leaving is cheap, lock-in mostly evaporates, and you keep leverage over your own infrastructure.

Putting it together

The headline GPU rate is the smallest part of the story. A realistic cost model looks more like this:

(GPU rate x hours x utilization gap) + storage + egress + the eventual cost of leaving

Two things follow. First, fixing utilization is usually the fastest win, because you are already paying for that waste today. Second, egress and lock-in are decisions you make once, at the start, that compound for as long as the project runs.

This is why the provider landscape is shifting. Specialized GPU clouds and regional providers increasingly compete on exactly these hidden costs: transparent hourly billing instead of a maze of ancillary fees, and zero egress so your data, and your freedom to move it, stays yours. Orion AI Factory in Europe is one example of that model, and the same logic shows up across a growing set of regional and specialized providers. The common thread is pricing the things that used to hide in the footnotes.

None of this needs exotic tooling. Watch your utilization, keep data close to compute, compress what you move, and know your exit cost before you are locked in. The teams that win on cost are not the ones with the biggest budgets. They are the ones who read past the hourly rate.

References

Cast AI, GPU utilization report, 2026
Anyscale, production GPU utilization analysis, January 2026
GPUPerHour, data egress pricing across 44+ providers (https://gpuperhour.com/reference/data-egress), April 2026
LeanOps, AI cloud cost optimization guide, 2026