Your API dashboard shows a spike, but the feature "felt small." Look at the tokens: what the model reads, writes, and re-sends on every step. Tokenization is not academic. Context limits, per-token pricing, and latency all use the same unit, and dense code or odd formatting can cost more than the word count suggests.
Why does billing use tokens instead of words?
Language models do not read whole words or raw characters. They read tokens, sub-word chunks from a fixed vocabulary. In English, a common word is often one token; rare words and code can split into several, and punctuation and spaces count too. OpenAI's rule of thumb is about four characters per token, or about three-quarters of a word, so 100 tokens is roughly 75 words.
That ratio drifts. Non-English text, code, and weird formatting usually burn more tokens per visible line. Before you argue about model tier, paste representative prompts into a tokenizer and note input tokens per request. Guessing from word count is how teams under-budget chat history and tool output.
What counts toward the context window each turn?
The context window is a token budget for one response, and the latest user message is only one part of it. System instructions, prior turns, tool definitions, and tool results all sit in that budget. Output tokens need room too. In chat, unless your app trims or summarizes, the full history goes back on the next turn, so input tokens climb every step.
Exceeding the limit is not one universal behavior. Depending on the API, the request may be rejected or generation may stop, and many applications truncate or compact the input before sending it. Compaction changes what the model sees next; it does not make the window bigger. Anthropic's documentation also warns that accuracy and recall can degrade as context grows, what it calls context rot. For cost planning, whether a prompt fits in the window and whether it is cheap to keep sending are different questions.
Where does the money go on each inference call?
For most teams on hosted models, inference is the bill that repeats: every message and agent step is another run. Training is someone else's capex; your product pays per call.
Providers charge input and output separately. On Anthropic's price list as of October 2026, output costs five times as much as input for each Claude model. Reasoning models can bill hidden reasoning as output, so a short visible answer can still be expensive. Repeated static instructions may qualify for prompt caching: Anthropic charges about a tenth of normal input to read a cached block, with a higher cost the first time you write it.
One successful task is rarely one call. Agents plan, call tools, retry, and often carry a growing context each time. Cost per task is roughly the sum over every call of (input tokens × input price) plus (output tokens × output price), including failures and retries.
Jeetu Patel, Cisco's president and chief product officer, argued in episode 67 that price per token could turn into a vanity metric: the price per token falls while long-running agents use more tokens. What matters to him is the price of the outcome. Your estimator should still start with tokens, because that is what you control in design.
How do I sketch a monthly token bill?
Use your provider's current input and output rates. If you only remember the Anthropic ratio above, set OUTPUT_PRICE = 5 * INPUT_PRICE as a placeholder until you paste real numbers from the price list.
Label this a sketch, not production accounting:
# Sketch: monthly inference cost from volume.
# Prices are $ per 1M tokens; replace the placeholders with your provider's list.
def monthly_cost(tasks_per_month, calls_per_task, avg_input_tokens,
avg_output_tokens, input_price, output_price):
calls = tasks_per_month * calls_per_task
per_call = (avg_input_tokens * input_price
+ avg_output_tokens * output_price) / 1_000_000
return calls * per_call
INPUT_PRICE = 1.0 # placeholder
OUTPUT_PRICE = 5 * INPUT_PRICE # the ratio on Anthropic's list
# 50k tasks, 8 model calls each, 12k tokens in and 800 out per call
bill = monthly_cost(50_000, 8, 12_000, 800, INPUT_PRICE, OUTPUT_PRICE)
Run three scenarios on the same workflow: today's averages, a heavy month (longer history, more retries), and a trimmed prompt (fewer input tokens or fewer calls per task). The line that moves the total most is where you spend engineering time.
Set calls_per_task realistically for agents. A single user action might be dozens of inferences. Divide total cost by completed tasks, not by API calls, when you compare features.
I think that today a lot of organizations are not thinking about their token spend regularly and they're getting hit with their bills.
Jiaona Zhang, Laurel
Jiaona Zhang ties that to "token maxxing": mandating AI use without asking whether the task earned the tokens, like using a model to redo slide fonts when one click would do. The estimator is how you catch that before finance does.
Which knob should I turn first?
If output tokens dominate (long answers, reasoning traces, verbose tool JSON), tighten schemas, cap completion length, or route easy steps to a smaller model. If input tokens dominate (system prompt, full chat log, fat tool results), trim history, summarize older turns, cache stable instructions, and stop pasting whole documents when a chunk will do.
Jeetu Patel describes rationing tokens like headcount: a fixed pool and an expected output. That matches measuring cost per completed task and watching context on every turn, themes that show up across AI infrastructure conversations on the show.
Checklist
- Tokenize real prompts and tool payloads; do not budget from word count alone.
- Budget context as instructions plus history plus tools plus reserved output tokens.
- Estimate with (input × input price + output × output price) × calls per task × tasks per month.
- Compare scenarios: trim input, trim output, and reduce calls per successful task.
- Report cost per completed task, retries included, not per isolated API call.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)