The first LLM bill that surprises you is almost never the one for a big job. It's the one for a chat feature that "barely gets used."
The reason is a single line most of us write on autopilot: messages.append(...). It looks like you're adding one message. You're actually re-buying every message that came before it.
Two words first, since the whole argument lives in them. A token is roughly ¾ of a word; models bill per million of them, at two different prices — input tokens (everything you send) and output tokens (what the model writes back), with output usually 5–6× the price of input. And the API is stateless: it remembers nothing between calls. The "memory" in your chat bot is you, resending the entire transcript on every single turn.
The cost grows with the square of the turn count
Say a system prompt of 500 tokens, user messages of 200, and replies of 400. On turn k, what you send is the system prompt, plus everything already said, plus the new question:
input(k) = 500 + 600·(k − 1) + 200
That 600·(k − 1) is the killer. It grows every turn, and you pay it again every turn. Sum it across a conversation of n turns and the total input is roughly 300·n² — quadratic, not linear.
| Turns | Input tokens billed | Output tokens billed |
|---|---|---|
| 10 | 34,000 | 4,000 |
| 30 | 282,000 | 12,000 |
| 60 | 1,104,000 | 24,000 |
Doubling the conversation from 30 turns to 60 didn't double the cost. It nearly quadrupled it. And the user in that 30-turn conversation typed 6,000 tokens of questions — you were billed for 282,000, a factor of 47.
What that costs in actual dollars
Numbers as of 2026-09-03, on claude-sonnet-5 at Anthropic's list price of $2.00 per 1M input and $10.00 per 1M output.
One 30-turn conversation: 282,000 input = $0.564, 12,000 output = $0.12. $0.68 a conversation — and 82% of it is history you already paid for.
Ten thousand of those a month is $6,840. That's the bill people describe as "coming out of nowhere," because the intuitive estimate — count the questions and answers, 18,000 tokens a conversation — lands at about $0.09. Off by 7×.
Note which side the money is on. Everyone tunes max_tokens and trims the model's replies; output is $0.12 of that $0.68. The bill is in the input, and the input is your own transcript.
Cap the history and the curve goes flat
The fix is a sliding window: keep the system prompt, keep the last N exchanges, drop the rest. Once the window is full, input(k) stops growing, and quadratic becomes linear.
SYSTEM = {"role": "system", "content": "..."}
def window(history, keep_turns=8):
"""System prompt + the last N user/assistant pairs."""
return [SYSTEM] + history[-keep_turns * 2:]
# Before sending, see what you're actually paying for:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
sent = window(history)
print(sum(len(enc.encode(m["content"])) for m in sent), "input tokens this turn")
On the same 30-turn conversation, an 8-turn window bills 143,400 input tokens instead of 282,000 — 49% off, from four lines of code. Print that number on every request for one afternoon and you will know whether you have this problem, which is more than most teams can say.
If the tail of the conversation genuinely matters, summarize it instead of dropping it: fold turns 1–20 into a 150-token summary with a cheap model and prepend that. Same shape, less amnesia.
What this doesn't fix
A window shrinks the token count. It does nothing to the other multiplier — the price per token — and your bill is the product of both. That's the case for a gateway: altrouter.ai resells the same models 10–25% below the vendors' own list prices (claude-sonnet-5 at $1.69 per 1M input against Anthropic's $2.00), over the OpenAI-compatible API, so it's a base_url change. The 30-turn conversation above comes to $0.58 instead of $0.68. Its honest gap on this exact topic: our usage log stores prompt tokens per request, with no conversation id, so it can't point at the conversation that went quadratic. That grouping has to happen in your app.
Prompt caching is the other half of the answer, and it works well here — the growing prefix is stable — but it has its own break-even point and is not automatic.
The one number to take away
Input tokens in a chat scale as n², and the user's typing is a rounding error inside them. Before you optimize anything else, log the input-token count of every request and sort descending. The top of that list is your bill.
Top comments (0)