How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops

# How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops *Published 26 September 2026 · Prices are Anthropic's list rates as of 26 September 2026. The method behind every cost figure is at the end of the article.* A long conversation costs far more than its number of turns suggests, because a language model holds nothing between calls: every request carries the system prompt and the entire history again, so total input grows with the square of the number of turns. The fixes, in the order to apply them, are prompt caching, which cut a 40-turn chat on Claude Sonnet 5 from $1.09 to $0.29 in our worked example, then shrinking what each turn adds, then trimming or compacting old turns, and finally bounding the context with a memory layer once conversations span sessions. Caching loses its grip when users return after the cache has expired, and in our arithmetic a bounded context becomes cheaper than cached full history at around 200 turns and 100,000 tokens of history on Sonnet 5, even after paying for the memory layer itself. The rest of this piece is the arithmetic behind that ordering, including the cases where the cheaper-sounding option is the more expensive one. ## Why a long conversation costs more than its turns An LLM API is stateless. [Anthropic's context window documentation](https://platform.claude.com/docs/en/build-with-claude/context-windows) describes each turn's input as "all previous conversation history plus the current user message", with previous turns "preserved completely". If a turn adds S tokens and the system prompt is P tokens, the request at turn k carries roughly P + (k − 1)·S tokens, and a conversation of N turns sends N·P + S·N(N − 1)/2 input tokens in total. The second term is quadratic, and past a handful of turns it is most of the bill. A concrete conversation makes it tangible: a 2,000-token system prompt, 150-token user messages and 350-token replies. Turn 1 sends 2,150 tokens. Turn 40 sends 21,650. The 40-turn conversation consumes 476,000 input tokens to produce 14,000 output tokens, and on [Claude Sonnet 5 at $2 input and $10 output per million tokens](https://platform.claude.com/docs/en/about-claude/pricing) it costs $1.09, of which 87% is input. Going from 40 to 100 turns multiplies the turns by 2.5 and the bill by 5.2, to $5.73. Two things make the curve steeper on modern models. Thinking tokens are billed as output, and on Claude Opus 4.5 and later, Sonnet 4.6 and later and the Fable and Mythos models, earlier thinking blocks stay in the conversation and count as input on every later turn. In agent loops, tool results do the same: a search result or file read added at turn 5 is paid for again on every turn after it. ## Lever 1: prompt caching, and what it saves Prompt caching stores a prompt prefix so that later requests read it instead of paying full price for it. On Claude, [a 5-minute cache write costs 1.25 times the base input rate and a cache read costs 0.1 times](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), with Claude Opus 5.5 reading at 0.05 times and Claude Fable 5.1 at 0.025 times. Automatic caching (a top-level cache\_control\ setting) places the breakpoint on the last cacheable block and moves it forward as the conversation grows, so each turn writes only the new exchange and reads everything before it. For the 40-turn conversation above, that takes the Sonnet 5 cost from $1.09 to $0.29 and the input share of the bill from 87% to about half. It needs no change to what the model sees, which is why it comes first. Cache hits also do not count against your rate limit. The 5-minute cache suits rapid turns and agent loops. The 1-hour cache costs 2 times the input rate to write and pays back from the second reuse, so it suits chat where users pause for more than five minutes but return within the hour. ## Where caching stops working Caching discounts the history; it does not shrink it. Four things limit how far it goes. - **Users leave and come back.** When a user returns after the cache has expired, the first request of the new session writes the entire history again at 1.25 times the input rate. For an assistant used in short sessions over days or weeks, that rewrite happens every session and grows with the history. - **Changes invalidate it.** The cache follows the order tools, then system, then messages. Changing a tool definition invalidates everything; adding an image or changing tool\_choice\ invalidates the message cache. A cache that misses returns no error, so the only place it shows up is cache\_read\_input\_tokens\ in the usage object. - **The window still fills.** Current Claude Opus, Sonnet and Fable models take 1 million tokens, and a cached history counts toward the window at full size. - **Quality does not improve with volume.** Anthropic's own [compaction documentation](https://platform.claude.com/docs/en/build-with-claude/compaction) justifies the feature by noting that "response quality degrades as a conversation grows". A cheaper token is still a token the model has to read past. ## Lever 2: shrink what each turn adds Before touching the history, cut what each turn contributes. **Clear old tool results.** Claude's [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing) (beta header context-management-2025-06-27\) clears tool results once the input passes a threshold (100,000 tokens by default) and keeps the three most recent tool uses, replacing the rest with a placeholder. It can also clear old thinking blocks. Clearing breaks the prompt cache at the point it clears, so the clear\_at\_least\ setting exists to make sure each clearing removes enough tokens to be worth a fresh cache write. **Cap output and thinking.** Output costs five times input on current Claude models, and thinking tokens bill as output. A max\_tokens\ limit and a lower effort setting on turns that do not need deep reasoning reduce both what you pay now and what later turns carry. **Route by task.** A summarisation or classification step inside a conversation rarely needs the model that holds the conversation. Claude Haiku 4.5 costs half of Sonnet 5 per token, and anything that can wait belongs on the Batch API at half price. ## Lever 3: trim or compact the history **A sliding window** keeps the last K exchanges and drops the rest. It is free and caps per-turn input, and it forgets everything outside the window, including the user's name if they gave it at turn 2. Keep a pinned block for facts that must survive (identity, standing constraints, open tasks), or the window will drop them. **Compaction** replaces older turns with a summary. Claude now does this server-side: [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) can run on demand or automatically when input tokens reach a threshold you set, with no summarisation code of your own. A summary costs a model call to produce and pays for itself quickly once the history is long, because every later turn reads the summary instead of the transcript. What compaction risks is detail. A summary that turns "takes 20mg of Atorvastatin daily" into "takes a cholesterol medication" has saved tokens and lost the fact that mattered. Whatever compacts your history should tell you what it kept. Maximem Synap's compaction, which we build, returns a validation score and a count of preserved facts on every pass, runs when a conversation passes 3,000 tokens, 10 messages or 5 minutes idle, targets 1,500 tokens, and keeps the last three exchanges verbatim. ## Lever 4: bound the context with a memory layer A memory layer changes the unit that is resent. Instead of replaying the transcript, it extracts what is worth keeping (facts, preferences, decisions, open tasks) as the conversation happens, stores them outside the prompt, and returns a ranked block of the relevant ones within a token budget on each turn. Per-turn input stops growing with the history: system prompt, a memory block (2,000 tokens by default in Maximem Synap), the last few exchanges verbatim and the new message. That is not free, and the comparison is only honest if the memory layer's own cost is counted. The table below does that. It uses the same conversation as above, split into sessions of 10 turns with the user returning after the cache has expired, on Sonnet 5. The bounded column includes Maximem Synap's published usage cost for storing each exchange and fetching context every turn (3 credits a turn at $0.00175 a credit). | Conversation length | No caching | Cached, cache cold at each session | Cached, cache never expires | Bounded context, memory layer included | |---|---|---|---|---| | 10 turns, 1 session | $0.12 | $0.06 | $0.06 | $0.16 | | 50 turns, 5 sessions | $1.61 | $0.51 | $0.38 | $0.84 | | 100 turns, 10 sessions | $5.73 | $1.56 | $1.01 | $1.68 | | 200 turns, 20 sessions | $21.46 | $5.27 | $3.01 | $3.37 | | 500 turns, 50 sessions | $128.65 | $29.29 | $15.02 | $8.45 | Read it in both directions. For conversations under about 100 turns, caching alone is cheaper, and a memory layer is not a cost decision there; if you adopt one at that length, adopt it for what it remembers across sessions, not for the bill. Around 200 turns, where the history passes 100,000 tokens, the bounded context overtakes cached history whose cache goes cold between sessions. At 500 turns it costs 29% of the cold-cache figure and 56% of a cache that never expires, and the gap keeps widening because one column grows with the square of the turns and the other grows linearly. The crossover moves with the model. Claude Opus 5.5 reads cache at 0.05 times its input rate, which makes cached history unusually cheap: on Opus 5.5 the bounded context beats the cold-cache case from 200 turns ($5.62 against $8.67) but only beats a cache that never expires at 500 turns ($14.09 against $17.40). An independent cost study, [arXiv 2603.04814](https://arxiv.org/abs/2603.04814), reached a consistent shape on different assumptions: at a 100,000-token context a fact-based memory system became cheaper than long-context prompting after about ten turns, with the break-even arriving sooner as context grows. The same study, with GPT-5-mini answering and a three-vote GPT-5-mini judge, found long-context prompting recalled more facts on two of its three benchmarks, which is the quality trade to test on your own conversations before you move a workload. ## Which technique for which workload | Workload | Start with | Add when | Watch for | |---|---|---|---| | Short chats, one sitting | Prompt caching | Rarely needs more | Silent cache misses | | Agent loops with heavy tool use | Caching plus tool-result clearing | Compaction past the window threshold | Clearing breaking the cache too often | | Long single sessions (research, coding) | Caching, then server-side compaction | A pinned block for facts the summary must keep | Detail lost in summaries | | Assistants people return to over days or weeks | Caching within sessions | A bounded context once histories pass about 100,000 tokens | Retrieval missing a fact the transcript had | | Support or sales agents serving many users | Caching plus a bounded context | Organisation context held separately from user memory | Scoping between users and tenants | ## How to measure it Measure per conversation, not per request. For every turn, log input\_tokens\, cache\_creation\_input\_tokens\, cache\_read\_input\_tokens\ and output\_tokens\ from the response, keyed by conversation and session. From that you get the three numbers that matter: cost per conversation by length bucket, cache read share per turn (a sudden drop means something is invalidating the cache), and the input tokens carried at the median and the 95th percentile turn. The distribution matters more than the average, because the longest conversations are where the quadratic term lives. Use the [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) to size prompts before launch. Set a context budget explicitly: the number of tokens a turn is allowed to carry beyond the system prompt. Without one, the budget is whatever the history happens to be, and it only ever grows. ## What this means for pricing your own product If your product charges a flat fee per user, your heaviest users cost you the most, and on unmanaged context they cost you disproportionately more, since a user with five times the conversation length can cost up to roughly twenty-five times as much to serve. Either bound the context so cost per turn stays flat, or price in a way that acknowledges length, through usage tiers or credits. Doing neither means your best-retained customers are your least profitable ones. ## Frequently asked questions **How do I reduce LLM token costs in long conversations?** Turn on prompt caching first; on Claude Sonnet 5 it cut a 40-turn conversation from $1.09 to $0.29 in our example. Then cap output and clear old tool results, compact or trim old turns once a single session runs long, and bound the context with a memory layer for assistants that users return to across sessions, where caching stops helping once the cache expires between visits. **Why do chatbot conversations get more expensive as they go on?** Because the API is stateless, each request resends the system prompt and every previous turn, so input per turn grows with the conversation and total input grows with the square of the number of turns. A 40-turn chat with a 2,000-token system prompt and 500 tokens per exchange sends 476,000 input tokens. **Does prompt caching solve the cost of long conversations?** Within a sitting, mostly. It bills the repeated history at a tenth of the input rate on most Claude models. It does not shrink the history, it has to rewrite the whole history when a user returns after the cache expires, and it does nothing for the context window limit or for answer quality on very long inputs. **Is summarising the conversation cheaper than sending the full history?** Once the history is long, yes, because every later turn reads the summary instead of the transcript. The risk is losing specific details, so check what each summary kept. **When is a memory layer cheaper than prompt caching?** In our arithmetic on Claude Sonnet 5, at around 200 turns spread over sessions, when the history passes roughly 100,000 tokens, including the memory layer's own cost. Below about 100 turns, caching alone is cheaper. On Claude Opus 5.5, whose cache reads are cheaper, the crossover comes later. **Does reducing context make responses faster?** Every input token has to be processed before the first output token, so a smaller prompt starts answering sooner. Caching reduces cost more than it reduces the amount the model reads, while trimming, compaction and a bounded context reduce both. ## Method Every cost figure comes from one synthetic conversation: a 2,000-token system prompt, 150-token user messages and 350-token replies, priced at Anthropic's list rates on 26 September 2026 (Claude Sonnet 5: $2 input, $10 output, $2.50 for a 5-minute cache write and $0.20 for a cache read per million tokens; Claude Opus 5.5: $4, $20, $5 and $0.20). "Cache warm" and "cache never expires" assume automatic caching where each turn writes the new exchange and reads the prefix. "Cache cold at each session" assumes sessions of 10 turns with the cache expired at the start of each, so the first request of a session writes the full prefix at the write rate. The bounded context sends the system prompt (cached within a session), a 2,000-token memory block, the last three exchanges and the new message at the full input rate, plus Maximem Synap's published usage cost of 3 credits a turn (2 credits to store a 500-token exchange and 1 for a fast context fetch) at the list rate of $0.00175 a credit; plan credits cost less than that. Output cost is included everywhere. Real conversations vary in length and shape, and whether a 2,000-token memory block holds enough for a given product is a quality question this arithmetic does not answer. --- *Sources, retrieved 26 September 2026: [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing); Claude Platform documentation on [context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows), [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction), [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing), [extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) and [token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting); Pollertlam and Kornsuwannawit, [Beyond the Context Window](https://arxiv.org/abs/2603.04814), March 2026. Maximem Synap figures come from its published documentation and pricing. For every Claude rate in one place, see [Claude API pricing in 2026](https://www.maximem.ai/blog/claude-api-pricing-2026).\*