...But Still, Stop Going Back For Seconds
There's a good video making the rounds about LMCache [https://www.youtube.com/watch?v=TvnOYwIoWoQ], the open-source project that basically turns the KV cache into a tiered, shared memory system across inference workers.
The short version is pretty straightforward.
If your coding agent keeps sending the same 100K tokens back to the model every turn, the GPU shouldn’t have to sit there and recompute the same attention state like it’s never seen the stuff before.
Keep the hot stuff in HBM. Spill the rest into RAM. Spill more into NVMe. Share it between workers. Reuse what you already paid to compute.
At scale, time-to-first-token can improve by ridiculous multiples.
Great engineering. Seriously. But I can already see where this is going, so I’d like to ruin the party before somebody turns it into conventional wisdom.
LMCache is a little like Ozempic for token bloat.
GLP-1 semaglutides may be remarkably effective at dealing with the consequences of eating too much. That does not suddenly make twelve donuts a sensible breakfast.
The number that should bother you
The video references a dataset of 739 anonymized Claude Code sessions.
The median session apparently started at around 20K input tokens and finished around 115K.
By the end of those sessions, the benchmark authors estimated that 96.9 percent of the context was theoretically reusable.
Same repository files.
Same system prompt.
Same tool definitions.
Same conversation history.
Then, way down at the bottom, a tiny sliver of whatever actually changed this turn. If you’re an infrastructure engineer, you look at that and think:
Holy crap, look at the caching opportunity.
And you’re right.
If you’re the guy paying the API bill, though, you might reasonably look at the exact same number and think:
Why in God's name are we dragging 111K tokens of luggage through the airport every time we want to change socks?
LMCache helps the baggage handlers move the suitcases faster.
Excellent.
I’m suggesting we might also stop moving the damn piano.
Caching does not make the bloat disappear
There are three things server-side caching does not magically fix:
1. You still pay for cached tokens.
Prompt caching discounts cache reads. It does not make them free.
If a provider charges roughly a tenth of the normal input rate for cached tokens, then congratulations: your 111K-token backpack full of mostly old crap is now 90 percent cheaper to carry around.
It is still a 111K-token backpack full of mostly old crap.
Do that every turn, all afternoon, and eventually somebody in accounting notices.
2. Your context window is still the same size.
A 200K-token context window holds 200K tokens whether those tokens came screaming fresh out of a GPU or were retrieved from the world's cleverest cache.
Every stale source file sitting in there is space you cannot use for whatever you are actually trying to accomplish now.
Caching makes old tokens cheaper to process. It does not make them smaller.
and 3. The model still has to look at the damned things.
The model is still reasoning over the contents of the context window.
If fifty files got dumped into context four tool calls ago and forty-seven of them are no longer relevant, they do not become intellectually invisible just because the KV state came out of RAM.
They are still competing with the useful stuff.
Caching fixes recomputation. It does not fix clutter.
So yes, when the working set fits and the cache is warm, the infrastructure has done its job beautifully. That does not mean you **did **yours.
Ozempic is not a nutrition plan
This is the distinction I think matters: There is a boundary between the client and the model.
Once a token crosses that boundary, all kinds of smart people can do all kinds of smart things with it:
- Prefill optimization
- KV caching
- Tiered storage
- Worker affinity
- Shared caches
- HBM
- RAM
- NVMe
- and whatever NVIDIA invents next Tuesday.
That is all downstream, and the provider owns that problem.
But before the token crosses the boundary, somebody decided to send it.
That part is upstream, and that part is yours.
LMCache, vLLM prefix caching, SGLang’s hierarchical caching, NVIDIA’s KV work and all the rest are essentially trying to make overeating less expensive.
Retrieval asks why we ordered six pizzas in the first place.
Those approaches are not enemies. Quite the opposite. They compound.
A small cached prefix is fantastic.
A giant cached prefix is better than a giant uncached prefix.
But it is still the second-best answer.
And right now an awful lot of money is being spent making second-best comfortable enough that nobody feels compelled to fix first-best.
The best part of the video has almost nothing to do with LMCache
The part I’d tattoo on a junior developer is the section about cache invalidation:
- Change serialization.
- Move something near the beginning of the prompt.
- Inject some supposedly harmless piece of metadata that changes every request.
- Trim the beginning of a long conversation.
Congratulations.
That beautiful 96.9 percent reusable prefix may suddenly become a full recompute. That rule matters upstream too.
We have the benchmark scars from jCodeMunch to prove it.
We measured what happens when the tool list changes in the middle of a session and invalidates the cached schema block.
The new block gets written at full freight, and it can take hundreds of subsequent requests before the change pays for itself. So we eventually put runtime refusals in both places capable of triggering it.
This week I killed what looked, on paper, like a clever per-request tool-selection scheme for exactly the same reason.
It was clever. It was adaptive. It was also going to rewrite the prefix every damned turn and never recover the cost.
So into the wood chipper it went.
That leaves us with two rules:
- Stable prefix.
- Small prefix.
In that order.
The server-side caching people are correctly screaming about the first one. There just happens to be considerably less financial incentive for anybody to scream about the second.
What "eat less" looks like for an AI agent
The funny part is that none of this is particularly exotic.
It is old-school scarcity engineering wearing an AI hat.
Index once. Retrieve by pointer.
If an agent asks for:
src/auth.py::AuthService.validate_token#method
…and gets forty relevant lines back, it does not need the whole file dumped into context. The stable identifier becomes the compressed reference.
That is basically the premise behind the jMRI specification, and it is why jCodeMunch consistently benchmarks around 95 to 97 percent fewer tokens than naive file reads.
Those are deliberately conservative numbers. The methodology is in the repository.
Expose a small tool surface and leave it alone.
- Six stable tools at the front door.
- Drill down when necessary.
- Do not reshuffle the menu every fifteen seconds because somebody discovered dynamic tool selection and got excited.
- Static profiles may be less clever.
They also do not light your prefix cache on fire.
Let stale context die.
If a source file mattered six turns ago and does not matter now, stop sending it.
This sounds embarrassingly obvious. And yet most agent harnesses are not especially good at it, which is how you end up with median sessions lumbering from 20K tokens to 115K.
We have somehow rebuilt memory leaks, except now they come with an invoice.
None of these techniques require a GPU.
Every one of them reduces how much work the GPU eventually needs to cache.
Take the Ozempic. Also stop eating the donuts.
Server-side KV caching is going to get much better. Fast. And some of those gains will probably eventually show up in API pricing.
Fantastic.
Take the discount. Use the cache. Let the infrastructure people perform their wizardry.
But remember what the discount is being applied to.
A number you largely control.
If you cut your context from 100K tokens to 10K and the provider figures out how to make those 10K cached tokens cheaper too, everybody wins. That is the actual best case.
The cheapest token is still the one you never sent.
LMCache makes bloated context considerably less painful. It does not turn bloated context into a good idea.
Ozempic may help after dinner.
Still, maybe we should avoid the buffet. Howzabout a salad...?
J. Gravelle
j@gravelle.us
Top comments (1)
The Ozempic analogy is unkind but correct. We run a fleet where caching discounts a fat context, and the uncomfortable math is the same: a 0.1x read price still compounds at 100K+ tokens per call across hundreds of calls a day. The cheapest tokens are the ones never sent, and context trimming is discipline work that demos badly compared to an infra milestone.
One addition from our side: a lot of the bloat is not lazy prompts, it is tool schemas — every capability added grows the rendered prefix forever, and nobody owns pruning it. Half of what we cut was "this tool has been unused for a month but is still in every request." Have you seen teams put an explicit budget on the rendered prompt itself, not just the model output?