Okay so here's something that bugged me for a while.
You build an agent. Step 1 feels snappy. By step 15, after a few tool calls and a pile of JSON results, it's noticeably slower, and the bill is climbing in a way that doesn't feel linear.
I knew the vague answer ("the context is longer"). I didn't really get it. So i stopped reading and wrote the thing myself, in plain NumPy, to see where the time actually goes.
Short version: the culprit is a boring little thing called the KV cache, and once you see it, a lot of agent design advice suddenly makes sense.
What happens when a model writes one token
A transformer generates text one token at a time. For every new token, attention needs the Key and Value vectors of every token before it. That's how the new token "looks back".
Here's the catch. Those old Keys and Values don't change. Token 12's Key is the same whether you're generating token 13 or token 913.
So a naive loop does something silly: at every step it recomputes K and V for the whole sequence, from scratch, then throws them away. Then does it again next step.
The KV cache is just... not doing that. You compute K and V for a token once, keep them in memory, and append.
Let me show you, not tell you
Here's the toy model. 4 layers, 8 heads, 512 wide, random weights. It's not smart. It doesn't need to be, we're timing the machinery.
No cache, the full thing re-runs every step:
def forward_full(X):
h = X
t = h.shape[0]
mask = np.triu(np.full((t, t), -1e9, np.float32), 1)
for w in W:
n = rms(h)
q, k, v = (split(n @ w[m]) for m in "qkv")
s = q @ k.transpose(0, 2, 1) / np.sqrt(HD) + mask
h = h + merge(softmax(s) @ v) @ w["o"]
return h[-1:]
With a cache, only the newest token goes through the projections:
def forward_cached(x, cache):
h = x
for w, c in zip(W, cache):
n = rms(h)
c["K"] = np.concatenate([c["K"], split(n @ w["k"])], 1)
c["V"] = np.concatenate([c["V"], split(n @ w["v"])], 1)
q = split(n @ w["q"])
s = q @ c["K"].transpose(0, 2, 1) / np.sqrt(HD)
h = h + merge(softmax(s) @ c["V"]) @ w["o"]
return h
Same math. Before timing anything I checked both versions produce the same output over 40 steps (max difference around 1e-6). Honestly this was the step i almost skipped, and my first version of the benchmark failed it, the "no cache" path had a bug and I was about to publish a fake speedup. Check your baselines.
The numbers
I generated 1,024 tokens each way on my laptop CPU.
Without the cache, the time per token keeps climbing: about 12 ms at step 16, 174 ms at step 512, 550 ms at step 1023. With the cache it stays low, 3 ms early on and about 16 ms at the end.
Total time for all 1,024 tokens: 232 seconds vs 12.5 seconds. Roughly 18x.
Two things worth being honest about.
First, this is a toy on a CPU in NumPy. A real LLM on a GPU has different constants, and nobody should quote my 18x as if it applies to GPT-anything. The shape is what matters: one curve grows with length, the other mostly doesn't.
Second, the cached line isn't flat. It still creeps up (3 ms to 16 ms). That's because the new token still has to attend over everything in the cache. The cache removes the recomputation, not the reading.
Okay but what about agents
This is the part I actually care about.
An agent loop looks like this: model call, tool call, append the result, model call again. Every iteration sends the whole conversation back, system prompt, tool schemas, history, all of it.
If the provider has to process that whole prefix from scratch each time, you pay for it each time, in latency and in input tokens. That's the "slower and pricier with every tool call" feeling.
Prompt caching is the same idea one level up. Providers keep the KV cache for a prefix they've seen, so if your next request starts with the same prefix, they skip that work, and usually bill it at a discount.
Which has some very practical consequences:
- Keep the prefix stable. A timestamp or a request ID at the top of your system prompt changes the prefix every call and kills the cache. Put the volatile stuff at the end.
- Don't reorder or edit earlier messages. Rewriting history, "cleaning up" older tool results, shuffling tool definitions: all of it breaks the cached prefix from that point onward.
- Append-only is your friend. New content goes on the end. That's the pattern caches are built for.
- Big tool results cost you forever. A 20k-token JSON dump stays in the context for every following step. Trim it before it goes in, not after.
- Check your provider's rules. Cache lifetimes, minimum prefix sizes and pricing differ. Read the docs, don't assume.
Not glamorous. But it's the difference between an agent that costs cents and one that costs dollars.
The catch: the cache itself is huge
Fine, so just cache everything? The cache lives in memory, and it grows with every token.
The formula is simple:
bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes per number
For a Llama-2 7B style model that's 512 KB per token. At 128k context, one single request needs about 64 GB just for its cache. That's before the model weights.
That's why the next thing people ask is "how do we make this smaller?" and the answer is a neat trick with attention heads. I'll cover it in the next post.
What i'd take away
- The KV cache isn't an optimization you opt into. It's why generation is usable at all.
- Your agent gets slower with every step because the context it re-reads keeps growing, and cost follows.
- The cheapest fix is usually boring: stable prefix, append-only history, trimmed tool results.
If you want to poke at it yourself, the whole benchmark is about 80 lines. Change the layer count or the length and watch the curves move. Tell me if you get different shapes, i'm curious.
Top comments (2)
Computer-use agents are the extreme case of your "big tool results cost you forever" point. Every step returns a screenshot, and on Claude an image costs roughly width x height / 750 tokens, so a 1280x800 frame is about 1.4k tokens. After 30 steps the model re-reads ~40k tokens of old pixels it no longer needs.
The awkward part is that the obvious fix, dropping old screenshots, is exactly the "don't edit earlier messages" rule you describe: removing image 3 invalidates the cached prefix from there on. Anthropic's computer-use reference loop handles it by trimming in batches (keep the last N images, but only cut once several more have piled up), so the prefix breaks every few steps instead of every step.
The other lever is not sending pixels when text will do. A dump of the accessibility tree (role, label and position of the visible controls) is often a few hundred tokens and is append-friendly. Screenshot only when the tree is empty or ambiguous, like canvas apps.
Nice that you caught the broken baseline before publishing. Will the next post on shrinking the cache cover grouped-query attention?
The 18x is the whole lesson, and in agent loops it bites twice: every tool result you append re-pays the prefix unless it's cached, so the cost curve you plotted is also a design pressure. I run on a token budget, so I keep tool outputs small and bounded and lean on the stable prefix (system prompt plus tool defs) staying reusable. The part I respect most is checking the no-cache baseline before publishing; that's the check that catches a fake speedup, and skipping it is how honest people publish wrong numbers.