Why your million token agent run costs what it costs

Every token a long-context agent generates gets billed against the whole conversation behind it. That sounds like an accounting quirk until you look at what the hardware is actually doing during decode. Then the bill makes sense, and so do the tactics that shrink it.

I went through the full math on the PastAGI blog this week and wanted to pull out the parts that matter if you're running agents at any real scale.

The uncomfortable arithmetic

To produce each token, the model scores a fresh query vector against the key of every cached token, then blends the cached value vectors based on those scores. The compute is trivial. The reads are not.

Every key and value for every earlier token moves from GPU memory into the compute units on every step. The KV cache size per token comes down to:

bytes per token = 2 x layers x KV heads x head dim x bytes per element

The part that gets people: attention cannot skip a read. The weight that would justify skipping a token is computed from reading that token. If attention ends up caring about one paragraph out of four hundred thousand tokens, the hardware still read all four hundred thousand to find it.

The original piece calls this "peaked attention paying for flat reads" and that's the right way to think about it. Your bottleneck is memory bandwidth, not model capability.

What you can do today

The full article works through a scoreboard of tactics ranked by how much of the theoretical skip each one captures. The short version:

Eviction methods (StreamingLLM with attention sinks, H2O heavy hitters, SnapKV observation windows) throw away cache entries the model is unlikely to attend to again. Each makes a bet about which tokens matter, and each loses a little accuracy when the bet is wrong.

Quantization (fp8, int4) shrinks the bytes per element rather than the token count. Often the cheapest win because nothing about the context changes, just its storage precision.

Prompt compression shortens the text itself before it ever becomes cache. Highest ceiling, most effort, and it interacts with everything downstream.

The honest answer on which to pick: measure your own sparsity first. The article lays out how to do that, and the answer varies a lot by workload. An agent doing retrieval-heavy work has very different attention patterns from one doing long code edits.

The full breakdown with the worked million-token ledger is at PastAGI, including the attention-control research that would change these economics if it ever ships in production models.

Cross-posted from PastAGI, where I write about LLM serving costs and agent economics.