LLM Inference: KV Cache, Sampling, and Temperature Measured

LLM Inference: KV Cache, Sampling, and Temperature Measured

Training gets the headlines and the giant bills. Inference gets the traffic. Every token a model generates costs a forward pass, and the difference between a fast service and a slow one is rarely the weights. It is the memory around them and the sampling settings on top of them. We ran GPT-2 on our own CPU, measured the KV cache byte by byte, timed generation with the cache on and off, and sampled the same prompt at five temperatures to see exactly where quality turns into repetition and where it turns into noise. The numbers below are ours, measured on 11 November 2026, plus the published figures from the papers that fixed these problems for everyone else.

This is Deep Part 4 of the AI Engineer Deep series. Deep Part 1 built attention by hand, Deep Part 2 built a tokenizer from scratch, and Deep Part 3 trained tiny transformers and read the loss curves. This part takes a trained model and asks what happens when it actually runs. If you only read one section, read the sampling tables. They settle more arguments about temperature than any amount of opinion.

The two phases nobody separates

Every generation has two phases, and they behave nothing alike. Prefill processes the whole prompt at once, in parallel, and builds the KV cache. Decode generates one token at a time, each step reading the entire cache back from memory. The first phase is compute bound. The second is memory bound. Confusing them is the most common mistake in inference discussions.

We measured both on GPT-2, 124 million parameters, running in float32 on 4 CPU threads. A 120 token prompt took 3,425 ms to prefill, about 28.5 ms per prompt token. One cached decode step took 493.8 ms. Prefill processed a token roughly 17 times faster than decode produced one, because prefill works on all tokens at once while decode works on one token and drags the whole cache through memory to do it. The absolute milliseconds are CPU numbers and would look very different on a GPU. The shape of the result would not: prefill is cheap per token, decode is expensive per token, and the gap grows with context length.

This is why serving systems quote two latencies, not one. Time to first token measures prefill. Time per output token measures decode. A long prompt hurts the first number. A long answer hurts the second. When a user says a model feels slow, they usually mean the second number, and the second number is almost entirely a memory problem.

The KV cache: memory you pay for every token

Attention needs the keys and values of every token that came before. Recomputing them at each step would make generation quadratic in the sequence length, so every serving system caches them. The cache is simple to size. Each token stores a key and a value vector in every layer, for every KV head, at the precision of the weights:

bytes per token = 2 x layers x KV heads x head dimension x bytes per number

We verified the formula against the real tensors. GPT-2 has 12 layers, 12 heads, and a head dimension of 64. For an 8 token prompt, the live cache held 589,824 bytes, which is 73,728 bytes per token in float32. The float16 formula predicts 36,864 bytes per token, exactly half, and the per layer key tensor shape (1, 12, 8, 64) matched the config exactly. The formula is not an approximation. It is the cache.

The table below applies it in float16, the precision real serving uses, to models people actually run. The last column is the one that matters: a 128k context window is not a free feature, it is tens of gigabytes of cache per request.

ModelLayersKV headsBytes per token4,096 token context128k token context
GPT-2 124M121236 KiB144 MiB4.5 GiB
Llama 3.1 8B328128 KiB512 MiB16 GiB
Mistral 7B328128 KiB512 MiB16 GiB
Qwen2.5 7B28456 KiB224 MiB7 GiB
Llama 2 70B808320 KiB1.25 GiB40 GiB
Llama 2 7B (full attention)3232512 KiB2 GiB64 GiB

Read the Llama 2 7B row against the Llama 3.1 8B row. Same size class, four times less cache per token, because the 8B model shares KV heads across query heads. That is the whole story of the next section. And read the 70B row against an 80 GB GPU: one request at 128k context takes 40 GiB of cache, half the card, before a second user arrives. At long context, the cache is the capacity plan. The weights are a fixed cost. The cache is the variable one, and it is the one that decides how many requests fit.

How much the cache is actually worth

A formula says the cache saves compute. We measured how much. Generating 48 tokens from a 120 token prompt took 23.05 seconds with the cache enabled, 480 ms per token, and 57.15 seconds with it disabled, 1,191 ms per token. The cache made generation 2.48 times faster on this CPU, for this prompt length, and the advantage grows with context because the no cache path recomputes the full prompt at every single step.

Two honest caveats. First, these are CPU timings, and the absolute numbers flatter nothing: 480 ms per token is slow, and a GPU would cut both numbers by orders of magnitude. What transfers is the ratio and the reason for it, not the milliseconds. Second, the cache is not free memory. It is a trade, memory for speed, and at some context length the memory runs out first. That trade is exactly what the next two sections manage.

Shrinking the cache at the source: grouped query attention

Full multi head attention gives every query head its own key and value head. That is the most expressive design and the most expensive cache. Multi query attention, from Shazeer in 2019, goes to the other extreme: one shared key and value head for all query heads. The cache shrinks by the head count, but quality drops enough that large models felt it.

Grouped query attention, from Ainslie and colleagues in 2023, is the middle setting that almost every modern model shipped. Query heads are split into groups, and each group shares one key and value head. The cache shrinks by exactly the ratio of query heads to groups, and the quality cost is small enough to measure and then ignore. On T5-XXL, grouped query attention with 8 groups scored 47.1 against 47.2 for full attention, at about five times the inference speed, after uptraining on only 5 percent of the original training compute. Llama 2 70B uses 64 query heads over 8 KV groups, so its cache is one eighth of the full attention version: 1.25 GiB instead of 10 GiB at a 4,096 token context. Llama 3 uses grouped query attention across the whole family, including the 8B model, which is why its row in the table above looks so different from Llama 2 7B.

Our take is simple. If you are choosing a model to serve and two candidates are close on quality, take the one with fewer KV heads. It is the cheapest capacity you will ever buy, because it is decided at training time and paid back on every single request forever.

PagedAttention: the operating system trick

Even with a small cache per token, serving systems wasted most of the memory they reserved. The classic design allocated one contiguous block per request, sized for the maximum sequence length, up front. A request that might reach 2,048 tokens reserved the full 2,048 tokens of space whether it used 200 or 2,000. Kwon and colleagues measured the damage in the PagedAttention paper at SOSP 2023: for their OPT-13B example, one token costs about 800 KB of cache and a 2,048 token request reserves about 1.6 GB, yet only 20.4 to 38.2 percent of the KV cache memory in existing systems actually held token states. The rest was reserved and empty, or lost to fragmentation between requests.

The fix is borrowed from operating systems. PagedAttention splits the cache into fixed size blocks, 16 tokens each by default, allocates blocks on demand as the sequence grows, and maps logical blocks to physical ones through a block table. Requests share the blocks holding a common prompt, and copy on write handles the point where their outputs diverge. Measured waste falls to about 4 percent, with 96.3 percent of KV cache memory holding real token states, and throughput rises 2 to 4 times over the systems of the time at the same latency, with no change to the model or its outputs. The same paper notes the honest cost: the paged attention kernel itself runs 20 to 26 percent slower than the contiguous version. The win comes from fitting more requests, not from faster math.

That tradeoff is the right mental model for all inference work. Throughput is usually a memory management problem wearing a compute costume. Before reaching for a bigger GPU, check whether the cache is the thing that is actually full.

Sampling: temperature is not a creativity dial

The second half of inference is choosing the next token, and this is where most folklore lives. Temperature divides the logits before the softmax. Lower temperature sharpens the distribution, higher temperature flattens it. That is all it does. It does not add creativity, it changes the odds.

We measured the effect on the first token after the prompt "The future of artificial intelligence in software engineering is". At temperature 0.2, the top token held 95.1 percent of the probability, the top 5 held 99.7 percent, and a single token covered 90 percent of the mass. The entropy was 0.25 nats. At temperature 1.5, the top token held 2.5 percent, the top 5 held 7.8 percent, and 3,327 tokens were needed to cover 90 percent of the mass. Entropy was 7.39 nats. Same model, same prompt, same logits. Temperature alone moved the choice from a near certainty to a lottery ticket.

TemperatureEntropy (nats)Top token probabilityTop 5 massTokens for 90 percent of mass
0.20.2595.1%99.7%1
0.73.6125.2%50.7%57
1.05.2810.0%25.1%368
1.57.392.5%7.8%3,327

Then we generated 60 tokens at each setting, same prompt, same seed, and counted what came out. Greedy decoding, which always takes the top token, scored a distinct token ratio of 0.22 and a repeated 4 gram rate of 0.70. It repeated the sentence "The future of artificial intelligence in software engineering is uncertain" three times in 60 tokens. Temperature 0.2 did better but still looped, with a repeated 4 gram rate of 0.37. Temperature 0.7 broke the loop: distinct ratio 0.77, repeated 4 grams 0.04, and readable prose. Temperature 1.0 and 1.5 both reached a distinct ratio above 0.85 with zero repeated 4 grams, but 1.5 drifted into invented facts, including a consultancy announcing 28 billion dollars of capital. The diversity is real. So is the nonsense.

SettingDistinct token ratioRepeated 4 gram rateWhat came out
Greedy0.220.70Same sentence three times
Temperature 0.20.500.37Loops after two sentences
Temperature 0.70.770.04Coherent, varied prose
Temperature 1.00.860.00Varied, slightly loose
Temperature 1.50.950.00Varied, invented facts
Temperature 0.7, top-p 0.90.730.02Coherent, slightly safer
Temperature 0.7, top-k 50.730.00Coherent, narrow vocabulary

The filters on top of temperature matter, and they matter differently. Top-k keeps a fixed number of candidates, which is a blunt tool: 5 is narrow everywhere, 50 is wide everywhere. Top-p, nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, so it adapts. When the model is confident, as at temperature 0.2 where one token holds 95 percent, top-p keeps almost nothing. When the model is unsure, as at temperature 1.0 where 368 tokens hold 90 percent of the mass, it keeps hundreds. That adaptivity is why we prefer top-p 0.9 over top-k as a default. It trims the tail exactly where the tail is long and leaves confident distributions alone. Hugging Face ships temperature 1.0, top-k 50, and top-p 1.0 as defaults, with greedy decoding as the default strategy, so the settings above are all choices you make, not choices the library makes for you.

What we run in production

Four rules, each earned by a number above. First, use greedy decoding only for short, factual, or structured output, and expect it to loop on anything longer than a sentence. Second, use temperature 0.7 with top-p 0.9 for prose, and treat anything above 1.0 as a deliberate tradeoff of accuracy for variety, never as a free upgrade. Third, size the KV cache before promising concurrency: multiply bytes per token by the longest context you allow by the number of simultaneous requests, and compare that against the GPU memory left after the weights. Fourth, remember that prefill and decode are different bills: a long system prompt taxes every request once, while a long answer taxes the cache for its whole lifetime. Prefix caching, which reuses the cache of a shared prompt across requests, attacks the first bill directly, and it is the cheapest latency win most teams have not turned on.

The opinion underneath all of it: inference quality is a systems problem, not a model problem. The model gives you a distribution. The cache decides how many people can use it at once, and the sampler decides what they actually read. Get those two right and a small model feels fast and smart. Get them wrong and a large model feels slow and repetitive, and no amount of prompt engineering fixes it.

What is next in this series

This is Deep Part 4 of the AI Engineer Deep series, a line by line revisit of the original 8 part series with real research and real measurements. Deep Part 5 takes on context windows: RoPE, long context, and why models forget, including what actually happens to attention when the context grows past the length the model trained on. The original series still stands as the map: Deep Part 3, Training LLMs is where the model in this article came from, Part 3, How LLMs Actually Work is where tokens and temperature first appeared, and Part 8 on LLMOps shows what the inference bill turns into once the model ships. If you want the cost side of this story now, our measured guide to cutting the LLM API bill puts numbers on five techniques.

Previous: Training LLMs: Pretraining, Loss Curves, and Scaling Laws

Next: Deep Part 5: Context Windows: RoPE, Long Context, and Why Models Forget (coming next in this series)

---

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.