What is a KV cache in LLM inference? (and why it, not the weights, limits your throughput)

I've been working through the inference chapter of a free course I've been studying — AI Engineering: Zero to Production — and
the KV-cache lesson finally made something click that I'd been hand-waving for months. Sharing my notes here in case it helps someone else.

The thing that surprised me: I always assumed the model weights were what filled up the GPU. Turns out the thing that actually caps how many users you can serve at
once is usually the KV cache. Here's what I took away.

## What the KV cache actually is

At each decode step, the model attends to every previous token. Recomputing the keys and values for all of them, every step, would be hugely wasteful — so they get
stored. That store is the KV cache.

One decode step, in plain terms:

1. New token — the latest token the model is processing.
2. Compute its K, V — think of K ("key") as what this token is about and V ("value") as the info it carries.
3. Append to the cache — this is why the cache grows every single step, and why long chats keep eating memory.
4. Attend over all cached K, V — to pick the next word, the model compares the new token against everything stored, without recomputing it. That's the whole speed win.

So: the KV cache is the model remembering the K/V of past tokens so it never redoes that work. Fast decode, but it's the main thing eating GPU memory.

## The part that actually surprised me — the arithmetic

  def kv_bytes(layers, heads, head_dim, seq_len, batch=1, bytes_per=2):
      # leading 2 = we store BOTH K and V
      return 2 * layers * heads * head_dim * seq_len * batch * bytes_per

  b = kv_bytes(layers=32, heads=32, head_dim=128, seq_len=2048)
  print(f"{b / 1024**2:.1f} MB")   # 1024.0 MB → 1 GiB for a SINGLE 2k sequence

~1 GB of KV cache for one 2,048-token request on an ordinary 32-layer/32-head model. That was the "oh" moment for me — the cache, not the weights, is what bounds
concurrency. And it grows on two axes at once:

• Context length — longer prompts = bigger cache per request.
• Concurrency — every simultaneous request needs its own.

## Two levers I hadn't fully understood

PagedAttention (the vLLM idea): instead of each request reserving one big contiguous block for its max possible length (wasteful), it manages the cache like OS
virtual memory — non-contiguous pages on demand. That's how a server fits way more concurrent requests in the same GPU.

Prompt caching: if requests share a prefix (long system prompt, RAG context, few-shot preamble), you cache its KV and skip re-prefilling it. On hosted APIs it's a
billing discount; self-hosting it's real compute saved.

  from anthropic import Anthropic

  client = Anthropic()
  BIG_SYSTEM = open("policy_manual.txt").read()   # long, reused preamble

  resp = client.messages.create(
      model="claude-opus-4-8",
      max_tokens=300,
      system=[{
          "type": "text",
          "text": BIG_SYSTEM,
          "cache_control": {"type": "ephemeral"},   # <- marks this block cacheable
      }],
      messages=[{"role": "user", "content": "Summarize section 4."}],
  )

  print(resp.usage)  # cache_creation_input_tokens vs cache_read_input_tokens

What made it concrete: run it twice. First call = big cache write, ~zero read. Second call = the read jumps, the write drops. The savings only land on the 2nd+
call that reuses the prefix.

## The mental model I'm keeping

• KV cache = remembered K/V for past tokens → fast decode.
• Grows with context length × concurrency → it, not the weights, limits how many users you serve.
• PagedAttention cuts waste when self-hosting; prompt caching reuses shared prefixes to cut repeat cost.

If you want to actually run the size formula and the caching example (the lesson has an in-browser Python terminal, no setup), the original is here 👉 KV-Cache &
Attention Memory in LLM Inference. The whole course builds RAG, agents, eval, and production LLM stuff by hand:
ai.studybydoing.in.