KV Cache Math: Why Llama 3.1 8B at 128K Context Won't Fit in 24GB

I quantized Llama 3.1 8B down to Q4 so it would fit comfortably on my 24GB card. The weights came out under 5 GB. Then I set the context to 128K, because the model card says it supports 128K, and the model promptly spilled onto the CPU and generated text at the speed of a fax machine.

The weights were never the problem. The KV cache math was. At 128K tokens, Llama 3.1 8B needs 16 GiB of KV cache in fp16, which is more than three times the size of the Q4 weights I had been so proud of shrinking.

This post is the arithmetic I should have done before touching that setting.

TL;DR

• KV cache size per token = 2 × layers × kv_heads × head_dim × bytes_per_element. For Llama 3.1 8B in fp16 that is exactly 128 KiB per token.
• At 131,072 tokens that is 16 GiB, on top of the weights. Quantizing the weights to 4-bit does nothing to this number.
• Most runtimes (llama.cpp, Ollama, vLLM) reserve KV memory for the full configured context, so a context setting you never use still costs you memory.
• Fixes, in order: set the context you actually need, quantize the KV cache to 8-bit (halves it), and budget KV across concurrent requests, since the cache scales with total tokens in flight.

What is the KV cache actually storing?

The KV cache stores the key and value vectors for every past token, in every layer, so the model doesn't recompute them for each new token. Without it, generating token 10,001 would mean re-running attention projections over the previous 10,000 tokens from scratch.

Each new token needs to attend to all previous tokens. Attention needs a key and a value for each of those tokens, at every layer. So the cache holds one K vector and one V vector per token, per layer, per KV head.

That is where the formula comes from:

bytes_per_token = 2            # one K, one V
                × n_layers
                × n_kv_heads
                × head_dim
                × bytes_per_element

Then multiply by the number of tokens and the number of concurrent sequences. That's the whole thing. No magic, just a very large multiplication that nobody does until something breaks.

How much KV cache memory does Llama 3.1 8B need at 128K context?

Llama 3.1 8B needs 16 GiB of KV cache at 131,072 tokens in fp16. Here are the numbers from its config:

• num_hidden_layers: 32
• num_key_value_heads: 8
• head_dim: 128 (4096 hidden / 32 attention heads)
• fp16: 2 bytes

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token
128 KiB × 131,072 tokens = 16 GiB

Now add the weights:

Weights	Weight size	KV at 128K (fp16)	Total before overhead
fp16	~15 GiB	16 GiB	~31 GiB
Q4_K_M	~4.6 GiB	16 GiB	~20.6 GiB

The fp16 row obviously doesn't fit in 24 GiB. The Q4 row looks like it squeaks in, until you remember the CUDA context, the compute buffers for prefill, and whatever your desktop is already using on that GPU. My runtime did the conservative thing and offloaded layers to system RAM. ollama ps stopped saying 100% GPU, and throughput fell off a cliff.

The lesson that took me too long: weight quantization and KV cache size are independent. Q4 shrinks the part that is fixed. The part that grows with context stays at 16 bits unless you change it separately.

Why didn't this matter with older models?

It did matter, but contexts were short enough to hide it. Llama 2 7B has no grouped-query attention: 32 layers, 32 KV heads, head_dim 128.

2 × 32 × 32 × 128 × 2 = 524,288 bytes = 512 KiB per token

That is four times worse per token than Llama 3.1 8B. But Llama 2 shipped with a 4K context, so the cache was 2 GiB and nobody cared.

Llama 3's 8 KV heads (grouped-query attention, where several query heads share one K/V head) cut the per-token cost by 4x. Then the context window grew 32x. Net result: the KV cache got bigger, not smaller.

Different models make very different tradeoffs here. Qwen2.5-7B uses 28 layers and only 4 KV heads:

2 × 28 × 4 × 128 × 2 = 57,344 bytes = 56 KiB per token

Same parameter class, less than half the KV cost of Llama 3.1 8B. If long context on a single GPU is your main constraint, num_key_value_heads in config.json matters as much as the parameter count on the model card.

How do I calculate KV cache size for any model?

Read the model's config.json and plug four fields into the formula. This is the script I now run before I set any context length:

import json

def kv_cache_gib(config_path, tokens, dtype_bytes=2, sequences=1):
    cfg = json.load(open(config_path))
    heads = cfg["num_attention_heads"]
    head_dim = cfg.get("head_dim") or cfg["hidden_size"] // heads
    kv_heads = cfg.get("num_key_value_heads", heads)  # missing = no GQA
    per_token = 2 * cfg["num_hidden_layers"] * kv_heads * head_dim * dtype_bytes
    return per_token * tokens * sequences / 2**30

print(kv_cache_gib("llama-3.1-8b/config.json", 131_072))  # 16.0
print(kv_cache_gib("llama-3.1-8b/config.json", 32_768))   # 4.0
print(kv_cache_gib("llama-3.1-8b/config.json", 131_072, dtype_bytes=1))  # 8.0

Two gotchas. Some configs define head_dim explicitly and it does not equal hidden_size / num_attention_heads, so prefer the explicit field. And sliding-window or hybrid-attention models cache less than this formula says for some layers, so treat the result as an upper bound for those.

Why does an unused context window still eat VRAM?

Because most runtimes allocate the KV cache for the configured maximum up front, not for the tokens you actually send. A 200-token chat on a server configured for 128K still reserves room for 128K.

• llama.cpp allocates the KV buffer at load time for the full n_ctx. The startup log prints the K and V sizes. If you see 8 GiB for K and 8 GiB for V, that's your 16 GiB, spent before the first prompt.
• Ollama wraps llama.cpp, so the same thing happens with num_ctx. It also estimates whether everything fits and silently splits layers between GPU and CPU when it doesn't. No crash, just slowness.
• vLLM pages the cache in blocks, which is far more efficient for many concurrent requests. But at startup it grabs gpu_memory_utilization of the GPU, subtracts weights and activation memory, and turns the rest into KV blocks. If one sequence of max_model_len cannot fit in those blocks, it refuses to start and tells you to raise gpu_memory_utilization or lower max_model_len.

vLLM's failure is the honest one. It fails loudly at boot. Ollama's failure is the sneaky one, because the model still answers, only slowly.

How do I fit long context on a 24GB GPU?

Cap the context at what you use, quantize the KV cache to 8-bit, and budget memory across concurrent sequences. In the order I'd apply them:

1. Set the context you actually need. Most of my workloads never exceeded 20K tokens. At 32K, Llama 3.1 8B needs 4 GiB of KV instead of 16.

# vLLM
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768

# llama.cpp
llama-server -m llama-3.1-8b-q4_k_m.gguf -c 32768

In Ollama, set num_ctx in the Modelfile or in the request options.

2. Quantize the KV cache, not just the weights. 8-bit KV halves the cache. 8-bit KV is generally a small quality hit; 4-bit KV is where I'd run my own evals before trusting it.

# vLLM: fp8 KV cache
vllm serve meta-llama/Llama-3.1-8B-Instruct --kv-cache-dtype fp8

# llama.cpp: V-cache quantization needs flash attention on
llama-server -m model.gguf -c 65536 -fa -ctk q8_0 -ctv q8_0

# Ollama
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

3. Remember the cache is per sequence. KV memory scales with total tokens in flight. Eight users at 16K each cost exactly as much as one user at 128K. On vLLM, max_num_seqs and max_model_len together decide how many long requests fit at once, and the startup log reports the maximum concurrency it computed for your context length. Read that line. It is the capacity plan you didn't write.

4. Pick the model with your context budget in mind. If you need 128K on one consumer GPU, a model with 4 KV heads buys you twice the room of one with 8, before any quantization.

What I changed

My setup is now boring, which is the goal: Q4 weights, q8_0 KV cache, 32K context, flash attention on. The KV cache is about 2 GiB, the whole thing stays on the GPU, and when I do need a genuinely long document I run a separate process configured for it instead of paying for 128K on every chat.

So why won't Llama 3.1 8B at 128K context fit in 24GB?

Because the KV cache, not the weights, dominates memory at long context. Llama 3.1 8B stores 128 KiB of keys and values per token in fp16 (2 × 32 layers × 8 KV heads × 128 head_dim × 2 bytes), which is 16 GiB at 131,072 tokens, reserved up front by llama.cpp, Ollama and vLLM for the full configured context. Add roughly 15 GiB of fp16 weights, or even 4.6 GiB of Q4 weights plus runtime overhead, and a 24GB GPU runs out. Quantizing weights doesn't shrink the cache. Lowering the context length, switching the KV cache to 8-bit, and budgeting tokens across concurrent requests do.

---

Written by the developer behind Preterview, an interview prep platform.