KV Cache VRAM Calculator: How Much Memory Does Context Length Need? (2026)

The KV cache is the memory a transformer uses to remember every token in the context, and its size grows linearly with context length: bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For Qwen3 32B at FP16 that is 256 KiB per token, or about 8 GB for a 32k-token context, on top of the model weights. This is why a model that loads fine can still run out of memory when you raise the context.

This post gives the formula, the per-token numbers for popular open models (read from their config.json), and how much context fits on 24, 32, 48 and 80 GB cards.

The formula

kv_bytes_per_token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes_per_value
kv_total_bytes     = kv_bytes_per_token × context_tokens × concurrent_sequences

• The 2 is for keys and values.
• num_key_value_heads is smaller than the number of attention heads in models with grouped-query attention (GQA), which is why modern models have much smaller caches than older ones.
• bytes_per_value is 2 for FP16/BF16, 1 for 8-bit KV cache, about 0.5 for 4-bit.
• Concurrent sequences multiply the cache. If your server keeps four parallel slots, budget four times the context.

Per-token KV cache for popular models (FP16)

Architecture values are from each model’s config.json on Hugging Face (read 2026-10-23). GB here means GiB (1024³ bytes).

Model	Layers	KV heads	head_dim	KV per token	8k ctx	32k ctx	128k ctx
Llama 3.1 8B	32	8	128	128 KiB	1.0 GB	4.0 GB	16 GB
Qwen3 8B	36	8	128	144 KiB	1.1 GB	4.5 GB	18 GB*
Qwen3 14B	40	8	128	160 KiB	1.25 GB	5.0 GB	20 GB*
Mistral Small 3 24B	40	8	128	160 KiB	1.25 GB	5.0 GB	— (32k max)
Qwen3 32B	64	8	128	256 KiB	2.0 GB	8.0 GB	32 GB*
Llama 3.3 70B	80	8	128	320 KiB	2.5 GB	10 GB	40 GB
Qwen2.5 72B	80	8	128	320 KiB	2.5 GB	10 GB	40 GB*

* Beyond the model’s native window. Qwen3 dense models are documented with a native 32,768-token context, extendable with YaRN rope scaling; check the model card before going past the native length.

The formula does not apply as-is to models with sliding-window layers (for example Gemma 3 and gpt-oss) or latent attention (DeepSeek V3-style MLA); their caches are smaller than this formula suggests.

How much context fits on each card

Total memory ≈ weights + KV cache + overhead. Weights are the Q4_K_M sizes on our /run pages; overhead is an assumed 1.5 GB for CUDA context, buffers and the runtime. The result is a ceiling for a single sequence; runtimes reserve extra buffers, so plan below it.

Qwen3 32B at Q4_K_M (≈ 19.5 GB weights, 256 KiB/token):

Card class	Left for KV after weights + 1.5 GB	Context ceiling (1 sequence)	Practical guidance
24 GB (RTX 4090 / 3090)	≈ 3 GB	≈ 12k tokens	about 8k; tight
32 GB (RTX 5090)	≈ 11 GB	≈ 45k (native max 32k)	16k–32k comfortable
48 GB (RTX 4090 48G)	≈ 27 GB	≈ 110k	full native 32k, or several parallel slots

Llama 3.3 70B at Q4_K_M (≈ 41.5 GB weights, 320 KiB/token):

Card class	Left for KV	Context ceiling (1 sequence)	Practical guidance
48 GB (RTX 4090 48G / A6000)	≈ 5 GB	≈ 16k	about 8k; tight
80 GB (A100 / A800)	≈ 37 GB	≈ 121k	32k comfortable, with room for parallel requests

The “practical guidance” column matches the per-model notes on our Qwen3 32B page and Llama 3.3 70B page.

A small calculator

def kv_gb(layers, kv_heads, head_dim, ctx, bytes_per_value=2, seqs=1):
    return 2 * layers * kv_heads * head_dim * bytes_per_value * ctx * seqs / 1024**3

def max_ctx(vram_gb, weights_gb, layers, kv_heads, head_dim,
            bytes_per_value=2, overhead_gb=1.5, seqs=1):
    free = vram_gb - weights_gb - overhead_gb
    per_tok = 2 * layers * kv_heads * head_dim * bytes_per_value / 1024**3
    return max(0, int(free / (per_tok * seqs)))

# Qwen3 32B (64 layers, 8 KV heads, head_dim 128), Q4 weights ≈ 19.5 GB
print(kv_gb(64, 8, 128, 32768))          # 8.0
print(max_ctx(32, 19.5, 64, 8, 128))     # ≈ 45k (native limit is 32k)
print(max_ctx(24, 19.5, 64, 8, 128, bytes_per_value=1))  # 8-bit KV: ≈ 24k

Read num_hidden_layers, num_key_value_heads and head_dim (or hidden_size / num_attention_heads) from the model’s config.json.

Three ways to fit more context

1. Quantise the KV cache. Halving bytes per value halves the cache.

# Ollama: 8-bit KV cache (requires flash attention)
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve

# vLLM: FP8 KV cache
vllm serve Qwen/Qwen3-32B-AWQ --kv-cache-dtype fp8 --max-model-len 32768

Quality impact is usually small at 8-bit and more noticeable at 4-bit; test on your own prompts.

2. Set the context you need, not the maximum. In Ollama, pass "options": {"num_ctx": 16384} per request (or set OLLAMA_CONTEXT_LENGTH). In vLLM, --max-model-len caps the context and --gpu-memory-utilization sets how much VRAM the cache may take.

3. Watch parallelism. Ollama’s OLLAMA_NUM_PARALLEL and vLLM’s batching both allocate cache per sequence. If memory runs out under load but not in single tests, this is usually why.

Troubleshooting order when you run out of memory

Say you run Qwen3 32B (Q4_K_M) with Ollama on a 24 GB RTX 4090, set num_ctx to 32768, and get out-of-memory errors or a sudden slowdown (layers spilling to CPU). The formula explains it: about 19.5 GB of weights plus about 8 GB of FP16 KV cache is more than 24 GB. Try these, least effort first:

1. Lower the context. Back to 8192 and the KV cache is about 2 GB; total about 23 GB — it runs, but tight.
2. Quantise the KV cache. With OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the same memory holds roughly twice the context — about 24k by the formula (a ceiling; leave headroom).
3. Check parallelism. Set OLLAMA_NUM_PARALLEL=1 to rule out several slots sharing the memory.
4. Change cards. Still short? Move to a 32 GB RTX 5090 or a 48 GB RTX 4090 48G. Switching means waiting about 2 minutes for the new instance and downloading the model again.

After each change, check real usage with nvidia-smi and note the number; next time you size a card, you will have your own data.

Picking a card from the numbers

Once you have weights + KV + overhead, round up to the next VRAM class and check Deploy for live stock. List prices on 2026-10-23 23:58 Beijing time: RTX 4090 24G $0.33/h, RTX 5090 32G $0.59/h, RTX 4090 48G $0.65/h, A800 80G $1.09/h, A100 80G $1.49/h, billed per minute. Instances start in about 2 minutes. For a side-by-side of those cards, see RTX 4090 vs RTX 5090 vs A100.

FAQ

How do I calculate the KV cache size for a model?

Multiply 2 × layers × KV heads × head dimension × bytes per value (2 for FP16) to get bytes per token, then multiply by context length and the number of parallel sequences. The values are in the model’s config.json.

How much VRAM does a 32k context need for Qwen3 32B?

About 8 GB of FP16 KV cache (256 KiB per token × 32,768 tokens), on top of about 19.5 GB of Q4_K_M weights, so a 32 GB card is the comfortable choice.

Why does my model run out of memory only at long context?

The weights are fixed, but the KV cache grows with every token of context and with each parallel request. Lower num_ctx / max-model-len, reduce parallel slots, or quantise the KV cache.

Does an 8-bit KV cache halve memory?

Yes, the cache itself is halved compared with FP16. In Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention on; in vLLM use --kv-cache-dtype fp8.

Does the formula work for every model?

It works for standard and grouped-query attention. Models with sliding-window layers or latent attention use less memory than it predicts.

---

Once you have your own numbers, the RTX 4090 vs RTX 5090 vs A100 comparison lines up the cards by VRAM class. Originally published at cloudgpu.app.

CloudGPU (cloudgpu.app) has no partnership with, authorization from, or endorsement by the model vendor. Model names are trademarks of their owners. Context ceilings are estimates under the stated assumptions, not measurements. Prices are list prices read on 2026-10-23 23:58 Beijing time (UTC+8); live prices and stock on /deploy take precedence.