"Half a gig of VRAM per billion parameters" is the rule everyone repeats when picking hardware for a local model, and it works fine right up until the model loads, runs for a few minutes, and then dies mid-conversation with an out-of-memory error. The model fit. The context didn't. I wrote up the full math on DevToolLab, and it comes down to two numbers most people never check.
First, quantized weights are bigger than the quant name implies. Second, at long context the KV cache can weigh more than the model itself. Here's what actually happens, checked against real GGUF files on Hugging Face rather than the arithmetic.
The quant name isn't the bit count
Q4 sounds like 4 bits per weight. Multiply that by parameter count and divide by 8, and an 8B model should land at 4.01 GB. Download the actual bartowski/Meta-Llama-3.1-8B-Instruct-GGUF Q4_K_M file and it's 4.92 GB, 23 percent heavier than the math promised.
That gap isn't a rounding error. K-quants store per-block scale and minimum values next to the quantized weights, and the "M"/"L" variants keep a handful of sensitive tensors (embeddings, output layer) at higher precision. The suffix names the dominant format, not the average across the whole file. The overhead shrinks as precision goes up:
| Quant | Naive size | Real size | Overhead | Real bits/weight |
|---|---|---|---|---|
| Q4_K_M | 4.01 GB | 4.92 GB | +23% | 4.90 |
| Q5_K_M | 5.02 GB | 5.73 GB | +14% | 5.71 |
| Q6_K | 6.02 GB | 6.60 GB | +10% | 6.58 |
| Q8_0 | 8.03 GB | 8.54 GB | +6% | 8.51 |
If you need one planning number, use 0.61 GB per billion parameters at Q4_K_M. That's where the "0.6 GB per billion" folklore actually comes from.
The KV cache is what actually runs you out of memory
Weights are a one-time cost. The KV cache grows with every token in the conversation and never shrinks until you start a new one. The formula is two tensors (K and V), per layer, per token:
cache_bytes_per_token = 2 x layers x kv_heads x head_dim x bytes_per_element
Llama 3.1 8B has 32 layers, a head dimension of 128, and only 8 KV heads (not 32 - grouped-query attention shares key/value projections across query heads, which is a 4x cache saving over full multi-head attention). Plug those in at fp16 and you get 128 KB per token:
| Context | KV cache (fp16) | KV cache (q8) | Total with Q4_K_M weights |
|---|---|---|---|
| 4,096 | 0.54 GB | 0.27 GB | 5.46 GB |
| 8,192 | 1.07 GB | 0.54 GB | 5.99 GB |
| 32,768 | 4.29 GB | 2.15 GB | 9.21 GB |
| 131,072 | 17.18 GB | 8.59 GB | 22.10 GB |
Look at the last row. The model itself is under 5 GB. The cache for its advertised 128k context window is 17.18 GB on top of that - 22.10 GB total for a model everyone calls "runs on anything." That doesn't fit on a 16 GB card and leaves nothing free on a 24 GB one. The full breakdown, including the runnable script, walks through deriving this for any model from its config.json.
Concurrency multiplies the cache, not the weights
Everything above is one conversation. Put a model behind an API two people hit at once, and the cache multiplies by however many sequences are in flight, because each one keeps its own K/V state. Four concurrent 8k sessions on this model is 4.29 GB of cache against 4.92 GB of weights. Sixteen sessions is 17.2 GB, gone, on a 24 GB card, while any single conversation still looks perfectly modest. This is why serving frameworks like vLLM invest so heavily in paged attention: naive allocation reserves space for tokens nobody has generated yet.
If you're sizing for a team instead of your own laptop, multiply the cache column by peak concurrency, and quantize the cache before you do anything else.
Rough hardware guide
| VRAM | Comfortable at Q4_K_M | Realistic context |
|---|---|---|
| 8 GB | 7B-8B | 8k-16k |
| 12 GB | 8B-13B | 16k-32k |
| 16 GB | 13B-14B | 32k, or 8B at 64k |
| 24 GB | 24B-32B | 32k comfortably, 8B at full 128k |
| 48 GB | 70B at Q4 | 32k |
| 96 GB+ | 70B at Q6+ | long context on large models |
Apple Silicon breaks this table a little because memory is unified, not dedicated - a 32 GB M-series Mac can hold a model plus cache that would need a 32 GB discrete GPU. It's slower per token than an equivalent NVIDIA card, but the ceiling is total RAM, not what fits on a board.
Four ways to buy headroom without a new GPU
-
Quantize the KV cache. The single biggest lever.
llama.cppexposes--cache-type-k/--cache-type-v, Ollama has an equivalent flag, and going to q8 roughly halves cache memory for a quality hit most people won't notice in chat. - Set context deliberately. Runtimes often reserve the full advertised window whether you use it or not. Dropping from 128k to 32k on this model frees nearly 13 GB, and most chat/coding sessions never get near 32k anyway.
- Drop a quant level before dropping model size. A 13B at Q4_K_M usually beats an 8B at Q8_0 for similar memory - bigger model, cheaper weights, wins until you go below Q3, where quality visibly falls off.
-
Offload layers instead of giving up.
llama.cppsplits layers between GPU and CPU. It gets slower, sometimes a lot slower, but a slow model beats one that won't load.
Measure it, don't just estimate it
Once the model's loaded, check the real number instead of trusting the arithmetic - allocator overhead, CUDA context and fragmentation all sit on top of the theoretical figure. On NVIDIA, nvidia-smi --query-gpu=memory.used,memory.total --format=csv run during a long conversation shows the cache filling in real time. On Apple Silicon, Activity Monitor's memory tab shows the same thing, and watch for the machine starting to swap, which shows up as tokens/sec collapsing rather than a clean error.
The tell-tale sign of a cache problem: generation is snappy for the first few exchanges, then slows hard or crashes as the conversation grows. Weights load once at startup, so a failure that shows up later is almost always the cache eating into memory you didn't budget for.
To size a model you haven't downloaded yet, pull four fields from its config.json on Hugging Face: num_hidden_layers, num_key_value_heads, hidden_size, and num_attention_heads (head dimension is hidden_size / num_attention_heads). Watch num_key_value_heads especially - when it equals num_attention_heads, the model uses full multi-head attention and its cache is several times larger per token than a GQA model of the same size. Two 8B models can differ 4x on cache memory for exactly that reason.
Sizing a local model is really two separate calculations: parameters times real bits-per-weight for the weights, and 2 x layers x kv_heads x head_dim x bytes per token for the cache. At long context the second number usually matters more than the first. Check both before you buy a card - DevToolLab's Data Storage Converter handles the GB-vs-GiB conversion, and the File Size Converter is useful for checking a downloaded GGUF against the size the repo advertises - and quantize the cache before you compromise on the model.
Top comments (0)