"Will it fit on my card?" comes up constantly in local-LLM threads, and the usual answer is a rule of thumb like "about half a gigabyte per billion parameters at 4-bit." That rule is close enough for small dense models. For mixture-of-experts models, models with sliding-window attention, and long contexts it can be badly off.
So instead of a rule of thumb, I put together real numbers. For 30 popular open-weight models I recorded the actual GGUF download sizes from Hugging Face and the KV-cache size per token, worked out from each model's config.json. This post uses the 26 models that have both a 4-bit file and a sourced KV-cache figure.
All the data is open (CC BY 4.0) at huggingface.co/datasets/vramneeded/llm-vram-requirements. It's the same data behind the comparison table on vramneeded.com.
The method: weights + KV cache + overhead
The memory a local model uses during inference comes from three parts:
- Weights. I use the actual GGUF file size, not params × bits. Real quant files mix precisions: embeddings and some tensors stay at higher precision, and MoE experts in some models stay in MXFP4. The file size captures all of that. Sizes are decimal (1 GB = 10⁹ bytes).
- KV cache. For every token in context, each attention layer stores a key and a value vector:
KV bytes per token = layers × KV heads × head_dim × 2 (K and V) × 2 bytes (FP16)
I count only layers whose cache grows with context. Fixed-size state from sliding-window or linear-attention layers is left out, which is why some newer models have tiny KV numbers. Where I couldn't source the figure, the field is blank. I don't guess.
- Overhead. Runtime buffers, the CUDA context, compute scratch and so on. Instead of making up a fixed number, the fit rule simply leaves headroom: a model "fits" a GPU tier only if weights + KV cache come to at most 90% of the card's memory.
The settings used here are the Q4_K_M quant (or the closest 4-bit build where no Q4_K_M exists, noted below), an FP16 KV cache, and an 8,192-token context. The GPU tiers are 8, 12, 16, 24, 32, 40, 48, 64, 96 and 128 GB.
The table
| Model | Params | 4-bit weights (GB) | KV @ 8K (GB) | Weights + KV (GB) | KV @ 32K (GB) | Smallest tier that fits |
|---|---|---|---|---|---|---|
| LFM3 1.7B | 1.7B | 1.10 | 0.10 | 1.20 | 0.40 | 8 GB |
| Llama 3.2 3B Instruct | 3.2B | 2.02 | 0.94 | 2.96 | 3.76 | 8 GB |
| Spark-X2.6 4B | 4.1B | 3.54 | 0.30 | 3.85 | 1.21 | 8 GB |
| Qwen2.5 7B Instruct | 7.6B | 4.68 | 0.47 | 5.15 | 1.88 | 8 GB |
| Mistral 7B Instruct v0.3 | 7.2B | 4.37 | 1.07 | 5.45 | 4.29 | 8 GB |
| Llama 3.1 8B Instruct | 8.0B | 4.92 | 1.07 | 5.99 | 4.29 | 8 GB |
| DeepSeek-R1-0528-Qwen3-8B | 8.2B | 5.03 | 1.21 | 6.24 | 4.83 | 8 GB |
| Qwen3 14B | 14.8B | 9.00 | 1.34 | 10.34 | 5.37 | 12 GB |
| Qwen2.5 14B Instruct | 14.8B | 8.99 | 1.61 | 10.60 | 6.44 | 12 GB |
| gpt-oss-20b | 20.9B | 11.67 | 0.20 | 11.87 | 0.81 | 16 GB |
| LFM2.5-24B-A2B | 24.0B | 14.53 | 0.17 | 14.69 | 0.67 | 24 GB |
| Qwen3.8 27B | 27.8B | 17.44 | 0.54 | 17.98 | 2.15 | 24 GB |
| Qwen3.8-35B-A3B (Empero Distill) | 35.1B | 21.71 | 0.17 | 21.88 | 0.67 | 32 GB |
| Qwen3 32B | 32.8B | 19.76 | 2.15 | 21.91 | 8.59 | 32 GB |
| Qwen2.5 32B Instruct | 32.8B | 19.85 | 2.15 | 22.00 | 8.59 | 32 GB |
| Qwen3.6 35B-A3B | 36.0B | 22.29 | 0.17 | 22.45 | 0.67 | 32 GB |
| Nanbeige4.3 33B | 33.1B | 20.79 | 1.74 | 22.53 | 6.98 | 32 GB |
| Recursive Flint ¹ | 39.8B | 25.05 | 0.23 | 25.29 | 0.94 | 32 GB |
| Mixtral 8x7B Instruct v0.1 | 46.7B | 26.44 | 1.07 | 27.52 | 4.29 | 32 GB |
| Llama 3.1 70B Instruct | 70.6B | 42.52 | 2.68 | 45.20 | 10.74 | 64 GB |
| Llama 3.3 70B Instruct | 70.6B | 42.52 | 2.68 | 45.20 | 10.74 | 64 GB |
| gpt-oss-120b | 116.8B | 62.84 | 0.30 | 63.14 | 1.21 | 96 GB |
| Ornith-1.5-122B ¹ | 125.1B | 79.71 | 0.20 | 79.92 | 0.81 | 96 GB |
| Qwen3.8 Flash Next | 180.0B | 119.60 | 0.20 | 119.80 | 0.81 | over 128 GB |
| DeepSeek-V4.1-Flash-1017 ¹ | 763.2B | 500.71 | 0.01 | 500.71 | 0.03 | over 128 GB |
| Mistral Large 4 | 1,049.4B | 634.80 | 0.69 | 635.49 | 2.75 | over 128 GB |
¹ No Q4_K_M build is published, so the 4-bit column is Unsloth's UD-Q4_K_XL.
Per-model notes from the dataset: gpt-oss keeps its MoE experts in MXFP4 in every GGUF, so its Q4_K_M, Q8_0 and BF16 files are close in size. For DeepSeek-V4.1-Flash-1017, keeping the Engram table on SSD brings UD-Q4_K_XL down to 297.9 GB of RAM/VRAM.
The chart: KV cache at 8K context
The cache size per token varies a lot from model to model, and it matters more than people expect. Here is the KV cache at 8K context (FP16) for a selection of the models above:
Llama 3.1/3.3 70B ███████████████████████████ 2.68 GB
Qwen3 32B ██████████████████████ 2.15 GB
Qwen2.5 32B ██████████████████████ 2.15 GB
Nanbeige4.3 33B █████████████████ 1.74 GB
Qwen2.5 14B ████████████████ 1.61 GB
Qwen3 14B █████████████ 1.34 GB
Llama 3.1 8B ███████████ 1.07 GB
Mixtral 8x7B ███████████ 1.07 GB
Llama 3.2 3B █████████ 0.94 GB
Qwen3.8 27B █████ 0.54 GB
Qwen2.5 7B █████ 0.47 GB
gpt-oss-120b ███ 0.30 GB
gpt-oss-20b ██ 0.20 GB
Qwen3.6 35B-A3B ██ 0.17 GB
Takeaways
1. At 8K, the KV cache is a small slice for most models, but it grows linearly. Llama 3.1 8B needs 1.07 GB of cache at 8K. At 32K that becomes 4.29 GB, almost as much as its 4.92 GB of weights. If you plan to use long contexts, budget for the cache rather than just the file size.
2. Architecture matters as much as parameter count. Llama 3.2 3B has a 0.94 GB cache at 8K, while the 36B-parameter Qwen3.6 35B-A3B has 0.17 GB, because most of its layers don't keep a growing cache. The 3B model's cache is more than five times bigger. At 32K the gap is 3.76 GB vs 0.67 GB.
3. 8 GB cards cover the whole 7–8B class at 4-bit and 8K. All seven models from 1.7B to 8.2B fit under 90% of 8 GB, with the largest (DeepSeek-R1-0528-Qwen3-8B) at 6.24 GB.
4. 12 GB is the 14B tier and 24 GB tops out around 27B dense. Both 14B Qwen models land at about 10.3–10.6 GB. Qwen3.8 27B comes to 17.98 GB, under the 21.6 GB that 90% of 24 GB allows. The 32B-class models (about 21.9–22.5 GB) go just over that line, so in this rule they need the 32 GB tier. A shorter context or a quantized KV cache can bring them closer, but at 8K with FP16 cache they are over the line.
5. The 4-bit 70B still needs more than one consumer GPU. Llama 3.1/3.3 70B at Q4_K_M plus 8K context comes to 45.20 GB, which is over 90% of 48 GB (43.2 GB). That puts it in the 64 GB tier here.
6. "Half a gigabyte per billion" is a floor, not a forecast. The 4-bit files in this set range from 0.54 GB per billion parameters (gpt-oss-120b: 62.84 GB for 116.8B, with MXFP4 experts) to 0.86 GB per billion (Spark-X2.6 4B: 3.54 GB for 4.1B). Llama 3.1 8B is 0.61 (4.92 GB for 8.0B). Check the real file.
Use the data
-
Dataset: vramneeded/llm-vram-requirements on Hugging Face. It includes
vram-models.csv(one row per model, with source links to the GGUF repo andconfig.json),quant-sizes.csv(514 quant sizes, not just Q4_K_M) and a cloud GPU price snapshot. Every row has its own checked date. - Calculator and per-model pages: vramneeded.com has the same numbers with a context-length slider and every quant.
- Badges: if you publish a model or a model guide, there are free "fits on X GB" README badges at vramneeded.com/badges/. They're built from the same 4-bit, 8K, 90%-headroom rule used in this post.
If a number looks wrong, it may well be: repos get re-uploaded and configs change. Open an issue on the dataset with the source link and I'll fix it.
Data: GGUF file sizes from Hugging Face (mostly bartowski's builds; each row names its repo) and KV cache figures from each model's config.json, checked 2026-10-23 to 2026-10-31. Weights + KV values in this post are computed from the dataset as q4_k_m_bytes + kv_bytes_per_token_fp16 × 8192.
Top comments (1)
the 90% headroom rule is the part i'd actually use, because the "half a gig per billion at 4-bit" rule of thumb quietly assumes the kv cache is small enough to ignore, and it isn't once context climbs. the sliding-window point is the one that surprises people too — gpt-oss and the linear-attention models show tiny kv numbers not because they're efficient but because their fixed-size state doesn't grow with context, so the "smallest tier that fits" for those is a real floor, not a rough one. i keep hitting the same wall on my own box: a 27b that fits comfortably at 8k context stops fitting the moment i push past 32k, and it's the cache, not the weights, that eats the card.