Before downloading a model to run locally, the question is whether it will fit in memory. The estimate is three terms added together and compared against what the machine can actually give the model:
need = weights + KV cache + overhead
fits if need <= 0.8 × usable memory
The numbers below follow the estimate used by the local model browser in the osFoundry desktop app, checked against what that app shows for a 16GB M3 MacBook.
Weights
Weights are parameter count times bytes per weight, and the bytes per weight depend on the quantization:
| Quant | Bytes per weight |
|---|---|
| FP16 / BF16 | 2.0 |
| Q8_0 | 1.0 |
| Q6_K | 0.82 |
| Q5_K_M | 0.69 |
| Q4_K_M | 0.56 |
| Q3_K_M | 0.44 |
| Q2_K | 0.31 |
Q4_K_M is 0.56 rather than 0.5 because K-quants store a scale for each small block of weights, which works out to about 4.5 bits per weight.
For a 14B model at Q4_K_M: 14 × 10⁹ × 0.56 = 7.84 × 10⁹ bytes, or about 7.3 GiB.
If you don't recognize the quant name, assume FP16. It's the pessimistic choice, so you won't decide something fits when it doesn't.
KV cache
During generation the model keeps keys and values for every token in the context. The size is:
2 × layers × kv_heads × head_dim × context_tokens × bytes_per_element
Qwen2.5 14B has 48 layers, 8 KV heads (it uses grouped-query attention) and a head dimension of 128. With an FP16 cache:
- 4K context: 2 × 48 × 8 × 128 × 4096 × 2 bytes = 768 MiB
- 32K context: 6 GiB
The cache grows linearly with context length, so a model that fits easily at 4K can stop fitting at 32K. Estimates that don't state a context length aren't much use; the one in the screenshot assumes 4K.
Overhead
llama.cpp also needs compute buffers and scratch space. A rough allowance is 10% of the weights plus about 200 MB. For the 14B model that's about 0.9 GB.
Adding the weights, the cache and the overhead puts a 14B model at Q4_K_M in the 8 to 9 GB range. The app shows 8.4 GB.
Usable memory
This is the part people get wrong most often.
- Discrete GPU: the card's VRAM is the ceiling. A 12 GB card means 12 GB, however much system RAM you have.
- Apple Silicon: memory is unified, but the GPU can't use all of it and the OS needs its share. Counting 75% of total RAM is a reasonable working number, so 16 GB gives about 12 GB.
Then compare:
- need ≤ 80% of usable: fits, with room for the OS and other apps
- 80% to 100%: tight; it loads, but close other apps first
- over 100%: it pages to disk and becomes unusably slow
On the 16GB Mac that gives: Q4_K_M 8.4 GB (70%, fits), Q5_K_M 10.4 GB (87%, tight), Q6_K 12.3 GB and Q8_0 14.9 GB (over).
Applied across models at Q4_K_M, the cut-off on this machine falls between 20B (11.9 GB, tight) and 27B (15.9 GB, too big).
Speed follows the same number
Generating each token requires reading every weight from memory once, so an upper bound on generation speed is memory bandwidth divided by model size. A base M3 has about 100 GB/s, so an 8.4 GB model tops out around 12 tokens per second, and a 4.9 GB 8B model around 20. Real numbers come in below that, but the ratio between models holds up well.
So the size estimate answers two questions at once: whether the model loads at all, and roughly how fast it will write once it does.
Top comments (0)