"Will this model fit on my GPU?" is one of the most common questions in local and self-hosted LLM work. The answer is three numbers added together. Here is the arithmetic, with a worked example.
All sizes below use 1 GB = 1 billion bytes. The model setup is illustrative, not a benchmark of a specific model.
1. Weights
The weights are the model's learned numbers. Every one must sit in GPU memory.
weights = parameters x bytes per weight
Bytes per weight depend on the precision: 4 (32-bit), 2 (16-bit), 1 (8-bit), about 0.5 (4-bit).
For an 8-billion-parameter model at 16-bit: 8e9 x 2 bytes = 16.0 GB.
2. KV cache
While generating, the model keeps short notes about the text so far, called the KV cache (K and V stand for key and value). Without them it would re-read everything for every new token. The notes cost memory, and they grow with every token and every open chat.
kv_per_token = 2 x layers x kv_heads x head_dim x bytes per cache value
kv_cache = kv_per_token x context_tokens x batch_size
The 2 is one key and one value per layer. For our example: 32 layers, 8 KV heads, head size 128, 16-bit cache values.
- kv_per_token = 2 x 32 x 8 x 128 x 2 = 131,072 bytes
- For 8 chats at 4,096 tokens each: 131,072 x 4,096 x 8 = 4.29 GB
3. Overhead
Real engines need room for temporary buffers and lose some memory to gaps. A rough planning default is 10 to 20 percent of weights plus cache. With 15 percent:
overhead = 0.15 x (16.0 + 4.29) = 3.04 GB
Total
16.0 + 4.29 + 3.04 = about 23.3 GB.
That barely fits a 24 GB card, with under 1 GB spare. Add a few more users or longer chats and it will not fit.
What you can change
- Weights are the biggest part here, so a smaller model or lower precision (for example 8-bit) is the biggest lever.
- If the cache dominates (long context, many users), shorten the context, serve fewer chats at once, or use a lower-precision cache.
Caveats
This is an estimate for planning. Real engines allocate memory differently, some architectures differ (for example mixture-of-experts models store all experts but use only a few per token), and you should read your model's config for the real layer count, KV heads and head size. Test on your own hardware.
Try it yourself
I put the calculator in one HTML file, MIT licensed, with the formulas shown: github.com/deployedlab/llm-vram-calculator.
There is also a step-by-step interactive version with diagrams and a "does it fit" table, free with no signup: GPUs and memory sizing. It is one of seven free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.
Top comments (0)