DEV Community

Cover image for How much GPU memory does an LLM need? The arithmetic, step by step
Deployed Lab
Deployed Lab

Posted on

How much GPU memory does an LLM need? The arithmetic, step by step

LLM VRAM calculator: GPU memory for any model | Deployed Lab

Free LLM VRAM calculator: estimate GPU memory for weights, KV cache and overhead from parameters, precision, context and batch size.

favicon deployedlab.dev

"Will this model fit on my GPU?" is one of the most common questions in local and self-hosted LLM work. The answer is three numbers added together. Here is the arithmetic, with a worked example.

All sizes below use 1 GB = 1 billion bytes. The model setup is illustrative, not a benchmark of a specific model.

1. Weights

The weights are the model's learned numbers. Every one must sit in GPU memory.

weights = parameters x bytes per weight
Enter fullscreen mode Exit fullscreen mode

Bytes per weight depend on the precision: 4 (32-bit), 2 (16-bit), 1 (8-bit), about 0.5 (4-bit).

For an 8-billion-parameter model at 16-bit: 8e9 x 2 bytes = 16.0 GB.

2. KV cache

While generating, the model keeps short notes about the text so far, called the KV cache (K and V stand for key and value). Without them it would re-read everything for every new token. The notes cost memory, and they grow with every token and every open chat.

kv_per_token = 2 x layers x kv_heads x head_dim x bytes per cache value
kv_cache     = kv_per_token x context_tokens x batch_size
Enter fullscreen mode Exit fullscreen mode

The 2 is one key and one value per layer. For our example: 32 layers, 8 KV heads, head size 128, 16-bit cache values.

  • kv_per_token = 2 x 32 x 8 x 128 x 2 = 131,072 bytes
  • For 8 chats at 4,096 tokens each: 131,072 x 4,096 x 8 = 4.29 GB

3. Overhead

Real engines need room for temporary buffers and lose some memory to gaps. A rough planning default is 10 to 20 percent of weights plus cache. With 15 percent:

overhead = 0.15 x (16.0 + 4.29) = 3.04 GB
Enter fullscreen mode Exit fullscreen mode

Total

16.0 + 4.29 + 3.04 = about 23.3 GB.

That barely fits a 24 GB card, with under 1 GB spare. Add a few more users or longer chats and it will not fit.

What you can change

  • Weights are the biggest part here, so a smaller model or lower precision (for example 8-bit) is the biggest lever.
  • If the cache dominates (long context, many users), shorten the context, serve fewer chats at once, or use a lower-precision cache.

Caveats

This is an estimate for planning. Real engines allocate memory differently, some architectures differ (for example mixture-of-experts models store all experts but use only a few per token), and you should read your model's config for the real layer count, KV heads and head size. Test on your own hardware.

Try it yourself

I put the calculator in one HTML file, MIT licensed, with the formulas shown: github.com/deployedlab/llm-vram-calculator.

There is also a step-by-step interactive version with diagrams and a "does it fit" table, free with no signup: GPUs and memory sizing. It is one of seven free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.

Top comments (0)