I keep seeing the same local LLM sizing mistake:
"The model file is smaller than my GPU, so it should fit."
That is only the first check. A 24 GB GPU does not give your model a clean 24 GB memory budget. The display stack, runtime, temporary buffers, model weights, and KV cache all compete for the same space.
Here is the worksheet I use before I download a model or rent a GPU.
1. Start with the weight floor
The simplest weight estimate is:
weight_memory_gib = parameters * bits_per_parameter / 8 / 1024^3
For a simple 4-bit estimate:
| Model size | Weight floor |
|---|---|
| 7B | 3.3 GiB |
| 13B | 6.1 GiB |
| 70B | 32.6 GiB |
These are floors, not promises. Real quantized files can also contain scales, metadata, and layers stored at higher precision. If you know the exact checkpoint size, use that instead of the simple bits-per-parameter estimate.
Also use total parameters for a sparse mixture-of-experts model unless your runtime really offloads inactive experts. Active parameters describe compute per token. They do not automatically describe how many weights must be stored.
2. Reduce the physical capacity to a usable budget
I normally start with 90 percent usable VRAM for planning:
usable_vram = physical_vram * usable_fraction
For a 24 GB card:
24 * 0.90 = 21.6 GiB usable
The exact reserve depends on the OS, display use, driver, runtime, graph capture, allocator behavior, and other processes. The important part is to stop treating the number on the box as fully available.
3. Add the KV cache
The KV cache is where context length and concurrency become expensive.
A useful planning formula is:
kv_cache_bytes =
2
* layers
* kv_heads
* head_dimension
* context_tokens
* concurrent_sequences
* bytes_per_kv_value
The factor of two stores keys and values.
Take a model with:
- 32 layers
- 8 KV heads
- 128 dimensions per head
- 8,192 cached tokens
- 1 concurrent sequence
- 16-bit KV values, which use 2 bytes
The KV cache is about 1 GiB.
Raise the context to 32,768 tokens and it becomes about 4 GiB. Keep that context and run four concurrent sequences, and it becomes about 16 GiB.
This is why a model can work in a short local chat, then fail when the server uses a larger context window or handles several requests.
Grouped-query attention matters here. Use the number of KV heads, not the total attention head count.
4. Add runtime headroom
I use this planning target:
planning_target = (weight_memory + kv_cache) * (1 + headroom_rate)
A 20 percent headroom rate is a reasonable first estimate when no runtime measurement exists. Replace it with measured data as soon as you can.
Here is an example for a hypothetical 32B model:
- 4-bit weight floor: about 14.9 GiB
- Example 32K KV cache: about 8 GiB
- Subtotal: about 22.9 GiB
- With 20 percent headroom: about 27.5 GiB
A 24 GB GPU with a 21.6 GiB usable budget is short by about 5.9 GiB. The 4-bit model file looked small enough, but the deployment did not.
The architecture values in this example are only a worksheet. Read the actual model configuration before making a hardware decision.
5. Do not assume multiple GPUs add perfectly
Two 24 GB cards do not always behave like one clean 48 GB pool.
Tensor parallelism, pipeline parallelism, layer placement, replicated buffers, interconnect speed, and runtime support all matter. A capacity estimate tells you whether the plan is plausible. It does not prove latency or throughput.
The five checks I use
Before calling a local model deployable, I write down:
- Exact checkpoint size or a clear weight estimate
- Usable VRAM per device
- KV cache from the real architecture and context
- Runtime headroom
- The exact serving runtime and GPU topology
If any one of those is missing, I call the answer a floor, not a deployment plan.
Try the worksheet
I put these formulas into a browser-only LLM GPU memory calculator. It does not upload the values you enter.
The two references I use most often are the Hugging Face model memory estimator guide and the Transformers KV cache guide.
What runtime-specific memory cost has surprised you most: context, concurrency, quantization overhead, or something else?
Disclosure: I used an AI assistant to help edit the structure and wording. I checked the numerical examples against the formulas above.
Top comments (1)
This is the right way to frame local LLM sizing. The part people underprice is that context is not a feature toggle, it is a balance-sheet item. A model that fits at 8K with one user can quietly become a different hardware problem at 32K or with a small queue.
I would add one more line to the worksheet for the failure mode you want to avoid. Occasional OOM during a hobby chat and latency collapse in a small service are not the same loss function, even if the VRAM math starts from the same place.