How Much VRAM CUDA Reserves Before Your Model Even Loads

#ai #llm #machinelearning #programming

There is no single number here, and any page that gives you one is guessing. What there is instead is a list of the things that occupy the card before your weights do, and a two-command procedure that turns them into a figure for your machine.

Why this term is missing from every table

There is a second reason this term is missing, beyond nobody having measured it: it does not belong to the model, so it does not fit in a table indexed by model and quant. It belongs to the machine. Two people running the same GGUF on the same card will have different reserves if one of them has a monitor plugged in, and the same person will have a different reserve after a driver update. A number that varies with everything except the thing the table is about is a number that gets dropped, and the arithmetic silently becomes optimistic by however much it was.

Published VRAM tables set the runtime reserve to zero, which is why their numbers look achievable and yours do not. The reserve is not small — on a card that also drives a display it can be a larger allocation than the difference between two quantization levels — and it is not constant, because it depends on your driver version, your CUDA version, whether a monitor is attached, what else is running, and which libraries the runtime initialised.

PyTorch documents the consequence directly: its reference for memory_allocated notes that the value it returns is likely to be less than what nvidia-smi shows, because some memory is held by the caching allocator and some context has to be created on the GPU. The gap between what your framework thinks it is using and what the driver reports is exactly the quantity this page is about.

What is actually resident

The CUDA primary context. Created lazily on the first CUDA call in a process and destroyed when the process exits. It carries driver bookkeeping, the memory manager’s own structures, and any kernel images that have been loaded. Every process that touches the GPU pays for its own — two inference servers on one card pay twice.
Loaded kernel modules. A CUDA binary contains compiled kernels for every architecture it targets, and historically all of them were loaded into context at startup. NVIDIA made lazy module loading the default from CUDA 12 onward, which loads a kernel the first time it is called instead, and reduced this component substantially. If you are on an older toolkit, or have set CUDA_MODULE_LOADING=EAGER, expect it to be larger.
Library workspaces. cuBLAS and cuDNN allocate scratch space when their handles are created, sized by heuristics rather than by your model.
The display. On a card driving a monitor, the framebuffer, the desktop compositor and every hardware-accelerated application hold memory. A browser with several tabs open is a real line item here. This is usually the largest single component on a desktop machine and it is entirely absent on a headless server, which is why the same model fits on one and not the other.
Windows-specific reservation. Under the WDDM display driver model the operating system, not the application, arbitrates GPU memory, and it holds back a share for the desktop. A Windows box and a Linux box with identical hardware will not report the same free memory.

Measuring yours

With nothing loaded, read the baseline: nvidia-smi --query-gpu=memory.used,memory.total --format=csv. On a headless server this should be near zero. On a desktop it will not be, and that number is a permanent deduction from your budget.
See who is holding it: nvidia-smi --query-compute-apps=pid,used_memory --format=csv lists compute processes, but the desktop compositor’s graphics allocation will not appear there. The difference between the total used and the sum of the compute processes is graphics.
Measure the bare context cost. Start a process that initialises CUDA and does nothing else — in Python, import torch; torch.cuda.init() then sleep — and read memory.used again. The increase is the primary context plus whatever modules that framework loaded eagerly.
Load your model with an explicit -c, read the buffer sizes llama.cpp prints, and compare their sum against memory.used for that process. The difference is the reserve for that exact combination of runtime, driver and model.

nvidia-smi reports what the driver has committed, which is not always what is in use — allocators hold freed memory. Read it as an upper bound on availability, which is the direction that matters when you are deciding whether something will fit.

The compute buffer, which is not overhead

One large allocation is often mistaken for CUDA overhead and is not. Every runtime allocates a scratch buffer for the activations of a forward pass, and llama.cpp reports it separately as a compute buffer per device. Its size is set by the widest tensor the graph will evaluate, which scales with the logical batch size — the number of tokens processed in one pass during prefill — and not with the total context.

llama.cpp load output, three separate allocations:

  llm_load_tensors:      CUDA0 model buffer size  =  4685.30 MiB
  llama_kv_cache_init:   CUDA0 KV buffer size     =  1024.00 MiB
  llama_new_context:     CUDA0 compute buffer size =  296.00 MiB

the third is a function of --batch-size / --ubatch-size, not of -c

Which makes it a lever. Lowering the physical batch size shrinks the compute buffer, at the cost of slower prompt processing, and on a card that is a few hundred megabytes short that trade is often the one that makes a configuration load.

There is a related lever with a much larger effect on the same allocation, and one people set for other reasons: flash attention. A naive attention implementation materialises the full score matrix for the batch, which is quadratic in the number of tokens processed at once, and that matrix lives in the compute buffer. An implementation that never materialises it removes the term entirely, so the buffer stops scaling quadratically with the prefill batch. On a long prompt this is the difference between a compute buffer of a few hundred megabytes and one of several gigabytes, which is why the same configuration can fail on one build and load on another with nothing about the model having changed.

None of this affects generation speed, which is bandwidth-bound for a different reason — see why batch size one never saturates a GPU.

Putting it back in the budget

The budget equation used across this cluster is C = (V - W - R) / k, and R is what this page is about. Treat it as three parts rather than one:

R = baseline_before_your_process   (measured at idle: display, other apps)
  + cuda_context                  (measured: a process that only inits CUDA)
  + compute_buffer                (read from the loader's own log)

Once measured on a given machine, the first two are stable across models.
Only the third moves when you change batch settings.

A measured R is worth more than a better estimate of anything else in the equation, because it is the only term the arithmetic cannot derive. Every other quantity — weights, cache per token — follows from published numbers, and those pages are the weights tables and the cache derivation. If a configuration fails to load and the arithmetic said it should fit, R is where the missing gigabyte is. The error itself is covered under CUDA out of memory.