CUDA Out of Memory the Instant a Model Starts Loading

#ai #debugging #llm #machinelearning

An OOM that arrives seconds after you press enter is a different bug from an OOM that arrives after an hour of traffic. The first is a budgeting mistake you made before the process started; the second is a workload that grew. This page is about the first.

The string

In the ggml stack — llama.cpp, and Ollama which embeds it — the failure prints as a pair, the allocator line and the loader line:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4300.00 MiB on device 0: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_model_load_from_file_impl: failed to load model

In the PyTorch stack — vLLM, Transformers, ExLlama, anything that loads a checkpoint into torch — it prints as an exception:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB.
GPU 0 has a total capacity of 23.65 GiB of which 18.06 MiB is free.

Both name a size. That size, and where in the sequence it appears, is the whole diagnosis.

Why it fails at load and not later

Weight loading is the largest single allocation any inference process makes, and it is made once, at the start, in one shot per device. If you get past it, you have proven the weights fit; everything allocated afterwards is comparatively small and incremental. So the timing is informative:

Immediately, during tensor loading — the weights do not fit in the space you gave them. This is a configuration error, and it is fully deterministic: it will fail the same way every time.
Immediately after loading, during context init — the weights fit but the scratch space did not. That is a different page: the compute buffer allocation failure.
Minutes or hours in, under traffic — the weights and the initial cache fit but something grows: concurrent sequences, longer prompts, a cache that is not being reclaimed. The general treatment of that is in the CUDA out-of-memory page.

A related and very common variation: it fails at load only after something else has taken the card. A leftover Python process, a display server, a previous run that did not exit. Check before you change any settings:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

If that lists a process you forgot about, you have your answer and none of the rest of this page applies.

What -ngl actually promises

In llama.cpp, -ngl (--n-gpu-layers) is the number of transformer layers to place on the GPU. In Ollama the same setting is exposed as num_gpu. It is worth being precise about what it does and does not do:

It is a request, not a budget. Asking for more layers than fit does not cause a graceful fallback to CPU; it causescudaMalloc to fail partway through and the load to abort.
The conventional -ngl 99 means “all of them” and is fine on a card that can hold the model. On a card that cannot, it is precisely the setting that produces this error.
Layers are not the only thing on the card. The token-embedding and output tensors, the KV cache and the compute buffers all want VRAM too, and on a model with a large vocabulary the output tensor is substantial on its own.

The second frequent cause is multi-GPU splitting that did not happen. If a machine has two cards and everything is landing on CUDA0, the split never took effect — check --tensor-split and, in Ollama, that the scheduler enumerated both devices — because an unsplit load onto one card is exactly the same failure as asking for too many layers.

Budgeting the card

You can estimate the weight footprint before you try, from numbers you can look up rather than guess. Weight bytes are approximately parameters times bits-per-weight divided by eight. A 70B checkpoint at a roughly 4.8-bit average quantization comes out at:

70e9 params x 4.8 bits / 8 = 42e9 bytes ~= 39 GiB of weights, before anything else

Then add the KV cache for the context you want, using the model’s published layer count and key/value head count, and a gigabyte or so of compute buffers. The bits-per-weight figure is a property of the quantization scheme and is documented per format; the parameter count is on the model card. Neither is a benchmark, and both are checkable.

The advertised VRAM on a card is not all usable. The display, the driver’s own allocations and fragmentation take a slice, so treat the number in nvidia-smi’s free column as the real ceiling rather than the marketing figure. For how much card is worth buying, see consumer GPUs for local models.

Getting to a configuration that loads

Confirm the card is empty. Run the nvidia-smi query above and kill anything unexpected. Repeat after a failed load — an aborted process sometimes holds memory briefly.
Load with zero offload once. -ngl 0 proves the file itself is good and the failure is a placement problem. If it still fails, the problem is system RAM, not VRAM.
Walk the offload up, not down. Try a value well below the layer count, confirm it loads, then increase. llama.cpp prints how many layers it placed and the per-device buffer sizes on success, which tells you exactly how much room the next increment needs.
Shrink the context before you shrink the model. The KV cache is often larger than the gap you are trying to close, and -c is a cheaper concession than a coarser quantization.
Then, if still short, drop a quantization level. Redo the arithmetic above at the new bits-per-weight to check it will actually close the gap; choosing a quantization level covers what you give up.

One thing not to do: setting an environment variable that makes the allocator more permissive does not create memory. If the weights do not fit, they do not fit, and the honest move is fewer layers on the card or a smaller file.

A card that cannot hold the model you want is a fixed constraint until you buy a different card, so the usual production answer is a split: the quantized local model for the requests it handles well, a hosted model for the ones it cannot fit. That means one code path talking to two backends with different auth, different streaming formats and different failure modes — the boundary a gateway exists to absorb.