llama.cpp: “failed to allocate compute buffers”

#ai #debugging #llm #machinelearning

The weights loaded. The layers were offloaded. Then the context initialisation died on a buffer nobody talks about, and reducing the model size barely helps.

The string, and the line above it

Current llama.cpp prints the failure from context initialisation, and recent builds distinguish the prompt-processing buffer from the token-generation one:

llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers

Older builds emit the same idea from llama_new_context_with_model without the pp/tg distinction. Either way, the line immediately above it is the one that identifies the cause, because the allocator names the backend and the size it asked for:

ggml_vulkan: Device memory allocation of size 4764745728 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers

On CUDA the equivalent pair names cudaMalloc; on Metal it names ggml_backend_metal_buffer_type_alloc_buffer. Read the size in that line and hold on to it — it is the number every fix below is trying to reduce.

What a compute buffer is

There are three separate pools of memory in a llama.cpp session, and conflating them is why this error gets misdiagnosed.

The weight buffers hold the model. Their size is fixed by the file and by how many layers you offloaded. They are allocated first, and if they fail, you never reach this message.
The KV cache holds keys and values for every token in the context. Its size is set by context length and cache precision, and it is allocated with the context.
The compute buffers hold the intermediate tensors of a forward pass: activations, attention scores, and the output logits. They are scratch space, reused every pass, and their size is set almost entirely by how many tokens you process at once.

The graph allocator reserves the compute buffers up front by running a dry pass over the computation graph at the maximum batch you configured, so it knows the worst case before serving a single token. That reservation is what fails here. It is also why the failure is deterministic: it happens on the reserve, not hours later under load.

Why batch size drives the size

One term dominates on modern models, and it is easy to compute. The final projection produces one score per vocabulary entry per token in the batch. Llama 3’s published vocabulary is 128,256 tokens, and llama.cpp’s default physical batch is 512, so the logits tensor alone is:

128,256 vocab x 512 tokens x 4 bytes (fp32) = 262,668,288 bytes ~= 250 MiB

That is one tensor, for one step, and it does not shrink when you quantize the weights — logits are computed in float regardless. Add the attention score tensors, which for a batch of n tokens against a context of c tokens scale with n × c per head, and you have a buffer that grows with batch and with context simultaneously. On a model with a 256K vocabulary the logits term doubles again.

This is why a user with 48 GB of VRAM per card can offload 22 GB of weights successfully and then fail on a 4.7 GB compute buffer: the weights fit, the scratch space for the batch did not, and no amount of re-quantizing the weights changes the scratch space by much.

It also explains the asymmetry between prompt processing and token generation, which is what the pp and tg in the modern message refer to. Generation runs one token at a time, so its compute buffer is sized for a batch of one and is small. Prefill runs a whole micro-batch at once, so its buffer is hundreds of times larger. A server that must reserve both reserves the large one, and a configuration that would generate happily forever still fails before it reads a prompt. If the message names pp specifically, the problem is entirely on the prefill side and -ub is the correct dial.

One consequence that surprises people: multi-GPU does not divide this cost the way it divides weights. Weights split across devices, so two cards hold half each. The compute buffers are allocated per device because each device runs its share of the graph, so adding a second card does not halve the buffer — it adds a second one. That is why the two-card report above failed at roughly 4.7 GB per device rather than at half of that.

Telling it apart from a weights OOM

Three tells, any one of which is decisive:

Where in the log it happens. A weights failure appears during llama_model_load and mentions loading tensors. This one appears after the model has finished loading and the layer offload summary has printed, during context initialisation.
Whether -b and -ub change anything. Halving the physical batch roughly halves the compute buffers and does nothing at all to the weights. If the reported allocation size moves when you change -ub, it is a compute buffer.
Whether the size is round relative to the model. Weight buffers come out near the file size. Compute buffers come out at a few hundred megabytes to a few gigabytes regardless of a 4 GB or 40 GB model.

The flags that move it

Reduce the physical batch first. -ub (the micro-batch, default 512) is the one the compute buffers scale on; -b is the logical batch above it and should be at least as large. Both together:
```
llama-server -m model.gguf -ngl 99 -c 8192 -b 512 -ub 128
```
Dropping the micro-batch slows prompt processing, because prefill is the part that benefits from batching. It does not slow token generation, which is one token at a time anyway.
Reduce the context. -c shrinks the KV cache and the attention-score part of the compute buffer at the same time. If your prompts are short, a smaller -c is free.
Move the pressure off the constrained device. Lowering -ngl keeps some layers on CPU, which frees VRAM for the buffers that remain. In the extreme, -ngl 0 runs everything on CPU and the GPU allocator stops being involved — slow, but it isolates the cause conclusively.
Check for a per-allocation limit, not a capacity limit. The Vulkan message above says the request exceeded the device’s maximum allocation size, which is a different constraint from running out of memory and can trip with plenty of VRAM free. When a backend reports that, the fix is a smaller single buffer — i.e. a smaller -ub — not more memory.

llama.cpp renamed its build and runtime options from the LLAMA_ prefix to GGML_ during 2024, and the batch flags have gained and lost aliases since. Check llama-server --help on the binary you actually have rather than copying flags from an old thread.

If the crash is an assertion rather than an allocation failure, it is a different problem — GGML_ASSERT crashes have their own causes. And if the model never got as far as printing its offload summary, start from the load-time CUDA allocation failure instead. Background on the runtime itself is in the llama.cpp guide.