torch.cuda.OutOfMemoryError: CUDA out of memory is not one error. The numbers PyTorch prints after that sentence distinguish three distinct failures — a single allocation that is too large, a device that is genuinely full, and an allocator that has enough free memory but not in one contiguous piece — and each of the three has a different fix.
The anatomy of the message
The wording has changed across PyTorch versions, but since roughly 2.1 the message carries four figures. A representative one:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 23.64 GiB of which 1.18 GiB is free.
Of the allocated memory 19.90 GiB is allocated by PyTorch, and 1.42 GiB is
reserved by PyTorch but unallocated. If reserved but unallocated memory is
large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
| Figure | Description |
|---|---|
| Tried to allocate | The size of the single allocation that failed. Not your total usage — one tensor, one buffer, one workspace. |
| Total capacity | The physical device memory, minus a little the driver keeps. A 24 GB card reports about 23.6 GiB. |
| Free | What the driver still has available to hand out to anyone, including other processes on the same card. |
| Allocated by PyTorch | Live tensors. Weights, activations, gradients, optimiser state, KV cache — everything currently referenced. |
| Reserved but unallocated | Memory PyTorch already took from the driver and is holding in its caching allocator, but which no live tensor occupies. This is the fragmentation number. |
What the numbers tell you to do
Read them in this order and stop at the first that matches.
- Is “tried to allocate” enormous? If a single allocation is several gigabytes on a card with a few gigabytes free, one tensor is the problem, not the sum of many. Attention scratch space and logits over the full vocabulary are the usual candidates, and both scale with batch size times sequence length. Halve one of those two and the allocation halves.
- Is “reserved but unallocated” larger than “tried to allocate”? Then you are not out of memory, you are fragmented: the allocator holds enough total space but no single contiguous block big enough. This happens with variable-length sequences, which is exactly what inference serving is. Set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truein the environment before the process starts — it lets the allocator grow segments rather than needing one contiguous run, and on variable-length workloads it can recover several gigabytes for one environment variable. Bucketing sequence lengths so allocations repeat at the same sizes has the same effect more laboriously. - Is “free” near zero and “allocated” near capacity? This is the honest case: the work does not fit. Go to the arithmetic below and decide what to shrink.
- Is “allocated by PyTorch” much smaller than capacity minus free? Something else is on the card — another process, a zombie, a display server, or a second copy of your own worker. See the last section.
Where the memory actually went
You can compute the requirement rather than discovering it by crashing, and the arithmetic is short enough to do on paper. Every figure below is derived, not quoted; substitute your own model’s numbers from its config.json.
Weights
Parameters times bytes per parameter. FP16 and BF16 are 2 bytes, FP32 is 4, 8-bit is 1, 4-bit is roughly 0.5 plus a small overhead for scales and zero points.
8e9 params x 2 bytes = 16.0 GB (bf16)
8e9 params x 1 byte = 8.0 GB (int8)
8e9 params x 0.5 byte = 4.0 GB (4-bit, before quantisation overhead)
A 24 GB card holds an 8B model in bf16 with 8 GB to spare, and does not hold a 70B model in bf16 at all — 140 GB of weights before anything else exists.
KV cache, which is the part people forget
Every token in the context stores a key and a value vector in every layer, for the whole time the sequence is alive. Per token:
bytes_per_token = 2 (K and V)
x layers
x kv_heads # NOT attention heads, if the model uses GQA
x head_dim
x bytes_per_element
Worked, for a model with 32 layers, 8 KV heads, head_dim 128, bf16:
2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token
8,192-token context -> 1.0 GiB per sequence
32,768-token context -> 4.0 GiB per sequence
8 concurrent sequences at 8k -> 8.0 GiB
That last line is the one that ends serving experiments. The weights fit, one request fits, and the eighth concurrent request does not. Note the multiplier that grouped-query attention buys you: the same model with 32 KV heads instead of 8 would need 512 KiB per token, four times as much. There is more on the mechanism in how the KV cache works and on sizing a machine in VRAM requirements.
Training adds three more copies
Full fine-tuning with Adam holds, per parameter: the weights, the gradient, and two optimiser moments — usually with an FP32 master copy as well. In mixed precision that is commonly quoted as about 16 bytes per parameter before activations:
2 bytes bf16 weights
2 bytes bf16 gradients
4 bytes fp32 master weights
4 bytes Adam first moment
4 bytes Adam second moment
---------
16 bytes per parameter
8B parameters x 16 = 128 GB, before a single activation.
Which is why full fine-tuning of an 8B model does not happen on one consumer card and LoRA does: training only the adapters removes the gradient and both moments for 99% of the parameters.
Fixes for inference
- Reduce max concurrent sequences or max context. Both multiply the KV term directly, and serving engines expose them as explicit limits rather than leaving them to chance. This is the first lever because it is the one with a known, computable effect.
- Quantise the weights. 8-bit halves the weight term, 4-bit quarters it, with a quality cost that depends on the method and the model — choosing a quantisation covers the trade. Note that this does nothing to the KV cache, so a long-context workload can still fail after quantising.
- Quantise the KV cache. Where the serving engine supports 8-bit KV, it halves the term that scales with context and concurrency, which is the one that grows.
- Set
expandable_segments:True. Free, and specifically effective for the variable-length allocation pattern that serving produces. - Check you are not holding a second copy. Loading a checkpoint to CPU and then moving it to GPU can transiently need both; loading directly to the device, or with a low-CPU-memory loading path, avoids the spike.
Fixes for training and fine-tuning
- Lower the per-device batch size and raise gradient accumulation to match. The effective batch size is unchanged, the memory is not. This is the first thing to try and it costs only wall-clock time.
- Shorten the maximum sequence length. Activation memory scales with it, and with attention implementations that materialise the score matrix it scales with the square. Check what fraction of your examples actually need the length you set — it is often a small tail.
- Turn on gradient checkpointing. Recomputes activations in the backward pass instead of storing them. A large memory saving for roughly 20–30% more compute, and it is a single flag in most trainers.
- Switch to LoRA or QLoRA. Removes optimiser state and gradients for the frozen weights, which per the arithmetic above is most of the requirement.
- Use a memory-efficient attention kernel. Anything in the FlashAttention family avoids materialising the full attention matrix, which removes the quadratic activation term.
- Offload optimiser state to CPU, last. It works and it is slow, so it belongs after the options that cost nothing.
When the GPU is full and nothing is running
If nvidia-smi shows most of the memory in use and you believe nothing is running, one of these is true:
nvidia-smi # who holds memory, by PID
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
ps -o pid,user,cmd -p <PID> # what that PID actually is
- A crashed process that has not been reaped. A Python process killed while a CUDA context was live can leave memory held until the process is really gone. Kill the PID from the list above, not the terminal it was started in.
- A notebook kernel. Jupyter keeps the kernel and therefore the tensors alive after the cell finishes and after you close the tab. Restart the kernel;
del modelalone does not free memory that a traceback or an output cell still references. - Two workers on one card. A server started with two workers loads the model twice. This shows up as exactly double the expected allocation, which makes it easy to spot once you have computed the expected figure.
- A display server or another user. On a shared or desktop machine, a couple of gigabytes can be gone before you start.
Inside a Python process, torch.cuda.empty_cache() returns cached-but-unallocated memory to the driver. It does not free live tensors and it will not rescue a genuine shortfall — if it appears to fix your OOM, what you actually had was fragmentation, and expandable_segments is the better answer.
Top comments (0)