“CUDA Error: Device-Side Assert Triggered” Running a Local Model

#ai #debugging #llm #machinelearning

RuntimeError: CUDA error: device-side assert triggered means a kernel called assert() on the GPU. The message does not say which kernel, which index, or which line of your code caused it — and the traceback usually points somewhere innocent.

The string, and why it is useless as printed

The full text is usually followed by advice to compile with TORCH_USE_CUDA_DSA, and by the note that CUDA kernel errors might be reported asynchronously at some other API call. That note is the important part and it explains everything confusing about this error.

CUDA launches are asynchronous. The host queues a kernel and moves on without waiting for it. When a kernel trips an assertion, the failure is recorded on the device, and the host only learns about it at the next synchronising call — a .item(), a .cpu(), a print, the next allocation. So the Python traceback points at whichever line happened to synchronise, which is typically several operations after the one that was actually wrong. People spend hours studying a line that is merely the first to ask.

Worse, once a device-side assert fires the CUDA context is corrupt. Every subsequent CUDA call in that process fails, often with different errors, and no try/except recovers it. The process must be restarted. If you are running this inside a server, that is why one bad request appears to poison every request after it.

Making the error name its own cause

Two settings turn an unlocatable error into a specific one, and they are the first thing to do, before changing any code:

# 1. Make launches synchronous, so the traceback points at the real line.
CUDA_LAUNCH_BLOCKING=1 python run.py

# 2. Or run the same input on CPU, where asserts become readable exceptions.
CUDA_VISIBLE_DEVICES="" python run.py

CUDA_LAUNCH_BLOCKING=1 forces the host to wait for each kernel, so the error is raised at the operation that caused it. It makes everything slower and is a debugging setting, never a production one. The CPU run is often even better: PyTorch’s CPU kernels raise a Python IndexError with the offending index and the tensor size in the message, which is the whole answer in one line.

With blocking enabled, the stderr above the Python traceback will contain the device assertion itself. In the embedding case it names the kernel and the failed condition, typically something of the form indexSelectLargeIndex ... Assertion `srcIndex < srcSelectDimSize` failed. That is the string to search on; the outer device-side assert triggered is not.

The usual cause: an ID outside the embedding table

In local LLM work, the overwhelmingly common assert is a bounds check in an index-select kernel, which is what an embedding lookup compiles to. The embedding matrix has one row per vocabulary entry. Hand it a token ID equal to or greater than that row count and the kernel asserts.

The mismatch has a small number of sources, and they are worth enumerating because the fix differs for each:

A tokenizer with more tokens than the model has rows. Adding special tokens — a pad token, tool-call markers, chat-template sentinels — grows the tokenizer without growing the model. Any added ID is out of range until model.resize_token_embeddings(len(tokenizer)) is called. Compare len(tokenizer) against model.get_input_embeddings().weight.shape[0]; if the first is larger, this is your bug.
A pad token borrowed from another checkpoint. Setting tokenizer.pad_token_id to a number copied from a different model is the classic version. Use tokenizer.pad_token = tokenizer.eos_token so the ID is guaranteed to be in range.
A tokenizer and a checkpoint from different repositories. Fine-tunes that extend the vocabulary ship their own tokenizer; using the base model’s tokenizer with the fine-tune’s weights, or the reverse, produces IDs the other side has never seen.
Labels rather than inputs. During training, a loss with ignore_index=-100 is fine, but any other negative or over-range label reaches the same kernel. If the assert appears only on the backward-adjacent step, look at labels, not inputs.

The check that settles it takes one line, run on CPU before any GPU work:

ids = tokenizer(text, return_tensors="pt").input_ids
vocab_rows = model.get_input_embeddings().weight.shape[0]
print(ids.max().item(), ids.min().item(), vocab_rows, len(tokenizer))
assert ids.max().item() < vocab_rows and ids.min().item() >= 0

The other asserts that reach the same message

Not every device-side assert is an embedding index. Three others are common enough to recognise:

A class index out of range in a loss. nll_loss and cross_entropy assert that every target is below the number of classes. In LLM training the number of classes is the vocabulary size, so this is the same mismatch arriving through a different door.
A gather or scatter with a bad index. Sampling code that builds indices by hand — top-k filtering, beam bookkeeping, speculative-decoding acceptance — can produce an index off the end after an edge case, such as a top-k larger than the surviving candidate set.
An attention mask of the wrong dtype or shape. Masks that are meant to be boolean but arrive as integers, or that are broadcast wrongly across heads, can drive indexing code out of range further down.

What is almost never the cause, despite frequent claims: running out of memory. That produces a distinct message about an unsuccessful memory allocation, covered in the CUDA out-of-memory page. A card too old for the compiled architectures produces a no-kernel-image error instead. If you are seeing one of those, this page is the wrong one.

Fixing it properly

Reproduce on CPU with the same input. Read the IndexError, which names the index and the bound.
If it only reproduces on GPU, rerun with CUDA_LAUNCH_BLOCKING=1 and read the assertion text above the traceback, not the traceback itself.
Compare len(tokenizer) with the embedding row count. If they differ, either resize the embeddings or load the tokenizer that belongs to the checkpoint.
Add a bounds guard at the boundary where text becomes IDs, so a bad input raises a readable Python error instead of killing the CUDA context for the whole process.
Restart the process. The context does not recover, so any verification done after the assert in the same process is meaningless.

The structural fix in a serving context is to validate token IDs at the edge. One request with an out-of-range ID takes down every request sharing the process, so the cheap bounds check before the forward pass buys you an error response instead of an outage.