“No Kernel Image Is Available for Execution on the Device”

#ai #debugging #llm #machinelearning

The driver is fine, the GPU is visible, memory is free, and the first kernel launch fails anyway. This error is a contract violation between the code somebody compiled and the silicon you are running it on.

The string

Through PyTorch, and therefore through vLLM, Transformers and most Python inference stacks:

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

Through the ggml stack it appears as a CUDA error attributed to a source location in ggml-cuda.cu, with the same underlying message. A useful early observation: it typically fires on the first real kernel launch, not at initialisation. torch.cuda.is_available() returns true, nvidia-smi is healthy, and the failure arrives the moment actual work is dispatched. That combination — device visible, kernels unlaunchable — is nearly diagnostic on its own.

What is inside a CUDA binary

A compiled CUDA library does not contain machine code for “a GPU”. It contains a fat binary holding, for each architecture the builder chose, either compiled SASS for that exact architecture (sm_XX) or portable PTX intermediate code (compute_XX), or both. At launch, the driver looks for SASS matching the installed device. If it finds none, it looks for PTX it can compile forward. If it finds neither, it reports that no kernel image is available.

So there are exactly two ways to get here: the build targeted architectures that do not include yours, or it targeted them without embedding PTX that could be JIT-compiled to yours. Nothing about drivers, CUDA runtime versions or memory is involved.

The two directions this happens in are both common:

Your GPU is older than the build assumed. Prebuilt wheels drop old architectures over time to keep binary sizes manageable, so an upgrade of the library can break a card that worked last week.
Your GPU is newer than the build assumed. A library compiled before your generation existed has no SASS for it, and PTX forward-compilation only works if PTX was included. This is the annual experience of anyone buying a card in its launch year.

Finding your compute capability

Compute capability is a version number for the instruction set of a GPU architecture, and NVIDIA publishes it per product. Ask the driver rather than looking it up:

nvidia-smi --query-gpu=name,compute_cap --format=csv

# from PyTorch, which also reports what the build supports
python -c "import torch; print(torch.cuda.get_device_capability(), torch.cuda.get_arch_list())"

That second command is the decisive one. get_arch_list() prints the architectures your installed PyTorch was compiled for. If your device’s capability does not appear in that list, the diagnosis is complete and no further investigation is needed.

Two conventions trip people up when reading the output. The capability is written as a major and minor pair — (8, 9) or 8.9 — while build flags want it with the dot removed, as sm_89 or the bare integer 89. And the minor version is not a compatibility suffix: sm_86 and sm_89 are different targets, and a binary built for one does not automatically run on the other even though both are 8.x. Within a major version the driver will sometimes accept a lower minor, but the guarantee is weak enough that the right habit is to name your exact value.

On a multi-GPU box, check every device rather than the first. A machine with two generations installed will report whichever card the query landed on, and a build that satisfies one can fail on the other — which presents as an intermittent error that follows device placement rather than load.

NVIDIA maintains the authoritative mapping from product to compute capability on its CUDA GPUs page. It gains entries with every generation, so read it rather than a table copied into a blog post.

Why PTX sometimes saves you and sometimes does not

PTX is forward-compatible only. The driver can JIT PTX for an architecture newer than the one it was emitted for; it cannot synthesise code for an older one, because the newer PTX may use features the old hardware lacks. That asymmetry explains the observed behaviour:

A build with PTX for a recent architecture will usually run on the next generation after a one-off JIT delay at first launch.
A build for recent architectures will never run on an older card, however much PTX it contains.
JIT results are cached, so the first launch is slow and subsequent ones are not. A mysterious multi-second first inference on new hardware is often this.

When you are on newer hardware than the build, forcing the driver to prefer JIT compilation can be enough:

CUDA_FORCE_PTX_JIT=1 python -c "import torch; torch.zeros(8, device='cuda').sum()"

If that works, you have confirmed the diagnosis and bought yourself time. It is not a permanent fix — a build with real SASS for your architecture will be faster.

Getting a binary that matches

Install a build that lists your architecture. For PyTorch, that means matching the CUDA variant to your hardware generation and preferring a recent release on new silicon. Verify with get_arch_list() after installing, not before.
If you compile, name the architecture explicitly. Never rely on the default. For a CMake project such as llama.cpp:
```
# example: 8.9 is Ada; substitute the value nvidia-smi reported
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j
```
For PyTorch extensions and anything built through its toolchain, the equivalent is the architecture list environment variable:
```
TORCH_CUDA_ARCH_LIST="8.9" pip install --no-build-isolation -e .
```
Rebuild every compiled extension, not just the top package. Quantization kernels, attention kernels and custom operators are separately compiled artefacts, and one stale wheel among them reproduces the error even after the main library is correct. This is the usual reason a reinstall “did not work”.
Fix the Docker case at the image level. A build inside a container has no GPU present, so architecture auto-detection cannot work — it must be stated. Use a devel base image if you compile in it, and pass the architecture list as a build argument.
Clear stale caches. Remove build directories and editable installs between attempts. Partially rebuilt trees are why correct flags appear to have no effect.

Two errors that look adjacent and are not: if CUDA is not present in the build at all you get Torch not compiled with CUDA enabled, and if kernels launch but fail on a bad index you get a device-side assert. Only this one is about the compile target. If the compile itself is what fails, see the llama-cpp-python wheel build failure.