You pulled the monitor cable, rebooted, and the card that worked yesterday is gone. Three different faults produce that outcome and they have different fixes; the error string tells you which one you have.
The error strings
These are the four you will actually see, and they are not interchangeable:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.
Failed to initialize NVML: Driver/library version mismatch
ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
no compatible GPUs were discovered
The first means the kernel module is not loaded or the device nodes do not exist. The second means a loaded module and a userspace library from different driver versions. The third is what llama.cpp prints when the CUDA runtime enumerates zero devices — usually a consequence of the first two, occasionally a consequence of the card being bound to the wrong kernel driver. The fourth is Ollama’s log line when it finds nothing usable and silently falls back to CPU, which is the worst of the four because it is not an error at all: the model runs, ten times slower, and nothing draws your attention to it.
Note what is not in that list: “no display attached” is not an error the driver emits. The absence of a monitor does not itself stop CUDA working. It changes the conditions under which the driver is initialised, and that is what breaks.
Cause 1: the device nodes never got created
On Linux, the NVIDIA kernel module is loaded and the /dev/nvidia* character devices are created when something first opens the GPU. On a desktop that something is the X or Wayland session, which starts at boot. Remove the display server and nothing opens the card, so the nodes do not exist, so nvidia-smi cannot talk to the driver, so it prints the first message above — and if you run it as root it may work, create the nodes as a side effect, and then appear to have fixed something.
Confirm which half you have:
lsmod | grep -E '^nvidia' # is the module loaded at all?
ls -l /dev/nvidia* # do the device nodes exist?
lspci -k -d ::0300 # which kernel driver is bound to the card?
If lspci reports Kernel driver in use: nouveau, the open-source driver claimed the card and the proprietary one never loaded; blacklist nouveau and rebuild the initramfs. If it reports vfio-pci, the card is reserved for passthrough to a virtual machine and the host cannot use it.
If the module is present but the nodes are missing, the fix is to keep the driver initialised without a display client. That is what NVIDIA documents as driver persistence: the persistence daemon holds the GPU open so the driver stays initialised and the nodes stay present. Enable it as a service — sudo systemctl enable --now nvidia-persistenced — rather than relying on the legacy nvidia-smi -pm 1, which is not persistent across reboots.
The same mechanism is behind a slower symptom people misdiagnose as a model-loading problem. Without persistence, the driver tears down its GPU state whenever the last client exits, and the next process pays the full initialisation cost before it does any work. On a machine that runs a short inference job per request rather than a long-lived server, that cost is paid on every single request.
Cause 2: driver/library version mismatch
Failed to initialize NVML: Driver/library version mismatch means the loaded kernel module and the userspace library are from different driver packages. On a headless server it almost always arrives via unattended upgrades: a package update replaces the userspace libraries and the module files on disk, but the running kernel module cannot be swapped out while it is in use, so the two disagree until a reboot.
Diagnose it by comparing the two directly:
cat /proc/driver/nvidia/version # version of the running module
modinfo nvidia | grep ^version # version of the module on disk
dpkg -l 'nvidia-driver-*' | grep ^ii
A reboot resolves it. If a reboot is not acceptable, the modules can be unloaded and reloaded — nvidia_uvm, nvidia_drm, nvidia_modeset, then nvidia, in that order — but only if nothing holds them open, and the persistence daemon does hold them open, so stop it first. On a machine whose whole purpose is inference, put the driver packages on hold and update them deliberately.
Cause 3: it works, but slowly
The dangerous case is the one with no error. The card initialises, CUDA enumerates it, models load, and the token rate is a fraction of what the card’s bandwidth implies. Two causes, both invisible unless you look.
The first is clock behaviour. Ask the driver directly why the clocks are where they are:
nvidia-smi -q -d PERFORMANCE # Clocks Throttle Reasons block
nvidia-smi -q -d CLOCK # current vs max graphics and memory clocks
Every throttle reason reading Not Active means the clocks are as high as they can be. SW Power Cap means the board power limit is binding. HW Slowdown is the one to worry about: it indicates a power delivery or cooling problem, which on a headless box tucked into a cupboard is entirely plausible — check the supply sizing before blaming the driver.
The second is that the workload is not on the GPU at all. Ollama logging no compatible GPUs were discovered and proceeding on CPU is the archetype, but llama.cpp will do the equivalent if -ngl is unset or if the binary was built without CUDA support. Check the offload count in the load log, and check nvidia-smi shows the process and non-trivial memory in use while it generates. A build with no CUDA in it is a common outcome of installing a prebuilt package on a headless server and never seeing the startup banner.
On Windows there is a third variant worth knowing: a discrete GPU is generally not available to a Remote Desktop session, because the RDP session gets a virtual display adapter rather than the physical card. A process started inside that session may find no CUDA device even though the same process started as a service or through SSH finds it normally. Running the workload as a service, or connecting with something that does not replace the console session, avoids it.
A headless box that stays working
- Enable and start the persistence daemon, and confirm it survives a reboot:
systemctl is-enabled nvidia-persistenced. - Pin the driver package version so an unattended upgrade cannot produce a mismatch between reboots.
- Record the baseline once:
nvidia-smi -q -d CLOCK,PERFORMANCEunder a real load, saved to a file. A future regression is only visible against a baseline. - Assert the GPU is actually being used in whatever starts your server, rather than trusting it. A one-line check of
nvidia-smi --query-compute-apps=pid,used_memory --format=csvafter startup catches the silent CPU fallback. - If the box is otherwise idle, consider a power cap — a headless inference server has no reason to hold peak clocks, and the reasoning is on the PSU page.
Driver package names, service names and the exact behaviour of persistence differ between distributions and between driver branches, and NVIDIA has changed the status of the legacy persistence mode over time. Check the documentation for the branch you have installed.
Top comments (0)