Pascal Quadro P600 SIGILL crash-loops: why Immich v3 ML died and I gave the card to NVENC

I have a Nvidia Quadro P600 hanging off a small single-board NAS on a mining riser. It is a 2 GB, 40 W, low-profile Pascal card (GP107) with no external power connector. The plan was simple: give Immich's machine learning a GPU. The plan did not survive contact with Immich v3, and the card has ended up doing a different job.

The hardware, briefly

The board's only usable slot is PCIe 2.0 x4 with a 25 W slot power limit. A 40 W card cannot run from that, so the P600 sits on a VER009S USB3-cable riser powered by a separate ATX supply. That riser is electrically x1 despite its x16 slot, so the best case is PCIe 2.0 x1, roughly 500 MB/s.

The NAS OS ships Nvidia's proprietary driver, 580.105.08, plus the container toolkit. That matters because 580 is the last driver branch that supports Maxwell, Pascal and Volta. The OS has no kernel build tree, so you get the driver it ships and nothing else.

Two months of the card not really working

In July the PCIe link never trained. lspci did not show the card at all and the root port read Width x0. No driver can fix that, because a driver only binds to a device that has already enumerated. The rest was physical work: reseating it, and my suspicion of a ground offset between two power supplies (the NAS on its own unearthed brick, the card on an earthed ATX supply, joined only by thin cable grounds).

On 4 August the card finally enumerated, the driver bound, and /dev/nvidia* appeared. Eight and a half hours after boot, dmesg said:

NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

The link had also negotiated itself down to 2.5 GT/s x1, which is below the riser's own ceiling, and the root port was logging correctable and non-fatal errors. Xid 79 is the textbook signature of power or signal-integrity trouble. The fix order I wrote down then still stands: swap the riser's USB3 cable, get both ends onto one ground domain, then reseat. The NAS lives in the loft, so none of that has happened yet.

The correction: a working GPU would have made Immich worse

On 5 August I found the fact that reshaped the whole project. Immich v3 dropped machine-learning support for Maxwell and Pascal. ONNX Runtime no longer ships kernels for compute capability 5.2 or 6.1, and the P600 is compute 6.1. On those cards the -cuda ML container does not print a polite message. It dies immediately with exit code 132, SIGILL.

At the time my ML container was configured correctly for CUDA and was falling back quietly:

CUDA failure 100: no CUDA-capable device is detected
Falling back to ['CPUExecutionProvider']

Smart search and face recognition still worked, just on CPU. That fallback only happened because the card was dead. With a dead card, Immich degrades to CPU. With a live card, the v3 -cuda image would crash-loop forever. So fixing the riser without changing the image would have taken ML from slow to broken.

I saw two honest options. One was to pin the ML container to v2.4.1-cuda, the last release that supports Pascal, and freeze that component while the server moves on. The other was to leave ML on CPU and use the card for NVENC, the fixed-function video encoder, which does not depend on CUDA kernels at all.

The same cliff turns up one layer down too. cuDNN dropped the generation at 9.11 (9.10.2 is the last usable), and the driver branch is the last one. Three vendors retiring the generation within a year is the real reason a cheap used Quadro is a worse deal than it looks. I wrote up the hardware side in a public repo: zimablade-gpu-immich.

24 September: the card came back on its own

A power cut on 24 September gave the NAS a true cold boot. Afterwards the P600 enumerated, the driver bound, and nvidia-smi answered: 32 °C, 3 MiB used, the link still at gen1 x1, and no Xid.

I nearly made a mess first. I had seen nvidia-persistenced with a restart counter of 85,467 and masked the service as a crash loop. Those restarts were from the previous boot, when the card was dead and /dev/nvidia0 did not exist. On the current boot the card was fine. I unmasked and re-enabled it within the same hour, and persistence mode now reads Enabled. The lesson for next time: read journalctl -b 0 before fixing anything, not -b -1.

Drift that would have crash-looped on the next re-apply

While checking what was actually using the GPU, I found the answer was nothing. The ML container was running the plain CPU image, v3.0.1, which is correct given the Pascal cliff. But the compose file on disk still said v3.0.1-cuda with an Nvidia device reservation.

With a dead card that difference had not mattered. With a live card, any re-apply from the NAS's app manager or a compose up of the ML service would have switched it to -cuda, and that means SIGILL and a crash loop. The running container and the file describing it had drifted apart, and the file was the dangerous one.

Giving the card the job it can still do

immich-server had no GPU at all: the default runc runtime, no device request, and only /dev/dri for the Intel iGPU. Immich's ffmpeg settings held a single value, a 2160 target resolution, with hardware acceleration disabled.

I staged one script and applied it once I had the go-ahead. It does four things:

1. Gives immich-server an Nvidia device request with the capabilities [gpu, compute, video]. video is the one that exposes NVENC.
2. Rewrites the ML service in the compose file to the CPU image with no GPU, so the file matches what is running.
3. Sets the ffmpeg config to accel: nvenc with accelDecode: false, written to the system-config row in Immich's database.
4. Recreates only immich-server with --no-deps, which took about a minute, then runs test encodes inside it.

Both h264_nvenc and hevc_nvenc encoded successfully inside the server container. The server came back healthy, the local ping returned 200, and I checked the public photos hostname through the Cloudflare edge as well.

One harmless failure: the script's last check ran nvidia-smi inside the container, and it is not there. The binary comes with the utility driver capability, which I had not requested. Under set -e the script stopped at that line, after everything had already been applied. If I ever want the tool in the container, I will add utility. NVENC does not need it.

What happens when the card falls off again

This is the part that makes the setup reasonable, not just hopeful. Immich retries a failed hardware transcode in software. If the P600 drops off the bus again with Xid 79, video transcoding falls back to CPU and uploads keep working. That is the opposite of what the ML container would do in the same situation.

Measuring instead of guessing

The last time the card lived, it died 8.5 hours after boot. The power problem has since been fixed, but the riser cable and the ground domains have not changed, so I do not know if it will last.

So there is now a small watch job. Every 15 minutes it records whether the card is alive or dead, along with the NAS's uptime. It posts to Discord only when the state changes. If the NAS itself cannot be reached, that run makes no judgement, because "can't reach the box" is not the same as "the GPU died". The log looks like this:

2026-09-24 16:30:02 alive uptime=6.3h Quadro P600, 32, 1
2026-09-24 16:45:01 alive uptime=6.5h Quadro P600, 31, 1

At the time of writing it has been alive for 6.5 hours, two hours short of where it failed last time. The next few runs will tell me whether the power fix was the whole story, or whether I still need to go up into the loft with a new USB3 cable.

What I would tell someone buying a Pascal card for Immich

• Check the compute capability against your Immich version before you buy. On v3, compute 6.1 means a -cuda ML container that SIGILLs, not one that runs slowly.
• A working old GPU can be worse than a missing one, because the fallback path only runs when there is no device.
• Compare the compose file with docker inspect, not with your memory. Mine said -cuda long after the running container had stopped using it.
• NVENC is a separate, fixed-function block. The card that Immich's ML dropped can still encode video, and Immich already knows how to fall back if it cannot.

---

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.