A dead GPU degrades gracefully. A working one crash-loops.

I put a used £30 Quadro P600 on a riser into a ZimaBlade, to accelerate machine
learning in Immich. It took months of intermittent debugging to get the PCIe link to
train.

Then I found out that fixing it would have broken the thing I was trying to speed up.

This is the write-up of both halves: the diagnostic method, which is reusable, and the
research I should have done first, which is the actual lesson.

The constraints nobody prints on the box

The ZimaBlade has a PCIe slot. What it does not have is a slot you can just use. From
lspci -vv on the root port:

LnkCap:  Speed 5GT/s, Width x4        # PCIe 2.0 x4, ~2GB/s ceiling
SltCap:  PowerLimit 25W               # slot budget
         HotPlug-                     # enumeration only at cold boot

Three things fall straight out of that:

• 25W slot budget. The P600 is a 40W card with no external power connector. It physically cannot be slot-powered here. You need a riser with its own 12V feed.
• No hot-plug. echo 1 > /sys/bus/pci/rescan is theatre. If the card wasn't there at cold boot, it isn't there.
• No Above-4G decoding. Every PCI window is sub-4GB, the largest about 1.25GB. Fine for a 2GB card with a ~256MB BAR aperture; a hard blocker for anything with real VRAM.

Also worth stating plainly, because the mining-riser marketing obscures it: a VER009S
riser is electrically x1, despite having an x16 slot on it. The USB3 cable is just the
physical medium, not the USB protocol. So the ceiling was PCIe 2.0 x1, roughly 500MB/s —
before anything went wrong.

And the stock ZimaBlade PSU is 12V/3A, 36W over USB-C. Board, RAM and two SATA disks
already eat 25–30W of that. There was never a version of this where the card ran off the
onboard supply.

The diagnostic worth stealing: presence detect

For a long time the link simply never trained — LnkSta: Width x0. The question that
matters when you have a card, a riser, a cable, a second PSU and a slot all in series is:
which end is at fault? You do not want to be swapping parts at random in a loft.

PCIe gives you a free answer. On every card, pins A1 (PRSNT1#) and B17 (PRSNT2#)
are shorted together on the card's own PCB. The host grounds A1 and pulls B17 high.
Seat a card, B17 gets pulled low through that short, and the slot reports PresDet+.

The useful property: this is pure DC continuity. It does not involve the data lanes, the
cable, the GPU, or the external supply. So:

SltSta: PresDet-   →  nothing is making contact in the slot. Stop.
                      Cable and ground theories are downstream and cannot be the blocker.
SltSta: PresDet+   →  the board is seated. The fault is further out.

Mine was flapping +/-, then went steady - — which told me the riser board was not
seated, and saved me from chasing driver and cable fixes for a mechanical problem.

I then over-read my own test, and it is worth admitting because it is the more useful
half. After reseating, PresDet read a solid + ten times running — and I briefly took
that as validating the whole chain. It doesn't. That A1↔B17 short is on the x1 board and
shorts two of the host's own pins inside the host's own ground domain. It reads + with
no cable and no GPU attached at all. It proves exactly one thing: that small board is in
the slot.

A test that can only pass tells you nothing. Know the narrowest claim your evidence
supports.

Two related rules that saved time:

• Driver ≠ detection. lspci enumerates with no driver loaded; a driver only binds to an already-enumerated device. No driver install can make an undetected card appear. Don't debug drivers while the width is 0.
• A spinning fan proves 12V and nothing else. It says nothing about data lanes.

It worked. Then it died 8.5 hours later.

The physical work eventually paid off. The card enumerated, the link trained, the
proprietary driver bound cleanly at boot, all four /dev/nvidia* nodes appeared.

Then, 30,709 seconds after boot:

NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from
      0x0 (None) to 0x1 (GPU Reset Required)

Xid 79 is textbook power-delivery or PCIe signal integrity. The corroborating evidence
was already in lspci:

LnkSta:  Speed 2.5GT/s (downgraded), Width x1 (downgraded)
DevSta:  CorrErr+ NonFatalErr+ UnsupReq+

Negotiated down to PCIe 1.0 x1 — below even the riser's own x1-at-5GT/s ceiling — and
the root port actively logging correctable errors. A link that trains but negotiates below
the worst thing in the chain is telling you the physical layer is marginal, not that it
works.

The prime suspect was a ground offset: the ZimaBlade on a floating two-pin 12V brick, the
GPU on an earthed ATX supply, the two domains bonded only through the thin ground wires in
a USB3 cable. PCIe receivers have a narrow common-mode range. That single theory predicts
all three observations — presence detect fine (internal domain), fan spinning (ATX 12V
present), lanes marginal (cross-domain signalling). Mining rigs solve this with tied PSU
grounds for exactly this reason.

One correction I had to publish against my own earlier notes: the ZimaBlade's power input
is USB-C, not a barrel jack. Every "pigtail it off a Molex" instruction I had written was
wrong. And be aware the SATA-to-6-pin adapters that ship in VER009S kits are a genuine fire
hazard — use proper Molex or PCIe leads.

The part that makes the whole project pointless

Before re-energising the card, I finally went and read what Immich v3 actually supports.

Immich v3 dropped machine-learning support for Maxwell and Pascal. ONNX Runtime no
longer ships kernels for compute capability 5.2 / 6.1. The P600 is Pascal GP107, compute
6.1.

The failure mode is the whole point:

The v3.x-cuda ML container does not print a helpful message and fall back. It dies on
exit 132 — SIGILL, an illegal instruction, and Docker restarts it forever.

Now compare that to the situation I was trying to escape. With the GPU dead, the container
logs CUDA failure 100: no CUDA-capable device is detected, falls back to
CPUExecutionProvider, and smart search and face recognition keep working — just slowly.

The broken state degraded gracefully. The fixed state would have crash-looped. Every
hour I spent on that riser was moving the system from "unaccelerated but working" to "down".

The cliff isn't only Immich, either. Driver branch 580 is the last one supporting
Maxwell, Pascal and Volta. cuDNN dropped them at 9.11 — 9.10.2 is the last usable
release. Three separate vendors retired that generation inside about a year.

That is the real reason a £30 used Quadro is a worse deal than the price suggests. You are
not buying a slow GPU. You are buying a component with an announced end date on every
software layer above it, and you find out by reading release notes, not benchmarks.

What I'd tell myself at the start

• Check the software cliff before the hardware. One afternoon reading ONNX Runtime and driver release notes would have cancelled the project before I bought the card. Hardware debugging is absorbing and feels like progress; compatibility research is boring and decides whether the work is worth doing at all.
• Graceful degradation can invert. "It's broken but working around it" is sometimes a better state than the fix. Ask what happens after success before you chase it.
• Know the narrowest claim your evidence supports — my presence-detect test was sound and I still over-read it.
• A link that negotiates below the worst component in the chain is a physical-layer warning, not a success.

The honest ending: the card is still dead, and I am no longer in a hurry to fix it. If I
do, it will be pinned to the last ML image that supports the architecture, or used purely
for NVENC transcoding — a fixed-function encoder, unaffected by any CUDA kernel cliff.

---

The full write-up, with the complete lspci output, the diagnostic decision tree and a
read-only script that runs every check above in one pass, is free and MIT on GitHub:
casareanderson/zimablade-gpu-immich.
There is a packaged copy on Gumroad if you'd
rather have it as one download — pay what you want, including nothing.

Every command output quoted here was captured from the running machine. The build is
documented at the point where the fault is understood and the cable has not yet been
replaced.