DEV Community

Cover image for Set up a GPU watch on a ZimaBlade: get told when the card falls off
Christian Anderson
Christian Anderson

Posted on

Set up a GPU watch on a ZimaBlade: get told when the card falls off

#linux #tutorial #ai #discuss

My ZimaBlade has a Quadro P600 on a powered riser for Immich. It works, for a while, then logs Xid 79, GPU has fallen off the bus and stays gone until a reboot. The nasty part is Immich doesn't break. It quietly falls back to the CPU and carries on, slowly. Mine went six days before I noticed.

So the GPU repo has a new script: zima-gpu-watch.sh. It checks every few minutes, logs how long each boot's GPU lasts, and alerts once when it goes and once when it's back. Not every five minutes.

The watcher run against the real ZimaBlade, whose GPU was off the bus at 408 hours of uptime, then the alive and dead check from the weekly setup run, alerting once per change

That first line is my actual Zima this morning: dead, 408 hours into the boot. The rest is the check the setup guide runs every week on a fresh machine, using a stand-in for nvidia-smi so it can fake alive and dead.

Diagnose first 

git clone https://github.com/casareanderson/zimablade-gpu-immich && cd zimablade-gpu-immich
bash diagnostics/zima-gpu-check.sh
Enter fullscreen mode Exit fullscreen mode

Nine read-only sections. As a normal user, five are accurate and the rest tell you they need root (on ZimaOS that's the privileged-container route in the guide). Read section 04 first: width 0 is a physical fault no driver will fix.

Then watch it

From another always-on box with an SSH key to the Zima:

ZIMA=you@zima GPU_MATCH=P600 WEBHOOK=https://discord.com/api/webhooks/... bash diagnostics/zima-gpu-watch.sh
Enter fullscreen mode Exit fullscreen mode

Put that in cron every 15 minutes. Two rules it follows:

  • Unreachable isn't dead. If SSH can't reach the Zima (rebooting, network blip) it logs unreachable and doesn't touch the state.
  • One alert per change. First run just records the starting state.

What CI can't prove

The real hardware. The weekly run proves both scripts run cleanly on a machine with no GPU and that the alive/dead logic alerts exactly once per change. Whether your card stays on the bus is between you, your riser cable and your grounds.

Full guide: github.com/casareanderson/zimablade-gpu-immich/blob/main/docs/SETUP.md

Anyone got a riser setup in a small box that's actually stable? What fixed it?

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.

Top comments (0)