To drink some milk, I'm fixing the barn.
singlefs's verification system needs GPUs for cross-checking. Each agent used to call its own scripts, which stayed messy and left resources poorly used.
I recently refactored that code and built a task management service. Memory management, sharding and multi-machine coordination now go much more smoothly than before.
Well, as expected, the unexpected happened. A GPU crashed...
It starts with Xid 62 and ends with Xid 154. In the kernel log, GSP reports "PMU has halted", with the watchdog at partition 4#0, task 3. It also can't be recovered by command: nvidia-smi --gpu-reset -i 0 returned "Not Supported" both times.
At first I didn't care, but it crashed five times in one day. This caused collateral damage to the task management service. When the service clears the stage, it checks every card on the machine, so if any one is broken, the whole machine can't start. Worse, the service marked each healthy card it was assigned as broken and disabled them one by one. One card's fault became a whole machine down.
So on the second crash I added an alert, on the third error tolerance, on the fourth task reassignment, and on the fifth, this stopped being fun...
After some searching, there were three possible causes:
- Power supply
- Firmware
- GSP scheduling
- The power supply can't be the problem: after a reboot the card recovers and can run tasks for a long time.
2 and 3 are really the same thing. The GSP firmware ships with the driver, and the PMU halt is reported from the firmware side. The problem surviving a driver change doesn't mean "not firmware"; it means none of those driver versions has fixed it yet. I read an article by an expert dated August 19, 2026, and it made a lot of sense: when the GPU switches frequently from very low power (P8) to high power (P0), the PMU state machine hangs and causes
Xid 62. It seems to be a scheduling deadlock in the Blackwell GSP firmware.
So I lined up per-second telemetry (P-state, clocks, power) against the GPU_LIFECYCLE timestamps emitted by the tests and the precise kernel timestamps. After the card sat idle in P8 for about 13 seconds, Xid 62 hit during the 94 ms in which the next test was creating resources on the card, 2 ms before any work was actually submitted.
After locking the minimum clock at 800 MHz or above, the same tests ran for 10 minutes without an error. The lock command is nvidia-smi -i <bus id> -lgc 800,3105; I made it a systemd unit, nvidia-5080-min-clock, that starts at boot. Resetting the card clears the lock, so it has to be reapplied.
The machine doesn't need a reboot: you can reset the card by unbinding and remounting it. 0000:01:00.0 below is the bus ID on my machine; check yours first with lspci | grep -i nvidia. The commands:
d=/sys/bus/pci/devices/0000:01:00.0
# 1. Preserve the evidence first: AER status registers (a reset clears them)
lspci -s 01:00.0 -vv | grep -iE 'UESta|CESta|DevSta|LnkSta'
# 2. The card must be fully released
fuser -v /dev/nvidia0
systemctl stop nvidia-persistenced
# 3. Unbind the audio and GPU functions
echo 0000:01:00.1 > /sys/bus/pci/drivers/snd_hda_intel/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/nvidia/unbind
# 4. PCI-level reset (remove + rescan alone without this step leaves GSP unable to boot: "unexpected WPR2 already up")
echo 1 > $d/reset
# 5. Remove both functions and rescan so the driver picks the card up again
echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove
echo 1 > $d/remove
sleep 2
echo 1 > /sys/bus/pci/rescan
# 6. Bring persistenced back and re-lock the clocks (a reset clears the lock)
systemctl start nvidia-persistenced
systemctl restart nvidia-5080-min-clock
# 7. Verify: recovery action is None, the three cards keep their indices, the 5080 idles outside P8, no new RmInitAdapter failures in the kernel log
nvidia-smi -i 00000000:01:00.0 -q | grep -i 'recovery action'
nvidia-smi --query-gpu=index,name,pci.bus_id,pstate,clocks.sm --format=csv,noheader
journalctl -k --since '-2min' | grep -E 'RmInitAdapter|bad state'
What am I doing up on the barn roof? Where's my milk?
Top comments (0)