THE SYMPTOM: My NVIDIA Tesla P40 would stop responding after a VM shutdown. No error messages, just a dead GPU that required a full host reboot to recover.
WHAT I EXPECTED: A clean shutdown of a VM with GPU passthrough should leave the GPU in a ready state. I assumed the host would handle power states gracefully.
WHAT ACTUALLY HAPPENED: The GPU went into D3cold, a low-power state that it couldn't exit without a full host reboot. This happened even after proper VM shutdowns. The issue was especially prevalent on Proxmox 8.4 with kernel 6.8.x and QEMU 8.0.1, where the lack of FLR support on the P40 made it impossible to reset the GPU from the host.
THE FIX: I disabled D3cold before passthrough by writing 0 to /sys/bus/pci/devices/0000:08:00.0/d3cold_allowed. I also pinned the GPU’s PCI address using a udev rule to prevent it from shifting on reboot. Here's the rule I used:
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{device}=="0x1b80", ATTR{bus}=="0000:08", SYMLINK+="gpu-passthrough"
This ensured the GPU stayed on the same PCIe bus and avoided the D3cold trap entirely. For Proxmox 8.4 users, I also had to explicitly set -machine q35 in the VM config to prevent QEMU from asserting on boot.
WHY THIS MATTERS: If you're running non-FLR GPUs like the P40 on Proxmox 8.4 or later, you're likely to hit this issue. It's not just a matter of setting up passthrough — you need to actively prevent the GPU from entering D3cold and lock its PCI address. If you skip either step, you're asking for a bricked GPU. I've seen this happen on more than one occasion, and the fix is always the same: disable D3cold and pin the address.
This isn't just a Proxmox-specific gotcha. Any system that doesn't support FLR on the GPU and relies on the kernel to manage power states is at risk. If you're using a Tesla P40, T4, or any other non-FLR GPU and you're seeing GPU failures after VM shutdown or reboot, this is the fix you need. I've also seen this issue surface with AMD GPUs under certain conditions, though the fix is slightly different.
If you're running AI workloads on Kubernetes or any other system that depends on GPU passthrough, this is a critical detail. You don't want to be the one who has to power cycle a node just to get a GPU working again. I've had to do it more than once. It's not fun. The key is to prevent the GPU from ever getting into a state where it can't reset itself.
For those who are considering moving away from GPU passthrough entirely, I've also found that running the NVIDIA driver directly on the host can be a much more stable option. It avoids all the PCIe bus instability and power state issues. I've tested this with the NVIDIA Container Toolkit and it's worked well for me in production environments.
I've written this post not because I want to scare you — but because I want to save you from the frustration of a bricked GPU. If you're running Proxmox, using older GPUs, and you've had this issue, you're not alone. I've been there, and I've found a way to avoid it.
Top comments (0)