DEV Community

nutthawit-l
nutthawit-l

Posted on

Configuring GPU Pass-Through for AMD cards on openSUSE Kalpa

#cli #linux #opensource #tutorial

Prerequisites

  • You need an additional display card on the host (eg. 1 igpu + 1 dedicate gpu or 2 dedicate gpu).
  • GPU pass-through is supported on the AMD64/Intel 64 architecture only.

Configuring the host

Verify the host environment

Verify that the host support VT-d technology and that it is already enabled in the firmware settings:

# dmesg | grep -e "Directed I/O"
[    0.283295] [      T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
Enter fullscreen mode Exit fullscreen mode

Enable IOMMU

IOMMU is disabled by default. You need to enable it at boot time in the /etc/kernel/cmdline configuration file.

1) For Intel CPU add this line to file:

intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
Enter fullscreen mode Exit fullscreen mode

2) For AMD CPU add this line instead:

amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
Enter fullscreen mode Exit fullscreen mode

3) Finally, when you cat the file, you will see content like this:

# cat /etc/kernel/cmdline 
root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
Enter fullscreen mode Exit fullscreen mode

Configure VFIO and isolate the GPU used for pass-through

1) Find the card vendor and model IDs:

# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
Enter fullscreen mode Exit fullscreen mode

2) Create the file /etc/modprobe.d/vfio.conf with the following content:

options vfio-pci ids=1002:7480,1002:ab30
Enter fullscreen mode Exit fullscreen mode

Load the VFIO driver

Including the driver in the initrd file

1) Create the file /etc/dracut.conf.d/gpu-passthrough.conf and add the following content (mind the leading whitespace):

add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
Enter fullscreen mode Exit fullscreen mode

2) Re-generate the initrd file:

# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4

# sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf

# sdbootutil update-all-entries 7
# sdbootutil mkinitrd 7
Enter fullscreen mode Exit fullscreen mode

3) Reboot the host machine

For most of the changes in the above steps to take effect, you need to reboot the host machine:

# reboot
Enter fullscreen mode Exit fullscreen mode

4) Verify IOMMU is Active

After your Kalpa host boots back up, open a terminal and check the kernel logs to confirm that the hardware isolation is working:

dmesg | grep -i "Adding to iommu group"
[    0.283068] [      T1] pci 0000:00:00.0: Adding to iommu group 0
[    0.283078] [      T1] pci 0000:00:01.0: Adding to iommu group 1
[    0.283091] [      T1] pci 0000:00:14.0: Adding to iommu group 2
[    0.283097] [      T1] pci 0000:00:14.2: Adding to iommu group 2
[    0.283106] [      T1] pci 0000:00:15.0: Adding to iommu group 3
[    0.283115] [      T1] pci 0000:00:16.0: Adding to iommu group 4
[    0.283122] [      T1] pci 0000:00:17.0: Adding to iommu group 5
[    0.283137] [      T1] pci 0000:00:1a.0: Adding to iommu group 6
[    0.283147] [      T1] pci 0000:00:1c.0: Adding to iommu group 7
[    0.283155] [      T1] pci 0000:00:1c.3: Adding to iommu group 8
[    0.283163] [      T1] pci 0000:00:1c.4: Adding to iommu group 9
[    0.283172] [      T1] pci 0000:00:1d.0: Adding to iommu group 10
[    0.283188] [      T1] pci 0000:00:1f.0: Adding to iommu group 11
[    0.283195] [      T1] pci 0000:00:1f.3: Adding to iommu group 11
[    0.283201] [      T1] pci 0000:00:1f.4: Adding to iommu group 11
[    0.283208] [      T1] pci 0000:00:1f.5: Adding to iommu group 11
[    0.283216] [      T1] pci 0000:01:00.0: Adding to iommu group 12
[    0.283224] [      T1] pci 0000:02:00.0: Adding to iommu group 13
[    0.283235] [      T1] pci 0000:03:00.0: Adding to iommu group 14
[    0.283245] [      T1] pci 0000:03:00.1: Adding to iommu group 15
[    0.283260] [      T1] pci 0000:04:00.0: Adding to iommu group 16
[    0.283268] [      T1] pci 0000:06:00.0: Adding to iommu group 17
[    0.283275] [      T1] pci 0000:07:00.0: Adding to iommu group 18
Enter fullscreen mode Exit fullscreen mode

Configuring the guest

This section describes how to configure the guest virtual machine so that it can use the host's GPU.

Requirements for the guest configuration

  • Use Q35 chipset if possible.
  • Install the guest VM using UEFI firmware.
  • Add the following emulated Graphic: Spice or VNC
  • Add the following emulated Device: qxl, VGA or Virtio
  • Add the host PCI device (03:00.0 in our example) to the guest.
  • For the best performance, we recommend using virtio drivers for the network card and storage.

💡 Credits & Inspiration

This article was inspired by Configuring GPU Pass-Through for NVIDIA cards

Top comments (0)