To know more about this you can check the official documentation: https://pve.proxmox.com/wiki/PCI(e)_Passthrough
In my case I have Intel LGA 2011-3 Xeon E5 2680 v4 14-core CPU and JGINYUE X99-TI D4 PLUS motherboard together with NVIDIA video card.
You need to be logged as root on proxmox node.
Now, we need to edit /etc/default/grub. To do that run
nano /etc/default/grub and edit the line that already exists there so it will look like this:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
Save the file by pressing Ctrl + X and then pressing Y to save the modified file.
Run update-grub
Now we need to edit another file, run nano /etc/modules-load.d/iommu.conf and edit it so it looks like this (it will be empty):
vfio
vfio_iommu_type1
vfio_pci
Save the file by pressing Ctrl + X and then pressing Y to save the modified file.
Now run update-initramfs -u -k all and feel free to reboot the node as modules won't be loaded otherwise.
After node has rebooted and you logged as root again, run lsmod | grep vfio and check that it looks like this:
root@proxmox:~# lsmod | grep vfio
vfio_pci 20480 0
vfio_pci_core 86016 1 vfio_pci
irqbypass 16384 2 vfio_pci_core,kvm
vfio_iommu_type1 49152 0
vfio 65536 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 126976 1 vfio
Now run dmesg | grep -e DMAR -e IOMMU to be perfectly sure that we do not have any errors (they may be there!) and check that it looks similar:
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[ 0.010917] ACPI: DMAR 0x0000000076D61C58 0000D4 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.010940] ACPI: Reserving DMAR table memory at [mem 0x76d61c58-0x76d61d2b]
[ 0.135278] DMAR: IOMMU enabled
[ 0.382776] DMAR: Host address width 46
[ 0.382777] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0
[ 0.382793] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap 8d2008c10ef0466 ecap f0205b
[ 0.382796] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[ 0.382802] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.382804] DMAR: RMRR base: 0x0000007742a000 end: 0x0000007743afff
[ 0.382806] DMAR: ATSR flags: 0x0
[ 0.382809] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 0.382812] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 1
[ 0.382814] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 1
[ 0.382815] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[ 0.382817] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.382818] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.383432] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.724836] DMAR: No SATC found
[ 0.724838] DMAR: dmar0: Using Queued invalidation
[ 0.724841] DMAR: dmar1: Using Queued invalidation
[ 0.728622] DMAR: Intel(R) Virtualization Technology for Directed I/O
Also you can run dmesg | grep 'remapping' and it should look like this:
root@proxmox:~# dmesg | grep 'remapping'
[ 0.383432] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.383433] x2apic: IRQ remapping doesn't support X2APIC mode
The last ones to check would be iommugroup and id. They must be different for different devices (group can be the same within 1 device). For example here we can see that NVIDIA card devices have the same group but different id:
root@proxmox:~# pvesh get /nodes/proxmox/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────┬──────┬──────────────────┬────
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_device │ sub
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════════════════════════╪══════╪══════════════════╪════
│ 0x010601 │ 0x8d02 │ 0000:00:1f.2 │ 43 │ 0x8086 │ C610/X99 series chipset 6-Port SATA Controller [AHCI mode] │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x010802 │ 0xa80c │ 0000:01:00.0 │ 44 │ 0x144d │ NVMe SSD Controller S4LV008[Pascal] │ │ 0xa801 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x020000 │ 0x8168 │ 0000:05:00.0 │ 46 │ 0x10ec │ RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller │ │ 0x0123 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x030000 │ 0x1d01 │ 0000:03:00.0 │ 45 │ 0x10de │ GP108 [GeForce GT 1030] │ │ 0x85f5 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x040300 │ 0x8d20 │ 0000:00:1b.0 │ 0 │ 0x8086 │ C610/X99 series chipset HD Audio Controller │ │ 0x7270 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x040300 │ 0x0fb8 │ 0000:03:00.1 │ 45 │ 0x10de │ GP108 High Definition Audio Controller │ │ 0x85f5 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
│ 0x060000 │ 0x6f00 │ 0000:00:00.0 │ 29 │ 0x8086 │ Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 │ │ 0x0000 │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼────
We can add PCI device like this to any vm:
Now you're all set! Have fun! 🚀
Top comments (0)