DEV Community

Kernel notes (sunshout)
Kernel notes (sunshout)

Posted on Originally published at sunshout.tistory.com

KVM SR-IOV: intel_iommu=on is not SR-IOV until you create VFs

#kvm #linux #networking #virtualization

A lot of homelab notes stop at GRUB:

pci=assign-busses iommu=pt intel_iommu=on
Enter fullscreen mode Exit fullscreen mode

update-grub or on CentOS:

grub2-mkconfig -o /boot/grub2/grub.cfg
Enter fullscreen mode Exit fullscreen mode

Reboot. dmesg shows Intel-IOMMU: enabled. That means the IOMMU is on. It does not mean you have virtual functions. SR-IOV is a PCIe capability on a physical function (PF). You still have to ask the PF to spawn VFs.

Commands I actually ran, with the dmesg checks, are on the original note:

https://sunshout.tistory.com/1615

Confirm the IOMMU before you blame the NIC

Intel:

dmesg | grep -e DMAR -e IOMMU
# expect Intel-IOMMU: enabled
Enter fullscreen mode Exit fullscreen mode

If that line is missing, the GRUB tokens did not reach the kernel (/proc/cmdline) or the board has IOMMU off in firmware.

Then create VFs

lspci — you want a PF that advertises SR-IOV. On the box I used it was an Intel 82599 10G:

04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ ...
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ ...
Enter fullscreen mode Exit fullscreen mode

Spawn four VFs on the first PF:

echo 4 > /sys/bus/pci/devices/0000:04:00.0/sriov_numvfs
lspci | grep -i virtual
Enter fullscreen mode Exit fullscreen mode

If sriov_numvfs does not exist, the firmware VF count is zero, the kernel driver does not support it, or you echoed into the VF by mistake. Use the PF address.

Make it survive reboot with a udev rule or a oneshot systemd unit. sysfs is not persistent.

vfio-pci bind fails after VFs appear

Usually IOMMU groups, not the echo. A device in a group with a sibling you cannot stub out will refuse assignment. find /sys/kernel/iommu_groups -type l and ACS / firmware settings are the next stop — not a second intel_iommu=on.

Isolation is a separate post

Pinning the datapath CPU so host interrupts do not land on the poll thread is isolcpus + IRQ affinity, not SR-IOV. That write-up:

https://sunshout.tistory.com/1620

If sriov_numvfs is missing on a card the vendor claims is SR-IOV capable, dump lspci -vvv -s 04:00.0 | grep -i sriov before you buy another NIC.

Related lab notes (canonical originals)

Top comments (0)