Running DPDK on a Raspberry Pi

In this post, I'm going to show you how to install DPDK on a Raspberry Pi. DPDK (Data Plane Development Kit) is a set of libraries and drivers designed to enable fast packet processing, typically by bypassing the kernel and interfacing directly with the network card. It's often used as a base for building tools like software defined networking functions, traffic generators and packet inspectors, usually running on enterprise grade servers. While the hardware capabilities of the Raspberry Pi are fairly limited, it's an interesting exercise to see that it can also run DPDK.

Prerequisites

First of all, you'll need a suitable Raspberry Pi board. It needs to be a 64-bit model in order to support hugepages, so any of the newer ones should work. In this tutorial I use a Raspberry Pi 4.
You'll also need a host machine running Linux to cross-compile a new kernel for the RPI. You could probably do it on the RPI itself if you want an extra challenge, but it takes a lot longer to do it this way.
Finally, you'll need all the peripherals to set up a Raspberry Pi such as screen, keyboard, power supply, SD card adapter etc. I don't recommend running this fully headless, as to run DPDK you'll want to mess around with networking drivers and it's good to have another way into it.

Building DPDK on the Raspberry Pi

This is relatively easy step, everything works pretty much out of the box.
Get the DPDK source code by going to https://core.dpdk.org/download/ and pulling the latest version with wget and extracting the tarball.
The Meson build system should already be installed on your RPI, but we need to install a few other dependencies:
sudo apt install python3-pyelftools
sudo apt install libnuma-dev
Build and install DPDK:
meson build
ninja -C build
cd build
sudo meson install
sudo ldconfig
You're now ready to build whatever apps you want. To build the example app:
cd examples/helloworld
make
The built app is now in examples/helloworld and you can try running it:
cd build
sudo ./helloworld -l 0-3 -n 4
At this stage you'll probably run into a hugepage error, which brings us to the next step of the tutorial.

Enabling hugepages on the Raspberry Pi

64-bit RPIs support hugepages, but they aren't enabled by default. To enable them you'll have to recompile the RPI kernel with hugepages enabled in the config. This isn't a difficult process, but it takes a little while. Below is a quick guide on how I got hugepages working on my setup, but refer to the official site for more information on compiling a custom RPI kernel.
First you need to get the kernel source code:
git clone --depth=1 https://github.com/raspberrypi/linux
Move to the kernel directory and set up:
cd linux
KERNEL=kernel8-hp
Set up the config for 64-bit RPIs
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
Run menuconfig and enable hugepages through File systems -> Pseudo file systems -> HugeTLB file system support
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
Compile the kernel - this may take a few hours, depending on the speed of your machine.
make -j 4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
Insert your RPI SD card in the host machine and mount both partitions. Note that /dev/sdb1 and sdb2 may be different for you, use lsblk to check the naming.
mkdir mnt
mkdir mnt/fat32
mkdir mnt/ext4
sudo mount /dev/sdb1 mnt/fat32
sudo mount /dev/sdb2 mnt/ext4
Install the new kernel onto the SD card. Note that I'm renaming the image to kernel8-new.img, this is so we don't overwrite the default kernel and we can switch between them.
sudo env PATH=$PATH make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=mnt/ext4 modules_install
sudo cp arch/arm64/boot/Image mnt/fat32/kernel8-new.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb mnt/fat32/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
sudo cp arch/arm64/boot/dts/overlays/README mnt/fat32/overlays/
Set the boot config to use the new kernel. This should be found on the last line of config.txt.
vi mnt/fat32/config.txt
kernel=kernel8-new.img
Unmount the SD card, reinstall in your RPI and reboot. The new kernel should come up straightaway.
sudo umount mnt/fat32
sudo umount mnt/ext4
From the RPI command line, you can check which kernel is running with:
uname -r
You can also switch back to the old kernel by editing the last line of /boot/config.txt.
Now that hugepages are enabled, we can set them up using the built in DPDK tool dpdk-hugepages.py. More information is available at https://doc.dpdk.org/guides/tools/hugepages.html, but a basic setup I used is:
sudo ./usertools/dpdk-hugepages.py -p 32768k --setup 256M
You can check them with:
sudo ./usertools/dpdk-hugepages.py -s

Using DPDK on the Raspberry Pi

Now that we have hugepages set up, we can go back to the helloworld application

luke@raspberrypi $ sudo ./helloworld -l 0-3 -n 4
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
hello from core 1
hello from core 2
hello from core 3
hello from core 0

We can also run the basic l2fwd application. As the Raspberry Pi ethernet interface doesn't have a poll mode driver we can use, we have to use the PCAP PMD. This can be enabled by setting the environment variable CONFIG_RTE_LIBRTE_PMD_PCAP=y and recompiling DPDK. We then run l2fwd with it enabled using the --vdev option.

luke@raspberrypi $ sudo ./examples/l2fwd/build/l2fwd -l 0-3 --no-pci --vdev=net_pcap0,iface=eth0 -- -p 1
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
TELEMETRY: No legacy callbacks, legacy socket not created
MAC updating enabled
Notice: odd number of ports in portmask.
Lcore 0: RX port 0 TX port 0
Initializing port 0... done: 
Port 0, MAC address: 02:70:63:61:70:00


Checking link statusdone
Port 0 Link up at 10 Gbps FDX Fixed
L2FWD: lcore 1 has nothing to do
L2FWD: lcore 2 has nothing to do
L2FWD: entering main loop on lcore 0
L2FWD:  -- lcoreid=0 portid=0
L2FWD: lcore 3 has nothing to do
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent:                        0
Packets received:                    0
Packets dropped:                     0
Aggregate statistics ===============================
Total packets sent:                  0
Total packets received:              0
Total packets dropped:               0
====================================================

By default, the statistics for l2fwd update every ten seconds. At this stage you should be able to ssh into the RPI over the ethernet port l2fwd is running on, and watch the number of packets increase as you send data back and forth.

Conclusion

So there you have it - a basic DPDK application up and running on the Raspberry Pi! Without a proper poll-mode driver the usefulness is quite constrained, but it's interesting to play around with some heavy-duty software on such a small platform. Let me know in the comments if you think of any interesting applications for this.