Quick QEMU setup for Linux kernel module debugging

As my first post here on dev.to, I have decided to share my little note on how to quickly setup up an environment for linux kernel module debugging in QEMU.

Step 1: Building linux kernel and busybox userland

Download and extract linux kernel and busybox sources.

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.tar.xz
$ tar xvf linux-5.6.tar.xz
$ wget https://busybox.net/downloads/busybox-1.31.1.tar.bz2
$ tar xvf busybox-1.31.1.tar.bz2

Configure and build linux kernel as usual. A couple of points:

• You can safely start with defconfig, enabling options your module depends on.
• Make sure that all options you will need during debugging are compiled-in and not built as modules, so you don't have to cram them into the initramfs.
• Enable at least these options if you want to use gdb (see "Kernel hacking" section in menuconfig):
  • CONFIG_DEBUG_KERNEL=y (enables kernel debugging facilities)
  • CONFIG_DEBUG_INFO=y (includes debug symbols)
  • CONFIG_KGDB=y (enables kernel GDB backend over serial line)
  • CONFIG_PROVE_LOCKING=y (enable lock dependency checker; not needed for gdb to work, but comes handy for deadlock debugging).

You can quickly check if your kernel boots at all:

qemu-system-x86_64 -kernel linux-5.6.2/arch/x86_64/boot/bzImage \
-nographic \
-append "console=ttyS0"

Now, it is time to build our busybox userland. Again, defconfig will do just fine, with a couple of points:

• Busybox must be linked statically: set CONFIG_STATIC=y (found in "Settings" - "Build Options" section)
• Expect surprises when building busybox on a system with latest glibc versions. In the case of Archlinux, I've got the following:

netstat.c:(.text.ip_port_str+0x50): warning: Using 'getservbyport' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
...
rdate.c:(.text.rdate_main+0xf8): undefined reference to `stime'

You can get around both problems by building busybox with musl instead if glibc:

# Archlinux
sudo pacman -S musl kernel-headers-musl
make CC=musl-gcc

# Ubuntu
sudo apt install musl musl-dev musl-tools
CFLAGS="-I../linux-5.6/usr/include" make CC=musl-gcc

If using kernel headers from the kernel tarball, you have to install them first locally (make headers).

Alternatively, you can try to proceed with glibc by applying this patch to fix stime calls (of course, this wont fix glibc broken static linking warnings).

Step 2: Building minimal initramfs

Busybox binary and ab init script are pretty much all we need for our simple initramfs. Let's put busybox in to /bin and create a /bin/sh symlink, so kernel can look up the shell for our init script:

mkdir bin
cp ../busybox-1.31.1/busybox bin/
ln -s /bin/busybox bin/sh
touch init && chmod 777 init

Our init script should at least do the following:

• mount procfs and sysfs;
• populate /dev using information provided by kernel at sysfs (busybox's mdev can nicely do it for us);
• do your initialization (load a module for debugging, run a test program, you name it); the snippet below also contains an example on how to run some code on per-VM instance basis, which is needed when you are debugging some module that has to communicate to its remote counterpart on another machine;
• drop to interactive shell.

Here's the init script template I use:

#!/bin/sh
# Install busybox applets as symlinks
/bin/busybox --install -s /bin

# Mount procfs & sysfs, populate /dev
mkdir proc sys
mount -t proc none /proc
mount -t sysfs none /sys
mdev -s

# Optional dynamic device creation:
# 1. Old method using mdev as an uevent helper
# echo /bin/mdev > /proc/sys/kernel/hotplug 
# 2. New method using Netlink (requires busybox 1.31 or newer)
# mdev -d

# Get virtual machine instance number passed as a kernel argument
inst=$(cat /proc/cmdline | grep -Eo 'vm_inst=[^ ]+' | cut -d '=' -f 2)

# Do your common initialization here (insert debugged module, run some test script, etc)
echo the init is running

# Do your per-vm instance initialization here (assign an IP address, etc)
[ $inst == 1 ] && echo first instance init
[ $inst == 2 ] && echo second instance init

# Drop to the shell
exec setsid cttyhack sh

A couple of notes:

• Automatic device creation will obviously not work with this simplistic setup, so when, say, you create a character device in your driver, you have to start mdev -s again manually.
• In order to enable hotplugging, you have to either use busybox 1.31 or later, witch introduced netlink interface support for mdev (mdev -d option). Alternatively, you can enable legacy CONFIG_UEVENT_HELPER option in Linux (disabled by default) and set hotplug executable to mdev: echo /bin/mdev > /proc/sys/kernel/hotplug.
• It is not very convenient to just drop to shell using exec /bin/sh because at startup the /dev/console is used by default as a tty. The problem is /dev/console can't be a controlling terminal, thus rendering the job control stuff broken (usually indicated by can't access tty; job control turned off message). This can be easily worked around with setsid cttyhack sh. setsid will exec a cttyhack as a session leader, which will enable job control, and cttyhack will find a real tty device (usually ttyS0) and reopen stdin/stdout.

Finally, let's generate the initramfs image (assuming initramfs is your initramfs root containing /init script and the /bin directory):

cd initramfs                                                                                                                            
find . -print0 | cpio --null -ov -H newc | gzip -9 > ../initramfs.tgz

Step 3: Running QEMU

Running QEMU is as easy as:

qemu-system-x86_64 \
-enable-kvm -cpu host -smp 1 \
-m 256M \
-nographic \
-kernel linux-5.6.2/arch/x86_64/boot/bzImage \
-initrd initramfs.tgz \
-chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \
-device pci-serial,chardev=gdb \
-append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200"

1. -enable-kvm -cpu host -smp 2 enables KVM acceleration and sets emulated CPU model to the host CPU; -smp sets the number of CPU cores the guest can use (1 by default);
2. -m 256M specifies the amount of memory guest can use (128M by default);
3. -nographic disables graphical output and instead emulates a serial port (ttyS0) attached to stdout/stdin;
4. -kernel and -initrd are quite self explanatory;
5. -chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 creates a socket listening for GDB frontend connections on TCP port 1234; this socket is later bound to the virtual serial port, which will appear as ttyS1 in the VM;
6. -device pci-serial,chardev=gdb creates an emulated PCI serial port device, redirecting its IO to the TCP socket;
7. -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200" specifies the kernel command line; remove kgdbwait to let the kernel boot without waiting for GDB connection.

Attaching the gdb to the running kernel is also easy:

$ cd linux-5.6
$ gdb vmlinux

(gdb) target remote :1234
(gdb) c

You'll probably need to explicitly allow gdb to load linux kernel debugging helper scripts. GDB will nicely warn about this if auto-loading is declined:

echo 'add-auto-load-safe-path /home/dav/Dev/qemu/linux-5.6/scripts/gdb/vmlinux-gdb.py' >> ~/.gdbinit

A bit more evolved example: running two machine instances communicating over emulated null modem.

Instance 1:

qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -smp 2 \
    -nographic \
    -kernel linux-5.6.2/arch/x86_64/boot/bzImage \
    -initrd initram.tgz \
    -chardev socket,id=comm,port=8909,host=127.0.0.1 \
    -device pci-serial,chardev=comm \
    -chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \
    -device pci-serial,chardev=gdb \
    -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS2,115200"

Instance 2:

qemu-system-x86_64 \
    -enable-kvm \
    -cpu host \
    -smp 2 \
    -nographic \
    -kernel linux-5.6.2/arch/x86_64/boot/bzImage \
    -initrd initram.tgz \
    -chardev socket,port=8909,server,id=comm,host=0.0.0.0 \
    -device pci-serial,chardev=comm \
    -chardev socket,id=gdb,port=1235,server,nowait,host=0.0.0.0 \
    -device pci-serial,chardev=gdb \
    -append "console=ttyS0 vm_inst=2 kgdbwait kgdboc=ttyS2,115200"

And that's all for now :) Hope you'll find this useful.