ddupard

Posted on May 25

Building a comfortable workflow for debugging an old version of the Linux kernel

#linux #kernel #security

When you want to work on the Linux kernel, for example to see how an exploit acts (like Dirty COW on kernel 4.7), you need to build a comfortable working environment.

I use:

Docker for compiling sources in their original version (with the GCC/LD versions corresponding to the source era).
QEMU for running the executables in a virtual machine.
VS Code as a debugger.

Through this article, I will show you how I configured my environment to achieve an efficient setup.

1) Compiling source code

The first step is to retrieve the kernel source code.

For relatively recent versions (beyond 5.10), the simplest way is to download the tar file from kernel.org. For older versions, the best solution is to fetch them via git.

For example, for 4.7, we start by retrieving the tags:

git ls-remote --tags git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git | grep "v4.7"

Then we clone it to get the version without downloading the entire history:

git clone --depth 1 --branch v4.7 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4.7

After a few minutes, you have a 4.7/ directory containing a working version of the 4.7 kernel sources.

However, these sources are not compilable as-is. Compiling a 4.7 kernel (released in 2016) with a modern compiler (GCC 13+) produces compilation errors due to changes in C language standards or security attribute handling. This is why you need to load versions of GCC and LD compatible with these sources (see the table below), and the easiest way to do this is via Docker.

Kernel Version	Ubuntu Version	Codename	Release Date	Default GCC
v2.6.x	6.06	Dapper Drake	June 2006	GCC 4.0
v3.2	12.04	Precise Pangolin	April 2012	GCC 4.6
v3.13	14.04	Trusty Tahr	April 2014	GCC 4.8
v4.4	16.04	Xenial Xerus	April 2016	GCC 5.4
v4.15	18.04	Bionic Beaver	April 2018	GCC 7.3
v5.4	20.04	Focal Fossa	April 2020	GCC 9.3
v5.15	22.04	Jammy Jellyfish	April 2022	GCC 11.2
v6.8	24.04	Noble Numbat	April 2024	GCC 13.2

We need to retrieve the Docker container corresponding to the kernel version we are interested in. For 4.7, it is Ubuntu 16.04.

To search for available containers, there are 2 solutions:
a) Run the command:

skopeo list-tags docker://docker.io/library/ubuntu

you should see something like

{
"Repository": "docker.io/library/ubuntu",
"Tags": [
"10.04",
"12.04",
"12.04.5",
"12.10",
"13.04",
"13.10",
"14.04",
"14.04.1",
...

...then run the command:

sudo docker pull ubuntu:16.04

b) Go to Docker Hub

and download it.

After a few minutes, you can run the following command to check that your container was downloaded

sudo docker images

and you should see

kernel-builder-515:latest 9f311ec05f16 580MB 147MB U

ubuntu:16.04 1f1a2d56de1d 195MB 46.5MB U

I always favor LTS (Long Term Support) versions like 16.04, because they guarantee library longevity and compilation tool stability, which is critical to avoid linking errors when generating vmlinux.

See the difference here: ubuntu:16.04 is the raw image we just downloaded, while kernel-builder-515 represents the kernel 5.15 image that I customized to include all the necessary compilation tools. We will now cover this customization step using the Dockerfile.

To customize the downloaded Docker container, we use a Dockerfile, here is an example:

### Example for kernel 4.7 (based on Ubuntu 16.04 - Xenial)
FROM ubuntu:16.04

### Install compilation dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    libncurses5-dev \
    bison \
    flex \
    libssl-dev \
    libelf-dev \
    gcc-5 g++-5 \
    && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 100

WORKDIR /usr/src/linux

To create a personal container version, just run the command:

sudo docker build -t my-ubuntu:16.04 .

if we launch sudo docker images, you should see the following result

kernel-builder-515:latest 9f311ec05f16 580MB 147MB U

my-ubuntu:16.04 3aa5b81c120f 822MB 188MB U

ubuntu:16.04 1f1a2d56de1d 195MB 46.5MB U

You can find other Dockerfile examples at https://github.com/ddupard/low-level-security-research.

If you have followed all previous steps, you should be able to compile without issues by running these commands in a shell:

sudo docker run -it \
  -v ~/Desktop/KERNEL/4.7:/build \
  my-ubuntu:16.04

make x86_64_defconfig

scripts/config --enable CONFIG_DEBUG_INFO
scripts/config --disable CONFIG_DEBUG_INFO_REDUCED
scripts/config --enable CONFIG_GDB_SCRIPTS
scripts/config --enable CONFIG_DEBUG_KERNEL
scripts/config --disable CONFIG_RANDOMIZE_BASE

make olddefconfig
make scripts_gdb

make -j$(nproc)

et voila

2) Creating the runtime

This step consists of two parts:

Retrieving and compiling BusyBox.
Creating the initramfs.cpio.gz.

Unlike a complete Linux distribution that weighs gigabytes, BusyBox bundles essential utilities (ls, cat, sh, etc.) into a single binary executable.

We start by creating a shell script (build_busybox.sh) to download the BusyBox sources and compile them within Docker using my-ubuntu:16.04.

#!/bin/bash

BUSYBOX_VERSION="1.36.1"
BUSYBOX_TAR="busybox-$BUSYBOX_VERSION.tar.bz2"
BUILD_DIR="$(pwd)/busybox_build"

if [ ! -f "$BUSYBOX_TAR" ]; then
    echo "Downloading BusyBox $BUSYBOX_VERSION..."
    wget "https://busybox.net/downloads/$BUSYBOX_TAR"
fi

tar xvf "$BUSYBOX_TAR"
cd "busybox-$BUSYBOX_VERSION"

make defconfig
sed -i 's/.*CONFIG_STATIC.*/CONFIG_STATIC=y/' .config

### 4. Compilation
echo "Compilation of BusyBox..."
make -j$(nproc)
make install CONFIG_PREFIX="$BUILD_DIR"

echo "BusyBox compiled in : $BUILD_DIR"

then we launch the following command

sudo docker run -it \
  -v "$(pwd)":/build \
  -w /build \
  my-ubuntu:16.04 \
  /bin/bash ./build_busybox.sh

the compilation fails with the following message indicating that wget and other executables are missing

sudo docker run -it \
-v "$(pwd)":/build \
-w /build \
my-ubuntu:16.04 \
/bin/bash ./build_busybox.sh
Téléchargement de BusyBox 1.36.1...
./build_busybox.sh: line 11: wget: command not found
tar: busybox-1.36.1.tar.bz2: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

so we need to modify the Dockerfile in order to include these new executables

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y \
    build-essential \
    gcc \
    make \
    libncurses5-dev \
    libssl-dev \
    bc \
    bison \
    flex \
    nano \
    git \
    cpio \
    openssh-client \
    wget \
    bzip2 \
    ca-certificates \
    linux-headers-generic \
    && rm -rf /var/lib/apt/lists/*


WORKDIR /opt
RUN git clone https://github.com/radareorg/radare2 \
    && radare2/sys/install.sh --install \
    && rm -rf radare2

WORKDIR /build

now we can recreate the container using

sudo docker build -t my-ubuntu:16.04 .

then we rebuild busybox with the following command

sudo docker run -it \
  -v "$(pwd)":/build \
  -w /build \
  my-ubuntu:16.04 \
  /bin/bash ./build_busybox.sh

however the compilation process wil fail with the following message

CC miscutils/mt.o
CC miscutils/nandwrite.o
CC miscutils/partprobe.o
CC miscutils/raidautorun.o
CC miscutils/readahead.o
CC miscutils/runlevel.o
CC miscutils/rx.o
CC miscutils/seedrng.o
miscutils/seedrng.c:45:24: fatal error: sys/random.h: No such file or directory

it's because you are compiling a recent version of BusyBox with an old Ubuntu 16.04 base.
in order to solve our problem, we will launch the following commands

sudo docker ps -a
in order to get the list of the last docker commands launched

CONTAINER ID	IMAGE	COMMAND	CREATED	STATUS	PORTS NAMES
49c7144e4dbb	my-ubuntu:16.04	"/bin/bash ./build_b…"	13 minutes ago	Exited (0) 10 minutes ago	mystifying_mayer
eb9d3d19f3a0	58aacbfbda6f	"/bin/bash ./build_b…"	23 minutes ago	Exited (0) 23 minutes ago	wonderful_jemison

then

sudo docker start 49c7144e4dbb
sudo docker exec -it 49c7144e4dbb /bin/bash

to restart the docker environment (root@49c7144e4dbb:/build#)
then the following commands

cd /build/busybox-1.36.1
sed -i 's/CONFIG_SEEDRNG=y/CONFIG_SEEDRNG=n/' .config

make clean
make -j$(nproc)

compilation should restart and go to the end.

then we launch the following command to make the install

make install CONFIG_PREFIX="$(pwd)/../busybox_build"

at the end a new directory should appear

Now that BusyBox has copiled, it's time to create initramfs.cpio.gz that sets up the file structure (/proc, /sys, etc.)

we are going to create and lauch a new shell script (create_initramfs.sh)

#!/bin/bash

BUILD_DIR="$(pwd)/busybox_build"
ROOTFS_DIR="$(pwd)/initramfs_staging"

rm -rf "$ROOTFS_DIR"
mkdir -p "$ROOTFS_DIR"
cp -a "$BUILD_DIR/"* "$ROOTFS_DIR/"

cd "$ROOTFS_DIR"
mkdir -p proc sys dev etc/init.d

cat <<EOF > init
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
echo "--- Système démarré avec succès ---"
exec /bin/sh
EOF
chmod +x init

find . -print0 | cpio --null -ov --format=newc | gzip > ../initramfs.cpio.gz

at the end of its execution, we should see something like
./bin/mknod
./bin/more
./bin/zcat
./bin/pidof
./bin/egrep
./bin/watch
./bin/link
./bin/pwd
./bin/dd
./bin/netstat
./bin/fgrep
./bin/stty
./bin/mt
./bin/gunzip
./proc
./etc
./etc/init.d
./init
5556 blocks

and a new file in the root directory (initramfs.cpio.gz)

to check that everything is fine , we are going to launch the VM using the following command

qemu-system-x86_64 \
-kernel ./4.7/arch/x86/boot/bzImage \
-initrd ./initramfs.cpio.gz \
-append "console=ttyS0 nokaslr" \
-nographic

and something like below should appear

if it happens , congrats your environment is working now

3) Debugging setup

To debug the kernel, the base tool is GDB, but it isn't very user-friendly. The idea here is to use VS Code's overlay for GDB.

First, ensure you compile with debug options in your compile_4.7.sh script:

./scripts/config --enable CONFIG_DEBUG_INFO
./scripts/config --disable CONFIG_DEBUG_INFO_REDUCED
./scripts/config --enable CONFIG_GDB_SCRIPTS
./scripts/config --enable CONFIG_DEBUG_KERNEL
./scripts/config --disable CONFIG_RANDOMIZE_BASE

Verify with:

file ./4.7/vmlinux

(You should see a line containing "with debug_info")

Then, launch QEMU with the -s and -S options:

-s: Opens a GDB server on tcp::1234.
-S: Freezes QEMU at the first instruction.

qemu-system-x86_64 \
-kernel ./4.7/arch/x86/boot/bzImage \
-initrd ./initramfs.cpio.gz \
-append "console=ttyS0 nokaslr" \
-nographic \
-s -S

In VS Code, install the C/C++ extension from Microsoft and create a .vscode/launch.json file:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "(gdb) Attacher au Kernel",
      "type": "cppdbg",
      "request": "launch",
      "program": "${workspaceFolder}/4.7/vmlinux",
      "miDebuggerServerAddress": "localhost:1234",
      "miDebuggerPath": "/usr/bin/gdb",
      "setupCommands": [
        { "text": "-enable-pretty-printing" }
      ],
      "cwd": "${workspaceFolder}"
    }
  ]
}

Debugging Workflow

Launch QEMU (using your script with -s -S).
In VS Code, go to the 4.7 directory and launch the "Debug Kernel LKM" configuration.
This triggers the kernel to start. You can now set breakpoints, view registers, and use the debug console to launch commands like the following

-exec info functions
-exec p/x $rax
-exec monitor xp/512gx 0x4786000

Note: If you load an LKM, you must load the symbol file and tell GDB where the source files are located using:
-exec add-symbol-file /path/to/my_attack.ko ...
-exec set substitute-path ...

Happy kernel debugging!

Appendix: Automation Script

To make the environment setup more efficient, I use a script (run_qemu.sh) to quickly switch between different kernel versions and automate the LKM injection process for the 5.15 LTS kernel.

You can save the following code into a file named run_qemu.sh in your ~/Desktop/KERNEL directory. Make sure to give it execution permissions with chmod +x run_qemu.sh.

#!/bin/bash

# This script runs on the host (Ubuntu) in ~/Desktop/KERNEL
cd "$HOME/Desktop/KERNEL"

echo "-------------------------------------------------------"
echo "  COGNITIVE FIREWALL LAB - TEST ENVIRONMENT"
echo "-------------------------------------------------------"
echo "Select the kernel version to test:"
echo "1) Kernel 4.7  (Dirty COW vulnerable)"
echo "2) Kernel 4.8  (Intro XDP)"
echo "3) Kernel 4.9  (Dirty COW patched)"
echo "4) Kernel 5.15 (LTS)"
echo "q) Quit"
echo "-------------------------------------------------------"

read -p "Your choice [1-4] : " choice

case $choice in
    1) VERSION="4.7"; KERNEL_PATH="./4.7/arch/x86/boot/bzImage" ;;
    2) VERSION="4.8"; KERNEL_PATH="./4.8/arch/x86/boot/bzImage" ;;
    3) VERSION="4.9"; KERNEL_PATH="./4.9/arch/x86/boot/bzImage" ;;        
    4)
        VERSION="5.15"
        KERNEL_PATH="./5.15/arch/x86/boot/bzImage"
        LKM2_PATH="./lkm/poc/my_attack.ko"

        if [ -f "$LKM2_PATH" ]; then
            echo "Building initramfs for Kernel 5.15..."
            mkdir -p .initramfs_root
            cd .initramfs_root
            zcat ../initramfs.cpio.gz | cpio -idmv &>/dev/null
            cp "../$LKM2_PATH" .

            # Automatic LKM injection script
            cat << 'EOF' > init
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev 2>/dev/null
if [ -f /my_attack.ko ]; then insmod /my_attack.ko; fi
exec /bin/sh
EOF
            chmod +x init
            find . -print0 | cpio --null -ov --format=newc 2>/dev/null | gzip -9 > ../initramfs.cpio.gz
            cd ..
            rm -rf .initramfs_root
        fi
        qemu-system-x86_64 -kernel "$KERNEL_PATH" -initrd "$HOME/Desktop/KERNEL/initramfs.cpio.gz" -nographic -append "console=ttyS0 loglevel=7 nosmep nosmap nokaslr" -s -S
        exit 0
        ;;
    q) exit 0 ;;
    *) echo "Invalid choice"; exit 1 ;;
esac

# Launch QEMU for versions 1, 2, 3
qemu-system-x86_64 \
  -kernel "$KERNEL_PATH" \
  -initrd "$HOME/Desktop/KERNEL/initramfs.cpio.gz" \
  -nographic \
  -append "console=ttyS0 nokaslr loglevel=7" \
  -s -S