DEV Community

Sergio Méndez
Sergio Méndez

Posted on

Linux Kernel Modules That Explain How Podman Really Works

#docker #containers #devops #linux

Hi, readers this time I want to show you how to build and run three Linux kernel modules that illustrate core Operating Systems concepts: kernel debug messages, character devices, and the relationship between kernel-level information and user-space processes/containers. Just like in my gRPC Hello World post, we are going to use a disposable Ubuntu playground on Killercoda as our environment, and the kernel-modules repository as our source code base. We are going to use Podman as our container engine to run the containers that back the workloads for module3.

This blog post will focus on:

  • Preparing an environment to compile Linux kernel modules.
  • Installing Podman on Ubuntu.
  • Building and loading 3 different kernel modules (module1, module2, module3).
  • Understanding what each module does and how it relates to processes and containers running on the same machine.

What you will learn

  • How to install kernel headers and build dependencies for kernel modules.
  • How to install Podman on Ubuntu.
  • How to compile a kernel module with make and load/unload it with insmod/rmmod, and read kernel debug messages with dmesg.
  • How to interact with a character device created by a kernel module.
  • How kernel-level process information relates to podman ps and OS process commands like ps.

Environment

We are going to work inside the Ubuntu playground on Killercoda. This gives us a disposable Ubuntu VM with root access, which is exactly what we need to build and load kernel modules (something you normally cannot do inside a regular container, since kernel modules run in the host's kernel space).

The source code for the three modules lives in the kernel-modules GitHub repository. Each module has its own folder (module1, module2, module3), and the repository ships an ins_dep.sh script that installs the required build dependencies (things like build-essential, linux-headers-$(uname -r), and make). Inside every module folder:

  • make all compiles the module and produces a .ko (kernel object) file.
  • make clean removes the build artifacts.

Statement

Build and run the 3 kernel modules, showing or explaining what is requested for each one.

How to install kernel headers and build dependencies for kernel modules

1. Clone the repository inside the Killercoda Ubuntu environment:

git clone https://github.com/sergioarmgpl/kernel-modules.git
cd kernel-modules
Enter fullscreen mode Exit fullscreen mode

2. Install the dependencies needed to build kernel modules (kernel headers, compiler, make, etc.):

chmod +x ins_dep.sh
./ins_dep.sh
Enter fullscreen mode Exit fullscreen mode

This script takes care of installing the Linux kernel headers that match the running kernel (linux-headers-$(uname -r)) plus the basic build toolchain, which is required because kernel modules must be compiled against the exact kernel version you are running.

How to install Podman on Ubuntu

Since the Killercoda Ubuntu playground doesn't ship with Podman preinstalled, the first thing we need to do is install it from the official Ubuntu repositories:

sudo apt-get update
sudo apt-get install -y podman
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

podman --version
Enter fullscreen mode Exit fullscreen mode

You should see an output similar to:

podman version 4.9.3
Enter fullscreen mode Exit fullscreen mode

Podman is daemonless: it doesn't need a long-running background service running as root to manage containers. Instead, it launches each container as a regular child process of the CLI itself, which fits nicely with what we are going to observe later with module3 when we inspect the kernel's process table.

How to compile a kernel module with make and load/unload it with insmod/rmmod, and read kernel debug messages with dmesg (Module 1)

module1 is the classic "Hello World" of Linux kernel programming. Its only job is to register init and exit functions and print a message to the kernel ring buffer (accessible with dmesg) when the module is loaded and unloaded, using macros like printk(KERN_INFO ...).

1. Compile the module:

cd module1
make all
Enter fullscreen mode Exit fullscreen mode

2. Load the module into the running kernel:

sudo insmod module1.ko
Enter fullscreen mode Exit fullscreen mode

3. Show the debug message printed by the module using the kernel ring buffer:

dmesg | tail -n 10
Enter fullscreen mode Exit fullscreen mode

You should see an entry similar to:

[ 1234.567890] Module 1: Hello, Kernel! Module loaded successfully.
Enter fullscreen mode Exit fullscreen mode

4. Once you are done, unload the module and clean the build:

sudo rmmod module1
make clean
Enter fullscreen mode Exit fullscreen mode

dmesg shows you both the load message (printed inside the module's init function) and, after rmmod, the unload message (printed inside the exit function). This is the most basic proof that a piece of your own code is now running as part of the Linux kernel.

How to interact with a character device created by a kernel module (Module 2)

What does the module do? module2 goes one step further than module1: instead of only printing a message, it registers a character device in the kernel. A character device is one of the ways the kernel exposes functionality to user space as if it were a regular file, implementing callback functions such as open, read, write, and release. When the module is loaded, it creates a device node (for example /dev/module2 or similar, depending on the major/minor number it registers), and every time a user-space process interacts with that file, the corresponding callback inside the module gets executed.

1. Compile and load the module:

cd ../module2
make all
sudo insmod module2.ko
Enter fullscreen mode Exit fullscreen mode

2. Check that the device was created (either automatically under /dev or by creating the node manually with mknod, depending on how the module registers itself), and look at the assigned major number:

dmesg | tail -n 10
cat /proc/devices | grep module2
ls -l /dev/module2
Enter fullscreen mode Exit fullscreen mode

3. Write to the device created by the module:

echo "Hello from user space" | sudo tee /dev/module2
Enter fullscreen mode Exit fullscreen mode

You can also read back from it to see how the module responds to the data it received:

sudo cat /dev/module2
Enter fullscreen mode Exit fullscreen mode

4. Explaining the kernel messages of module 2:

Every time you load the module, write to the device, or read from it, module2 prints tracing messages through printk, which you can inspect with:

dmesg | tail -n 20
Enter fullscreen mode Exit fullscreen mode

These messages typically show:

  • A message when the device is opened, confirming a process obtained a file descriptor to /dev/module2.
  • A message when data is written, usually including how many bytes were received and, in some implementations, echoing back the content that was sent.
  • A message when the device is read, showing how many bytes were copied back to user space.
  • A message when the device is released (closed).

This illustrates one of the core ideas of an Operating System: user-space processes never touch hardware or kernel memory directly, they go through system calls (open, read, write, close), and the kernel module is the piece of code that decides what happens on the other side of that system call.

5. Clean up:

sudo rmmod module2
make clean
Enter fullscreen mode Exit fullscreen mode

How kernel-level process information relates to podman ps and OS process commands like ps (Module 3)

1. Before loading the module, let's create some containers so we have real workloads running on the system. Create an nginx, a redis, and a mongo container using Podman:

podman run -d --name web nginx
podman run -d --name cache redis
podman run -d --name db mongo
Enter fullscreen mode Exit fullscreen mode

2. List the containers using Podman:

podman ps
Enter fullscreen mode Exit fullscreen mode

You should see the three containers (web, cache, db) with their container IDs, image names, and status.

3. List the same workloads from the Operating System's point of view, using standard process commands. Since every container is ultimately just one (or more) Linux processes running in isolated namespaces and cgroups, you can find them with:

ps -ef | grep -E "nginx|redis|mongo"
pstree -p | grep -E "nginx|redis|mongod"
top
Enter fullscreen mode Exit fullscreen mode

This shows that containers are not "magic": nginx, redis-server, and mongod show up as regular PIDs on the host, the same way any other Linux process would. This is especially visible with Podman, since it is daemonless and runs each container as a child process of the conmon/podman process tree itself, rather than hiding it behind a separate background service.

4. Now compile and load module3:

cd ../module3
make all
sudo insmod module3.ko
Enter fullscreen mode Exit fullscreen mode

5. Inspect the information the module loads into the kernel log:

dmesg | tail -n 30
Enter fullscreen mode Exit fullscreen mode

Explaining what module 3 loads and how it relates to the previous commands:

module3 walks the kernel's internal task list (the same in-kernel data structure that backs commands like ps and top) and prints information about the currently running processes: PID, process name (comm), and often parent PID or state. In other words, it is doing at the kernel level exactly what ps -ef does at the user-space level, except it reads the data directly from kernel structures (such as task_struct, traversed with helpers like for_each_process()) instead of going through /proc.

This is why the output of dmesg after loading module3 will include entries for nginx, redis-server, mongod, their conmon parents, and other system processes: they are all regular entries in the kernel's process table, the exact same list that podman ps indirectly depends on (Podman only adds a mapping between container IDs/names and the PIDs and namespaces the kernel is already tracking, without needing a central daemon to do so).

Putting it all together:

  • podman ps shows you the container abstraction: names, images, ports.
  • ps / pstree / top show you the OS process abstraction: PIDs, parents, CPU/memory usage.
  • module3 shows you the kernel data structures that make both of the previous views possible in the first place.

6. Clean up everything:

sudo rmmod module3
make clean
podman rm -f web cache db
Enter fullscreen mode Exit fullscreen mode

Conclusion about kernel modules, virtualization, and processes

These three modules build on top of each other conceptually. module1 proves that your own code can run inside the kernel and log messages. module2 shows how the kernel exposes an interface to user space through a device file, using the same open/read/write model that every driver in Linux follows. module3 closes the loop by showing that container runtimes like Podman are built on top of the very same primitives the kernel already exposes for processes, namespaces and cgroups: there is no separate "container world", it's all Linux processes, all the way down — and Podman's daemonless design makes that especially visible, since every container you run stays traceable as an ordinary process tree.

Thanks for reading! Last to say, see you in my next blog post.

References

Top comments (0)