DEV Community: Carlo Lobrano

Test Kernel changes on QEMU

Carlo Lobrano — Sat, 23 Apr 2022 13:26:52 +0000

This is all about testing Linux kernel changes on a QEMU virtual machine in cycles of build, install and run shortest than 5 minutes.

The Internet is full of tutorials about building the Linux Kernel, yet I found less content about applying the changes to a virtualized environment, so that will be the goal of this post.

Running QEMU virtual image

Also this content is quite common, so I will be short

# Create the disk: a qcow2 format image of 2G
$ qemu-img create -f qcow2 alpine.img 2G

# Download a live CD ISO image (e.g. Ubuntu, Alpine, whatever), then run the live image and install
# it into the disk created above.
$ qemu-system-x86_64 -cdrom alpine-downloaded.iso alpine.img -m 512M -enable-kvm

# Run the newly installed image from the disk
$ qemu-system-x86_64 -enable-kvm -m 512M -smp 4 -cpu host -drive file=alpine.img

Running QEMU with a custom kernel

You downloaded and built the Linux Kernel under ~/workspace/linux, that is you know have a /linux/arch/x86_64/boot/bzImage, the command to run it in QEMU is

$ qemu-system-x86_64 \
        -enable-kvm -m 512M -smp 4 -cpu host \
        -kernel ~/workspace/linux/arch/x86_64/boot/bzImage \
        -append "root=/dev/sda3 console=ttyS0 rw" \
        -drive file=alpine.img

Make a change and test it

Consider now you made a change to the kernel. Any change, maybe even so silly like the one below

diff --git a/drivers/usb/serial/option.c b/drivers/usb/serial/option.c
index e7755d9cf..534e8051e 100644
--- a/drivers/usb/serial/option.c
+++ b/drivers/usb/serial/option.c
@@ -2138,7 +2138,7 @@ static struct usb_serial_driver option_1port_device = {
        .owner =    THIS_MODULE,
        .name =     "option1",
    },
-   .description       = "GSM modem (1-port)",
+   .description       = "it's me, Mario!",
    .id_table          = option_ids,
    .num_ports         = 1,
    .probe             = option_probe,

How do you test it on your QEMU VM?

First, mount the VM image on the host

$ mkdir /tmp/alpine
$ guestmount --add alpine.img --root /dev/sda3 /tmp/alpine
$ ls /tmp/alpine
bin   dev  home  lost+found  mnt  proc  run   srv   sys  usr
boot  etc  lib   media       opt  root  sbin  swap  tmp  var

Second and last, during the build, tell the kernel to install the modules inside the mounted image

$ make -j$(nproc)
...
$ make -j$(nproc) modules
$ INSTALL_MOD_PATH=/tmp/alpine make modules_install
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/drivers/usb/serial/option.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/drivers/usb/serial/usb-serial-simple.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/drivers/usb/serial/usb_wwan.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/drivers/usb/serial/usbserial.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/fs/efivarfs/efivarfs.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/ipv4/netfilter/iptable_nat.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/nf_log_syslog.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/xt_LOG.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/xt_MASQUERADE.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/xt_addrtype.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/xt_mark.ko
INSTALL /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty/kernel/net/netfilter/xt_nat.ko
DEPMOD  /tmp/alpine/lib/modules/5.18.0-rc2-custom-ga1901b464e7e-dirty

don't forget to UNMOUNT the image

$ umount /tmp/alpine

Run again your kernel, the change (at least the one above) will be visible loading "option" module.

Final words

This process is so fast that one can test changes to the kernel in few seconds, moreover you are going to have a wide range of debug features provided by QEMU and, not less important, your machine is preserved.

Let mkdir create parents directories

Carlo Lobrano — Mon, 21 Mar 2022 15:17:04 +0000

I noticed that it is more than a year since the last post, so I thought that a small new content would have been easier to write.
This in particular refers to an extension I wrote some time ago for the mkdir system call to make it a bit more proactive 😄.

90% of the times, I use mkdir to create a new folder in the current directory, so the few times I want to create a new folder in a deeper path I always forget to add the --parent flag.

me/blog $ mkdir foo/bin/bar
mkdir: cannot create directory ‘foo/bin/bar’: No such file or directory

as the Man-pages would tell you, since the parent directory foo/bin does not exist, mkdir fails to create bar, but what if mkdir were so kind to tell me just that, and suggests to add the flag on my behalf? 😄

This is actually somehow easy.

First of all, you need a way to define new functions in your shell. I use zsh, but in bash would be the same: write a function definition in a file and source it in .bashrc or .zshrc.

This comes from my .zshrc

#
# Functions and aliases
#
source ~/.dot/.config/cconf/zsh/functions.zsh

What to do for mkdir?

override it with a custom function ➡️ function mkdir() {}
find the parent directory mkdir expects to exist already ➡️ echo $1 | grep -E -q '[\S+/]+'
if it doesn't exist, ask if it must be created or not ➡️ echo "Press ENTER to run mkdir with --parents."

Ok, point 2 probably requires some explaination, so let's back to the example above

me/blog $ mkdir foo/bin/bar

bar is the folder to be created, while foo/bin/ is the parent.
The grep expression above looks for any sequence of substrings ending with slash /: foo/, foo/bar/, etc., but not bar, or foo, which would be a directory in the current folder.

So putting all these pieces together:

function mkdir() {
    # extend mkdir with custom features
    # propose adding "--parents" flag.
    echo $1 | grep -E -q '[\S+/]+'
    if [[ $? == 0 && ! -d $1 ]]; then
        # one of the directories in $1 path do not exist
        # suggest adding "--parents" flag.
        echo "# Some parents in $1 do not exist. Press ENTER to run mkdir with --parents."
        read
        command mkdir --parents $@
        echo "# done"
    else
        command mkdir $@
    fi
}

Let's try again

me/blog $ mkdir foo/bin/bar                                                                                    
# Some parents in foo/bin/bar do not exist. Press ENTER to run mkdir with --parents.

# done

me/blog $ tree | grep -e foo -e bar -e bin
├── foo
│   └── bin
│       └── bar

Of course in a similar manner you can override rm as well (see here).

How to use GitHub Actions to connect with Launchpad

Carlo Lobrano — Mon, 02 Nov 2020 08:42:12 +0000

Since the beginning of the Yaru project, three years ago, we always had a little problem: two different places for bug tracing.
Our main repository is hosted on GitHub, but we also have the Launchpad page taking care of the yaru-theme package.

TLDR

Whith this python script, and GitHub actions, I mirrored the bugs from one bug tracking system to the other.

The full story

I can not tress enough how much I appreciate our users that take time to report problems, desires, ideas, but indeed our responsiveness on Launchpad has not been the best so far, and we shall thank the Ubuntu Desktop Team that took care of these reports.

A couple of weeks ago, we thought that automatically mirroring Launchpad bugs on our GitHub repository could have been a solution, and we decided to take care of this with our CI.

Ideally, our solution would have the following:

Perform daily checks.
Get the list of all open bugs on Launchpad.
For any new bug found, create a GitHub bug with ID, Title and link to the original report.

We already have some GitHub Actions to keep track of our upstreams. It's configured to run periodically, and create a PR when there is new content we shall take care of. This configuration looked promising for the first point.

launchpad provides Launchpadlib, a nice python library to interact with the servers.

launchpadlib is an open-source Python library that lets you treat the HTTP resources published by Launchpad's web service as Python objects responding to a standard set of commands. With launchpadlib you can integrate your applications into Launchpad without knowing a lot about HTTP client programming.

Playing with this library was pretty fun, and the best way to learn it was through the listed examples. For instance, I must thank Bughugger, that showed me the way to get the list of bugs of a given application.

import os
from launchpadlib.launchpad import Launchpad

HOME = os.path.expanduser("~")
CACHEDIR = os.path.join(HOME, ".launchpadlib", "cache")

lp = Launchpad.login_anonymously(
    "Yaru LP bug checker", "production", CACHEDIR, version="devel"
)

ubuntu = lp.distributions["ubuntu"]
archive = ubuntu.main_archive
packages = archive.getPublishedSources(source_name="yaru")
package = ubuntu.getSourcePackage(name=packages[0].source_package_name)

bug_tasks = package.searchTasks()
for task in bug_tasks:
    print(task)

I then extract three field from the task:

ID: task.id
Title: task.title
Link: "https://bugs.launchpad.net/ubuntu/+source/yaru-theme/+bug/" + str(task.id)

The third point is actually made of two different steps:

Identify new issues.
Create an issue.

Both points have been resolved using HUB.

hub is a command line tool that wraps git in order to extend it with extra features and commands that make working with GitHub easier.

GitHub provides its own CLI tool, which I use on a daily bases, but the point that convinced me to use HUB is the following:

hub can also be used to make shell scripts that directly interact with the GitHub API.

Create issues

Let's start from the last step. Creating an issue with HUB is simple:

hub issue create -m <title> -m <message> -l Launchpad

Here I used the -m flag twice to set the title:

LP#[ID] [TITLE]

and the body of the issue:

Reported first on Launchpad at https://bugs.launchpad.net/ubuntu/+source/yaru-theme/+bug/[ID]

then I added a Launchpad label (-l), which makes bug management easier, and ended up very useful for the next step.

Create only NEW bugs

HUB can list all the bugs from the repository, but - at the time of writing - Yaru has more than 1.000 bugs (only 44 open 😀), then it takes a while to get them all at once. Luckily, HUB can filter by label!

hub issue --state all --label Launchpad

Parsing the output is easy with Python, all the rest is just a little glue logic to put all together.

Results

I am satisfied with the end result. We can be more responsive to our user base requests now, and I had fun writing the python script, and learned something new of GitHub Action.