DEV Community

Cover image for Hello Embedded World - booting a minimal Linux with Busybox on RISC-V, from source
Donald Sebastian Leung
Donald Sebastian Leung

Posted on

Hello Embedded World - booting a minimal Linux with Busybox on RISC-V, from source

#linux #embedded #riscv #opensource

Last time we saw how to boot Ubuntu for RISC-V on the QEMU virt board and set up a development environment for C and RISC-V assembly. That was fun and all, but nothing compares to compiling our own Linux kernel and userspace utilities and get that to boot on a virtual (or physical) RISC-V board. So we're gonna do it today!

The RISC-V docs actually outlines a process for doing so, but unfortunately, it is very brief and skips a lot of details and thus may not be suitable for readers new to the embedded world. At least it took me a lot of fumbling, extra Googling and experimentation to finally get it working. This article thus attempts to bridge the gaps in the official docs so it would be easier to follow for a newcomer. Let's go!

Preparation

A proper Linux environment. If on Windows / macOS, run a full-blown Linux VM with a hypervisor like VirtualBox or VMware. WSL2 on Windows may or may not work, and will not be supported in this article.

The reference distribution is Ubuntu 22.04. You may have to adapt the instructions and commands accordingly if on another Linux distribution. Or for minimal hassle, run an Ubuntu VM anyway.

It is assumed you are already well-versed in Linux commands and administration. If you get an error like gpg2: command not found halfway, you'll be expected to figure out to sudo apt install gnupg2 instead of complaining ;-)

Setting up the host, cross-compiling Linux and BusyBox for RISC-V

Main article: Running 64- and 32-bit RISC-V Linux on QEMU

Refresh repository metadata:

$ sudo apt update
Enter fullscreen mode Exit fullscreen mode

Install build dependencies for Linux and BusyBox:

$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev \
                 gawk build-essential bison flex texinfo gperf libtool patchutils bc \
                 zlib1g-dev libexpat-dev git
Enter fullscreen mode Exit fullscreen mode

You'll also need to install qemu-system for emulating the RISC-V virt board:

$ sudo apt install -y qemu-system
Enter fullscreen mode Exit fullscreen mode

Linux

Head over to kernel.org and download the latest stable kernel, or any other sufficiently recent kernel of your choosing. For example, at the time of writing (2022-08-14), the latest stable kernel is 5.19.1:

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.1.tar.xz
Enter fullscreen mode Exit fullscreen mode

Fetch also the corresponding kernel signature. For something as important as an OS kernel, it's best to verify its authenticity and integrity:

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.1.tar.sign
Enter fullscreen mode Exit fullscreen mode

Decompress the kernel tarball (but do not extract the archive yet):

$ unxz linux-5.19.1.tar.xz
Enter fullscreen mode Exit fullscreen mode

https://kernel.org/category/signatures.html explains how to verify the signature of the kernel tarball. First install gnupg2:

$ sudo apt install gnupg2
Enter fullscreen mode Exit fullscreen mode

Now import the keys for Linus Torvalds and Greg Kroah-Hartman, creator of Linux and lead kernel developer (respectively):

$ gpg2 --locate-keys torvalds@kernel.org gregkh@kernel.org
Enter fullscreen mode Exit fullscreen mode

Trust the imported keys. Replace the hashes shown below based on the output you got from the previous command:

$ gpg2 --tofu-policy good 38DBBDC86092693E
$ gpg2 --tofu-policy good 79BE3E4300411886
Enter fullscreen mode Exit fullscreen mode

Now verify:

$ gpg2 --trust-model tofu --verify linux-5.19.1.tar.sign
Enter fullscreen mode Exit fullscreen mode

Expected output:

gpg: assuming signed data in 'linux-5.19.1.tar'
gpg: Signature made Thu Aug 11 11:22:54 2022 UTC
gpg:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Good signature from "Greg Kroah-Hartman <gregkh@kernel.org>" [full]
gpg: gregkh@kernel.org: Verified 1 signatures in the past 0 seconds.  Encrypted
     0 messages.
Enter fullscreen mode Exit fullscreen mode

Now unpack the tarball:

$ tar xvf linux-5.19.1.tar
Enter fullscreen mode Exit fullscreen mode

And enter the source tree:

$ pushd linux-5.19.1/
Enter fullscreen mode Exit fullscreen mode

In order to cross-compile for RISC-V, we need a cross-compiler. Install gcc-riscv64-linux-gnu:

$ sudo apt install -y gcc-riscv64-linux-gnu
Enter fullscreen mode Exit fullscreen mode

Now configure the kernel for RISC-V:

$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- defconfig
Enter fullscreen mode Exit fullscreen mode

And build it (this can take a while):

$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j$(nproc)
Enter fullscreen mode Exit fullscreen mode

Now we can leave the source tree:

$ popd
Enter fullscreen mode Exit fullscreen mode

BusyBox

Head over to busybox.net for the BusyBox source code. The latest release at the time of writing (2022-08-14) is 1.35.0.

Fetch the compressed tarball:

$ wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2
Enter fullscreen mode Exit fullscreen mode

And the SHA256 hash, to verify its integrity:

$ wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2.sha256
Enter fullscreen mode Exit fullscreen mode

There also appears to be a signature file for verifying the tarball signature, but we'll not cover it here.

Verify the checksum:

$ sha256sum -c busybox-1.35.0.tar.bz2.sha256
Enter fullscreen mode Exit fullscreen mode

Expected output:

busybox-1.35.0.tar.bz2: OK
Enter fullscreen mode Exit fullscreen mode

Unpack the archive:

$ tar xvf busybox-1.35.0.tar.bz2
Enter fullscreen mode Exit fullscreen mode

Now enter the source tree:

$ pushd busybox-1.35.0/
Enter fullscreen mode Exit fullscreen mode

Configure and build for RISC-V - make sure the resulting binary is statically linked:

$ CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make defconfig
$ CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make -j$(nproc)
Enter fullscreen mode Exit fullscreen mode

We can now leave the source tree:

$ popd
Enter fullscreen mode Exit fullscreen mode

Preparing the virtual disk, rootfs

Before we can boot our virt board, we need to prepare a disk image with a root filesystem (rootfs). The rootfs will mainly be provided by BusyBox, though we'll need to create a few additional directories for mount points, startup scripts and the like.

The simplest way to do so is with dd - let's make a virtual disk image busybox 1GB in size:

$ dd if=/dev/zero of=busybox bs=1M count=1024
Enter fullscreen mode Exit fullscreen mode

Format it with ext4 filesystem (or another supported filesystem of your choice):

$ mkfs.ext4 busybox
Enter fullscreen mode Exit fullscreen mode

Create a mount point rootfs:

$ mkdir -p rootfs
Enter fullscreen mode Exit fullscreen mode

Now mount our virtual disk on our newly created mount directory:

$ sudo mount busybox rootfs
Enter fullscreen mode Exit fullscreen mode

We can now install Busybox on this rootfs:

$ sudo CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make -C busybox-1.35.0/ install CONFIG_PREFIX=../rootfs
Enter fullscreen mode Exit fullscreen mode

Create a few directories for mounting key filesystems like procfs, sysfs and devtmpfs for BusyBox to boot correctly:

$ sudo mkdir -p rootfs/proc rootfs/sys rootfs/dev
Enter fullscreen mode Exit fullscreen mode

Make sure /etc/fstab exists to silence a warning on poweroff:

$ sudo mkdir -p rootfs/etc
$ sudo touch rootfs/etc/fstab
Enter fullscreen mode Exit fullscreen mode

Create a directory /etc/init.d for startup scripts:

$ sudo mkdir -p rootfs/etc/init.d
Enter fullscreen mode Exit fullscreen mode

BusyBox runs a script /etc/init.d/rcS on system startup. Let's fill it in and make it executable:

$ sudo bash -c "cat > rootfs/etc/init.d/rcS" << EOF
#!/bin/sh

echo "Hello Embedded World!"
echo "Hello RISC-V World!"
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ip addr add 10.0.2.15/24 dev eth0
ip link set dev eth0 up
ip route add default via 10.0.2.2 dev eth0
EOF
$ sudo chmod +x rootfs/etc/init.d/rcS
Enter fullscreen mode Exit fullscreen mode

Unmount our virtual disk:

$ sudo umount rootfs
Enter fullscreen mode Exit fullscreen mode

Now onto the exciting stuff!

Booting our RISC-V virt board

Let's run our emulator:

$ qemu-system-riscv64 \
    -nographic \
    -machine virt \
    -kernel linux-5.19.1/arch/riscv/boot/Image \
    -append "root=/dev/vda ro console=ttyS0" \
    -drive file=busybox,format=raw,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -netdev user,id=eth0 \
    -device virtio-net-device,netdev=eth0
Enter fullscreen mode Exit fullscreen mode

Most of the options will look familiar to you if you followed our last article, so we'll just cover what's new:

  • -kernel linux-5.19.1/arch/riscv/boot/Image: remember last time we specified the bootloader (Das U-Boot) here? This time we specify an actual kernel image, the one we just built. This skips the bootloader stage straight to the kernel, also know as direct kernel boot. It's required in this case since our BusyBox rootfs is not a bootable image
  • -append "root=/dev/vda ro console=ttyS0": here we append some kernel command-line options. For example, our rootfs at /dev/vda (as seen from within the VM) is mounted read-only (ro), and we specify the console

Since the BusyBox userspace is extremely lightweight, it should boot fully within about 1 second. Here's what you should see:

...
Hello Embedded World!
Hello RISC-V World!

Please press Enter to activate this console.
Enter fullscreen mode Exit fullscreen mode

Press Enter as prompted. You should drop into a root shell.

Let's play around. View the list of running processes:

# ps aux
Enter fullscreen mode Exit fullscreen mode

Example output:

PID   USER     TIME  COMMAND
    1 0         0:00 init
    2 0         0:00 [kthreadd]
    3 0         0:00 [rcu_gp]
    4 0         0:00 [rcu_par_gp]
    5 0         0:00 [netns]
    6 0         0:00 [kworker/0:0-eve]
    7 0         0:00 [kworker/0:0H-ev]
    8 0         0:00 [kworker/u2:0-ev]
    9 0         0:00 [mm_percpu_wq]
   10 0         0:00 [rcu_tasks_trace]
   11 0         0:00 [ksoftirqd/0]
   12 0         0:00 [rcu_sched]
   13 0         0:00 [migration/0]
   14 0         0:00 [kworker/0:1-eve]
   15 0         0:00 [cpuhp/0]
   16 0         0:00 [kdevtmpfs]
   17 0         0:00 [inet_frag_wq]
   18 0         0:00 [khungtaskd]
   19 0         0:00 [oom_reaper]
   20 0         0:00 [writeback]
   21 0         0:00 [kcompactd0]
   22 0         0:00 [kblockd]
   23 0         0:00 [ata_sff]
   24 0         0:00 [rpciod]
   25 0         0:00 [kworker/0:1H-ev]
   26 0         0:00 [xprtiod]
   27 0         0:00 [kswapd0]
   28 0         0:00 [kworker/u2:1-ev]
   29 0         0:00 [nfsiod]
   30 0         0:00 [uas]
   31 0         0:00 [mld]
   32 0         0:00 [ipv6_addrconf]
   39 0         0:00 [jbd2/vda-8]
   40 0         0:00 [ext4-rsv-conver]
   48 0         0:00 -/bin/sh
   49 0         0:00 init
   50 0         0:00 init
   51 0         0:00 init
   52 0         0:00 ps aux
Enter fullscreen mode Exit fullscreen mode

That's a really lightweight system! You can even run top to view processes and CPU usage in real time:

# top
Enter fullscreen mode Exit fullscreen mode

Press q to quit.

Let's see how much memory we're using versus what's available:

# free -m
Enter fullscreen mode Exit fullscreen mode

Example output:

              total        used        free      shared  buff/cache   available
Mem:            108          10          95           0           3          95
Swap:             0           0           0
Enter fullscreen mode Exit fullscreen mode

Again, extremely lightweight - the VM has about 128MB memory available, and we only used 10MB. Compare that with Ubuntu 22.04 server on RISC-V that fails to boot with 512MB memory due to insufficient memory ;-)

Check how much disk space is used / available:

# df -Th
Enter fullscreen mode Exit fullscreen mode

Example output:

Filesystem           Type            Size      Used Available Use% Mounted on
/dev/root            ext4          973.4M      1.7M    904.5M   0% /
devtmpfs             devtmpfs       53.0M         0     53.0M   0% /dev
Enter fullscreen mode Exit fullscreen mode

We allocated 1GB for our virtual disk, and the BusyBox rootfs takes less than 2MB.

Let's also test the network. Try pinging the host:

# ping -c5 10.0.2.2
Enter fullscreen mode Exit fullscreen mode

Example output:

PING 10.0.2.2 (10.0.2.2): 56 data bytes
64 bytes from 10.0.2.2: seq=0 ttl=255 time=16.085 ms
64 bytes from 10.0.2.2: seq=1 ttl=255 time=3.775 ms
64 bytes from 10.0.2.2: seq=2 ttl=255 time=3.810 ms
64 bytes from 10.0.2.2: seq=3 ttl=255 time=3.770 ms
64 bytes from 10.0.2.2: seq=4 ttl=255 time=4.322 ms

--- 10.0.2.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 3.770/6.352/16.085 ms
Enter fullscreen mode Exit fullscreen mode

Great! Let's see if we can reach servers from the Internet. Here's a public IP address for google.com: 142.250.204.46:

# ping -c5 142.250.204.46
Enter fullscreen mode Exit fullscreen mode

Example output:

PING 142.250.204.46 (142.250.204.46): 56 data bytes
64 bytes from 142.250.204.46: seq=0 ttl=255 time=15.262 ms
64 bytes from 142.250.204.46: seq=1 ttl=255 time=8.313 ms
64 bytes from 142.250.204.46: seq=2 ttl=255 time=8.136 ms
64 bytes from 142.250.204.46: seq=3 ttl=255 time=11.132 ms
64 bytes from 142.250.204.46: seq=4 ttl=255 time=8.752 ms

--- 142.250.204.46 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 8.136/10.319/15.262 ms
Enter fullscreen mode Exit fullscreen mode

Let's also view information on the CPU through procfs:

# cat /proc/cpuinfo
Enter fullscreen mode Exit fullscreen mode

Output:

processor   : 0
hart        : 0
isa     : rv64imafdc
mmu     : sv57
Enter fullscreen mode Exit fullscreen mode

So our emulated board has a single RISC-V CPU core with a single hardware thread (hart), and the CPU core supports the RV64IMAFDC ISA specification, where "IMAFD" can be simplified to just "G" to give RV64GC ("G" for general extensions, "C" for compressed instructions). You can read more about the RISC-V ISA specification on GitHub, which is highly modular with a minimal base ISA plus many optional extensions.

Play around a bit more, then power down the board:

# poweroff
Enter fullscreen mode Exit fullscreen mode

That's it - congratulations! You've successfully compiled your own Linux kernel and minimal BusyBox userspace, and booted it on a virtual RISC-V virt board with QEMU.

Next steps

If this article had you craving for more, here are a few things you could try to further your adventure. The list is by no means exhaustive:

  • If you played around a bit more, you might've noticed that DNS resolution does not work inside the board. Try to figure out why and fix it
  • Try following this article (or the official RISC-V docs) with a real, physical RISC-V SoC. After all, physical hardware is the real deal
  • If you manage to boot your own embedded Linux on physical RISC-V hardware, try doing something useful with it, such as making it an IoT project as part of your smart home

Top comments (2)

Collapse
 
grafeno30 profile image
grafeno30 •

excellent tutorial, it is also very educational. Thanks, it's a good contribution!!

Collapse
 
donaldsebleung profile image
Donald Sebastian Leung •

Thanks for your encouraging words, and glad you found it educational :-)

Read next

opensourcee profile image

The evolution of Web Development: from dial-up to instant React

OpenSourcee -

prometheus_78 profile image

Use Spotify made for Developers

tanish vashisth -

fernandezbaptiste profile image

πŸš€ 5 PRO Tips for an Unbeatable GitHub README! πŸ₯Š

Bap -

bogomil profile image

The "Modern email" devroom at FOSDEM is looking for proposals.

Bogomil Shopov - Π‘ΠΎΠ³ΠΎ -