DEV Community: Amarjargal

Testing ublk on Ubuntu 22.04

Amarjargal — Mon, 16 Jan 2023 06:45:25 +0000

ublk is a generic framework for implementing block device logic in the userspace. It's developed by our mentor Ming Lei. Here is an article that described the workings and future possibilities of ublk. Reading all about its possibilities, I'm super excited to work with this project.

ublk is comprised of two parts: kernel ublk driver which is newly introduced to the Linux kernel since version 6.0 and userspace daemon (ublksrv).

When testing ublk driver on a Fedora 37 VM, it worked out of the box. But those who want to use it on Ubuntu, for now, will have to update their kernel to version 6.0 or up and configure it to enable ublk driver. I have Ubuntu 22.04 Jammmy Jellyfish and kernel 5.15. So the first step is to configure and build the kernel.

1. Configure and build the Linux kernel

Download the latest stable kernel from kernel.org. The latest kernel as of this writing was 6.1.4.

Extract it:

tar xvf linux-6.1.4.tar.xz

Go to the extracted directory and copy the current kernel's configuration:

cd linux-6.1.4
cp -v /boot/config-$(uname -r) .config

The new kernel will have many new config options. To keep the old configuration and also to configure the new options to their default values, run the following:

make olddefconfig

Configure ublk_drv by running the GUI menuconfig, navigating to Device drivers -> Block devices and add M to the Userspace block driver (Experimental) to make it a loadable module. Click Save and Exit.

make menuconfig

Disable the conflicting security certificates. Without disabling them, the make will fail with error. This might not be the best solution, but the easiest one and serves my purpose.

scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS

Build and install. Use -j option to speed up the build process as it takes a lot of time. nproc will return the number of cores you can use.

make -j16
make modules
sudo make modules_install
sudo make install

Update grub bootloader:

sudo update-grub

Reboot

2. Build and install liburing

libiring is library for the new Linux asynchronous IO interface called io_uring. This is the interface ublk uses and will require version 2.2 or up.

You can install liburing with apt as shown below, but it was not the latest version.

sudo apt install liburing2 liburing-dev

Therefore, I just built it from the cloned source.

$ git clone https://github.com/axboe/liburing
$ cd liburing
$ ./configure
$ make
$ sudo make install

3. Build ubdsrv (userspace daemon)

git clone https://github.com/ming1/ubdsrv.git
autoreconf -i
./configure
make

4. Try ublk

First, load ublk_drv as a module. After it's loaded, a device called /dev/ublk_control is created. Userspace daemon will communicate with this device.

sudo modprobe ublk_drv

Then I tried ublk by following the Quick start tutorial on ubdsrv's source page. For now, ublk commands need root permission to run.

null disk

$ sudo ublk add -t null
dev id 0: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 524288000
    max rq size 524288 daemon pid 235918 flags 0x2 state LIVE
    queue 0: tid 235919 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"dev_size":268435456000,"name":"null","type":0}

After the above command executed, ublk block device ublkb0 and a char device ublkc0 is added by the ublk driver in /dev/

$ ls /dev/ublk
ublkb0        ublkc0        ublk-control

loop device

A raw disk image created with qemu-img is used.

$ qemu-img create -f raw foo.img 4G
$ sudo ublk add -t loop -f foo.img 
[sudo] password for amar: ublk
dev id 1: nr_hw_queues 1 queue_depth 128 block size 4096 dev_capacity 8388608
    max rq size 524288 daemon pid 329999 flags 0x2 state LIVE
    queue 0: tid 330000 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"backing_file":"foo.img","dev_size":4294967296,"direct_io":1,"name":"loop","type":1}
$ ls /dev/ublk
ublkb0        ublkb1        ublkc0        ublkc1        ublk-control

You can format the block device /dev/ublkb1 with xfs, mount it and can do anything you want with it.

$ sudo mkfs.xfs /dev/ublkb0 
meta-data=/dev/ublkb0            isize=512    agcount=4, agsize=262144 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=1048576, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.
$ sudo mount /dev/ublkb0 /mnt/tmp/
$ sudo mkdir /mnt/tmp/test
$ ls /mnt/tmp/
test
$ sudo umount /mnt/tmp

qcow2 disk

qcow2 support is still experimental, but I was able to add the Fedora disk image created in my previous blog post.

$ sudo ublk add -t qcow2 -f ../fedora/fedora37.qcow2 
dev id 2: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 62914560
    max rq size 524288 daemon pid 243562 flags 0x2 state LIVE
    queue 0: tid 243564 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"backing_file":"../fedora/fedora37.qcow2","cluster_bits":16,"dev_size":32212254720,"header_length":112,"l1_size":60,"name":"qcow2","refcount_order":4,"refcount_table_clusters":1,"type":2,"version":3}

ublkb2p1, ublkb2p2 and ublkb2p3 are added for each partition of the disk. Then we can mount the main partition and access the filesystem.

$ ls /dev/ublk
ublkb0        ublkb1        ublkb2        ublkb2p1      ublkb2p2      ublkb2p3      ublkc0        ublkc1        ublkc2        ublk-control
$ sudo mount /dev/ublkb2p3 /mnt/tmp/
$ ls /mnt/tmp/
afs  bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
$ sudo umount /mnt/tmp

List all the devices:

$ sudo ublk list 
dev id 0: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 524288000
    max rq size 524288 dae$ sudo umount /mnt/tmpmon pid 235918 flags 0x2 state LIVE
    queue 0: tid 235919 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"dev_size":268435456000,"name":"null","type":0}
dev id 1: nr_hw_queues 1 queue_depth 128 block size 4096 dev_capacity 8388608
    max rq size 524288 daemon pid 329999 flags 0x2 state LIVE
    queue 0: tid 330000 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"backing_file":"foo.img","dev_size":4294967296,"direct_io":1,"name":"loop","type":1}
dev id 2: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 62914560
    max rq size 524288 daemon pid 243562 flags 0x2 state LIVE
    queue 0: tid 243564 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
    target {"backing_file":"../fedora/fedora37.qcow2","cluster_bits":16,"dev_size":32212254720,"header_length":112,"l1_size":60,"name":"qcow2","refcount_order":4,"refcount_table_clusters":1,"type":2,"version":3}

Remove all devices:

sudo ublk del -a

QEMU Storage Daemon

Amarjargal — Fri, 06 Jan 2023 17:34:57 +0000

qemu-storage-daemon is a tool that provides disk image functionality for a VM without running the VM itself. It can export disk images, run block job operations, and perform other disk-related operations. This article was very helpful to get a better understanding of the QEMU storage daemon.

Here, I used two of the export methods for qsd to access the Fedora disk image that was created in the previous blog post.

1. Fuse export

FUSE (Filesystem in Userspace) is an interface for userspace programs to export a filesystem to the Linux kernel.

There is a really nice tutorial on FUSE export on the QEMU website which I followed through.

First of all, install some dependencies:

sudo apt install libfuse3-3 libfuse3-dev

Then configure qemu with --enable-fuse option and build it again.
```
./configure --enable-fuse
```
Also, uncomment user_allow_other in /etc/fuse.conf. By default, only user who exported the image will have an access to it. Enabling user_allow_other option here will allow non-root users to be able to use allow_other mount option which enables users other than the owner to be able to access the mounted fuse image.
Create a file that will be a mount point where the filesystem will be mounted on. It's also possible to mount the image on itself.
```
touch mount-point
```

Export:

qemu-storage-daemon \
   --blockdev node-name=prot-node,driver=file, 
filename=fedora37.qcow2   \
   --blockdev node-name=fmt-node,driver=qcow2,file=prot-node   
   --export type=fuse,id=exp0,node-name=fmt-node, 
mountpoint=mount-point,writable=on

After the above command is executed, the mount-point will appear as a raw VM disk. Then, we'll be able to use tools like parted, kpartx, file,mount as they can't parse qcow2 images and can only work with raw images.

$ qemu-img info mount-point
image: mount-point
file format: raw
virtual size: 30 GiB (32212254720 bytes)
disk size: 1.58 GiB

List the partition table:

$ parted mount-point p
WARNING: You are not superuser.  Watch out for permissions.
Model:  (file)
Disk /home/amar/projects/outreachy/qemu/fedora/mount-point: 32.2GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  2097kB  1049kB               primary  bios_grub
 2      2097kB  1076MB  1074MB  xfs          primary
 3      1076MB  32.2GB  31.1GB  xfs          primary

Add device mappings over partitions:

$ sudo kpartx -av mount-point 
[sudo] password for amar: 
add map loop31p1 (253:6): 0 2048 linear 7:31 2048
add map loop31p2 (253:7): 0 2097152 linear 7:31 4096
add map loop31p3 (253:8): 0 60810752 linear 7:31 2101248

Mount the main partition:

$ sudo mount /dev/mapper/loop31p3 /mnt/tmp/
$ ls /mnt/tmp/
afs  boot  etc   lib    media  opt   root  sbin  sys  usr
bin  dev   home  lib64  mnt    proc  run   srv   tmp  var

Unmount the partition after done using it.
```
sudo umount /mnt/tmp
```
Stop qsd by just hitting Ctrl+C or if it's running in the background, just kill the process.

2. NBD export

NBD (Network Block Device) is a simple protocol that allows users to export a block device to a client over a network. Here is a nice introduction of NBD as suggested by my mentors.

First, load NBD module:
```
sudo modprobe nbd
```

Export the image file over NBD. With the below settings, NBD server will start running on localhost port 10809.

qemu-storage-daemon     \
    --blockdev node-name=prot-node,driver=file,
filename=fedora37.qcow2     \
    --blockdev node-name=fmt-node,driver=qcow2,
file=prot-node     \
    --nbd-server addr.type=inet,addr.host=localhost,
addr.port=10809     \
    --export type=nbd,id=exp0,node-name=fmt-node,
name=guest-disk,writable=on

Mount the NBD exported device as a local device /dev/nbd0 by starting the nbd-client. For -N or -name option, use the name provided in the export command above. If this option is not supplied, nbd-client will complain about oldstyle negotiation.
```
$ sudo nbd-client localhost 10809 /dev/nbd0 -N guest-disk
[sudo] password for amar: 
Negotiation: ..size = 30720MB
Connected /dev/nbd0
```
Block devices for each partition of the original image will be automatically created.
```
$ ls /dev/nbd0<tab>
nbd0    nbd0p1  nbd0p2  nbd0p3
```

Mount the main partition

$ sudo mount /dev/nbd0p3 /mnt/tmp/
$ ls /mnt/tmp/
afs  boot  etc   lib    media  opt   root  sbin  sys  usr
bin  dev   home  lib64  mnt    proc  run   srv   tmp  var

After finished, unmount and disconnect. Proper clean-up is important because if not done correctly, the next export will not work.
```
umount /mnt/tmp
nbd-client -d /dev/nbd0
```

Running Fedora 37 on QEMU x86_64 machine

Amarjargal — Fri, 06 Jan 2023 17:33:43 +0000

In my previous post, I played with QEMU by installing Debian on an emulated arm machine. It was challenging and I ended up spending a lot of time for it. But I think it was a good exercise nevertheless.

However, as my mentors suggested, it's much better to use the same QEMU target architecture as your host. Therefore, here I'll describe the steps I followed to run Fedora 37 on QEMU x86_64 machine.

But first, here's a little bit about KVM.

What is KVM and how does it relate to QEMU?

KVM (Kernel-based Virtual Machine) is an open source full virtualization solution for Linux on x86 machine with virtualization hardware extension (Intel VT or AMD-V). To see if the virtualization extension exists, run the following to look for svm (for AMD-V) or vmx (for Intel VT) flags.

grep -E 'svm|vmx' /proc/cpuinfo

On my computer, the result contains svm which means I have a AMD-V.

To see if KVM is enabled, run the following:

Hypervisor is a software that provides the virtualization to the underlying hardware; it creates and runs Virtual Machines. This IBM tutorial was a great read to understand about virtualization and hypervisors. KVM is a type 1 (bare metal) hypervisor which means that it has a direct access to the hardware resources.

Whereas, QEMU is a type 2 hypervisor which means it runs on top of the host OS. QEMU is also tightly integrated with KVM. If KVM is enabled on the host, and guest and host OS have the same architecture, QEMU can have KVM to translate some of the instructions (CPU and memory calls) making it much faster.

Source: Researchgate

Libvirt

Libvirt is collection of software that provides a convenient way to manage virtual machines and other virtualization functionality.
Tools such as virt-builder, virt-install and virt-manager that are based on libvirt make building and installing virtual machines much easier.

Virt-builder

Virt-builder is a tool a tool that quickly builds virtual machine images. It's part of libguestfs which is a set of tools to build, access, modify virtual machine disk images and more.

First, install libguestfs.

sudo apt install libguestfs-tools

The following will list all the virtual machine images you can create:

virt-builder --list

Build Fedora image. The size specified with -size 30G will not be the actual size of the image. It just means that the newly created image can be expanded up to 30G.

virt-builder  \
    --ssh-inject=root:file:/home/amar/.ssh/id_rsa.pub  \ 
    --root-password=password:vm_root_password   \
    --output=fedora37.qcow2 --format=qcow2   \
    --selinux-relabel --size 30G fedora-37

Virt-install and Virt-manager

Create a VM using the disk image created above with virt-builder:

virt-install 
    --connect qemu:///system --name f37-vm–import \
    --ram 2048 --vcpus 2 --cpu host   \
    --disk bus=virtio,path=fedora37.qcow2   \ 
    --network network=default,model=virtio  \
    --os-variant fedora37

Or use virt-manager which is a GUI tool that allows you to create a VM with one-click.

However, when I try to use virt-install or virt-manager, I got the following error even though virtualization hardware is present and enabled.

ERROR Host does not support any virtualization options

Direct install

I had a network problem when I tried direct install with qemu-system-x86_64. Therefore, first install slirp, configure and build QEMU again with --enable-slirp.

sudo apt install libslirp-dev
../configure --enable-slirp

Then directly run VM with QEMU:

qemu-system-x86_64 -smp 2 -m 1G -M q35,accel=kvm   \
    -drive file=fedora37.qcow2,format=qcow2,if=virtio  \   
    -nic user,model=virtio

When the VM runs, login using the root password set when building the disk image.

Running Debian on a QEMU emulated arm machine

Amarjargal — Mon, 12 Dec 2022 05:21:20 +0000

Here are the steps I followed to play with QEMU to get myself acquainted with it. My host is Ubuntu 20.04.1.

1. Building QEMU

There are several ways to download and build QEMU. You can install the pre-packaged version on Ubuntu with apt or download and build the latest release from the QEMU website.

But to have the latest release and stay up to date, you can clone the git repository which is the way I'm doing.

Install the required additional packages as instructed here. I also installed the recommended additional packaged as well.

sudo apt-get install git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev ninja-build

Clone the source code:

git clone git://git.qemu-project.org/qemu.git

Initialize and update the submodules.

cd qemu
git submodule init
git submodule update --recursive

Create a new build directory:

mkdir build
cd build

Configure and build. If you want to build for a specific target(s) only which is faster, just add a --target-list option. Run ../configure --help to see the list of available targets and other configuration options.

../configure
../configure --target-list=arm-softmmu

Build and if necessary, test which takes a lot of time.

make
make check
make install

2. Installing Debian

Download the arm disk image from Debian or using wget from the server as shown below. It's the current version of Debian that will run on ARM machines. At the time of this post, the current version was codenamed bullseye.

wget http://ftp.debian.org/debian/dists/stable/main/installer-arm64/current/images/netboot/mini.iso

Download also the vmlinux kernel and initrd.gz initramfs.

wget http://ftp.debian.org/debian/dists/stable/main/installer-arm64/current/images/netboot/debian-installer/arm64/initrd.gz
wget http://ftp.debian.org/dget http://ftp.debian.org/debian/dists/stable/main/installer-arm64/current/images/netboot/mini.iebian/dists/stable/main/installer-arm64/current/images/netboot/debian-installer/arm64/linux

There are 3 different arm ports available: arm64, armhf and armel which are explained here in detail. Basically, armel (arm EABI) is for older architecture versions (4T, 5T and 6), armhf (arm hard float) is for floating point support and arm64 for 64-bit ARMv8 architecture which I'll use.

qemu-system- executables built in the previous step are for simulating different machines. qemu-system-aarch64 will simulate arm64 machine.
The version used here is as shown below.

$ qemu-system-aarch64 --version
QEMU emulator version 7.1.93 (v7.2.0-rc3)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developersurl

Create a hard disk image for our system by running qemu-img command. qcow2 (QEMU copy on write) is a disk image file format native to the QEMU.

qemu-img create -f qcow2 debian-arm.sda.qcow2 5G

Start the installation process:

../qemu/build/qemu-system-aarch64 -M virt -cpu cortex-a53 -m 1G -kernel ./linux -initrd ./initrd.gz -hda debian-arm.sda.qcow2 -append "console=ttyAMA0" -drive file=./mini.iso,id=cdrom,if=none,media=cdrom -device virtio-scsi-device -device scsi-cd,drive=cdrom -nographic

After a few seconds, the kernel boots and Debian installer will start. You can just jump through the screens to configure your language, location, locale, keymap etc.

But I encountered a network set-up problem and therefore couldn't access the Debian archive mirror. I'll look into it and post again.

Small tip: I didn't know how to exit the installer. Going back and choosing "Aborting the installer" option only reboot and restarted the installer. Instead you can choose "Execute a shell" and then type poweroff.

Update

After a bit of research, it looks like setting up network on Debian is tricky. It could be a missing device driver/firmware since most of the firmware images are non-free, they are not included in the official Debian installation images. Unofficial images including non-free firmware are available, but there were none available for arm.

Therefore I decided to use the DVD image that doesn't require network during the installation. I loosely followed the instructions this blog.

Again, download the necessary files.

  curl -O http://ftp.us.debian.org/debian/dists/stable/main/installer-arm64/current/images/cdrom/initrd.gz
  curl -O http://ftp.us.debian.org/debian/dists/stable/main/installer-arm64/current/images/cdrom/vmlinuz
  curl -O -L https://cdimage.debian.org/debian-cd/current/arm64/iso-dvd/debian-11.6.0-arm64-DVD-1.iso

Boot the VM to install Debian.
debian-arm.sda.qcow2

qemu-system-aarch64 -m 4G -machine type=virt -cpu cortex-a53 -initrd "./initrd.gz" \
  -kernel "./vmlinuz" -append "console=ttyAMA0" \
  -drive file="./debian-11.6.0-arm64-DVD-1.iso", id=cdrom,if=none,media=cdrom \
    -device virtio-scsi-device \
    -device scsi-cd,drive=cdrom \
  -drive file="./debian-arm.sda.qcow2", id=hd,if=none,media=disk \
    -device virtio-scsi-device \
    -device scsi-hd,drive=hd \
  -nographic

The network setup will fail again, but you can choose Do not configure the network at this time and the installation will run smoothly after that.

When prompted about the bootloader, you can just choose Continue.

After installation finishes, don't click on the Continue because it'll start the installer all over again. Rather, choose Go back, Execute a shell and then type poweroff.

First week of my internship with QEMU through Outreachy

Amarjargal — Mon, 12 Dec 2022 05:17:35 +0000

This is the first week of my internship with QEMU through Outreachy.

What is Outreachy?

Outreachy is an amazing opportunity for those who are underrepresented in the tech industry to work on open source and open science projects under the guidance of experienced mentors. I think most of the applicants are female and are students, but really it's open for everyone who feel disadvantaged to enter the industry.

For myself, I'm a mother of two little boys and I've been putting my career on hold ever since I became a mother. When I wanted to get back to the industry, I needed a little more flexibility which is difficult to get accommodated in the traditional style workplaces in my home country. Therefore I'm so grateful for Outreachy for providing me this opportunity to work remotely on such an interesting project as QEMU.

What is QEMU?

QEMU is an open source machine emulator. It can emulate plethora of architectures ranging from x86, ARM, PowerPC, you name it, it probably can emulate. It can emulate full system such as a Xilinx development board or a Raspberry PI.

For the duration of the internship I'll work on a project "Extend QEMU Storage Daemon, supporting the new Linux's userspace block driver (ublk) framework".

I hope to update here frequently about what I learned and share the knowledge with others who are new to QEMU the same as me.