DEV Community: Akira Moroo

9pfsPkg: Network Boot from Bell Labs

Akira Moroo — Sat, 01 Aug 2020 07:11:08 +0000

I developed a Plan 9 file system (9P) client for UEFI to enable network booting from a commodity 9P server. By leveraging the simplicity and flexibility of 9P, the UEFI can do network boot from cloud storage without any effort. This blog post gives you a brief overview of 9pfsPkg.

The source code and introduction slides are available at:

yabits / 9pfsPkg

9P Client File System for UEFI

9pfsPkg

9pfsPkg is a Plan 9 file system protocol (9P) client for UEFI. It provides EFI_SIMPLE_FILE_SYSTEM_PROTOCOL interface for network transparent file system operation.

License

9pfsPkg is released under the BSD-2-Clause Plus Patent License.

View on GitHub

What is Network Boot?

Network boot is a boot method which loading boot images over the network. To make this possible, the BIOS has its network stack.
There are two methods for network booting: PXE Boot and HTTP Boot.

PXE (Pre-eXecution Environment) Boot is the most widely used method as it exists from the Legacy BIOS era. It is standardized and implemented as not only proprietary but also open source. PXE boot uses TFTP to transfer files. This protocol is not popular, so it requires a dedicated TFTP server.

HTTP Boot uses HTTP for transferring images. It has been standardized from UEFI 2.5 in 2015[0]. It supports modern features like DNS and TLS. Since it uses HTTP, we can use commodity HTTP servers (e.g. Apache HTTP Server, Nginx).

Below is the interface of the HTTP protocol.

Configure() sets the configuration, Request() sends a request, and Response() receives a response. By using these functions, we can implement HTTP Boot bootloader like this:

EFI_STATUS HttpBootLoader()
{
  // Send request
  Status = Http->Request (Http, TxToken);
  // Recieve response
  Status = Http->Response (Http, RxToken);
  // Start loaded image
  Status = gBS->StartImage (ImageHandle, NULL, NULL);
  return Status;
}

This example shows how it is easy to boot with HTTP on UEFI.

UEFI is Extensible

UEFI is an abbreviation of the Unified Extensible Firmware Interface. As it includes the word "Extensible," it has a modular design. The modules are called "Protocol" and UEFI has features that load external protocols in its core. By calling EFI_BOOT_SERVICES.InstallProtocolInterface() with passing a loaded protocol to Interface argument, it installs an external protocol.

As an example of a UEFI protocol, I introduce the Simple File System Protocol. This protocol provides a file system independent file operation interfaces. Here is a figure of the interfaces.

OpenVolume() in Simple File System Protocol opens a volume and returns File Protocol Root that represents the root directory. File Protocol provides file operation functions like Open(), Read(), Write().

However, even it has abstract interfaces, UEFI supports the FAT file system only by default. There are some third-party non-FAT file system drivers. Here is an example of the use of such a file system driver: UEFI Rootkits. A rootkit is malware that targets kernels or firmware. Once infected, it installs other rootkits and/or agents. Hacking Team's rkloader[2] and LoJax[3] are such UEFI rootkits. They have NTFS UEFI drivers to embed kernel rootkits to the target Windows system. This driver is a port of NTFS-3G, an open-source NTFS implementation, and has Simple File System Protocol as an interface. The following snippet from the rkloader shows how Simple File System Protocol makes embedding an agent easy.

EFI_STATUS
EFIAPI
InstallAgent(
  IN EFI_FILE_HANDLE CurDir,
  IN CHAR16 *  FileNameUser
  )
{
  // Open FileNameScout as FileHandle
  Status = CurDir->Open (CurDir, &FileHandle, FileNameScout, EFI_FILE_MODE_READ|EFI_FILE_MODE_WRITE|EFI_FILE_MODE_CREATE, 0);
  // Write pSectiondata to FileHandle
  Status = FileHandle->Write(FileHandle,&VirtualSize,(UINT8*)(pSectiondata));
  // Close FileHandle
  Status = FileHandle->Close(FileHandle);
  return EFI_SUCCESS;
}

First of all, open a file by Open(), deploy the agent by Write(), and close the file with Close(). As you can see, it is simple and does not have any trick.

Plan 9 File Protocol

Now, let's think about network boot. I pointed out that the current network boots are network-aware and less flexibility. Thus, we want a network transparent file system and protocol for network boot while maintaining the feasibility of the file system.

Here is the answer: Plan 9 File Protocol (9P)[8]

Plan 9 from Bell Labs (Plan 9)[7] is a Unix successor OS developed by the original Unix developer in Bell Labs. "Everything is a file.", which is a well known Unix philosophy, is a core design decision on Plan 9. P9 is a protocol developed by Plan 9 developers to deal with remote files in the same manner as local files. Below is a flow of loading a file with 9P.

The client negotiates by version and connects with attach to get a file descriptor of the root directory. walk searches to the target file, open it, and read it. As you can see, 9P is a protocol that file operations and messages correspond to one-by-one.

9P is popular in the fields of not related to Plan 9 due to clarity and simplicity. For instance, the Linux kernel has a 9P client file system called v9fs[4]. VirtIO has a virtio-9p 9P server to share the host file system with the guests[5].

Recently, Microsoft has released Windows 10 update, and Windows Subsystem for Linux 2 (WSL2) is now officially supported. It runs a guest Linux on the Hyper-V VM in contrast to WSL1. Because VM disk image is a monolithic file, it is hard to access inside files with the same manner of host file access. To solve this issue, WSL2 uses 9P to access guest files from the host. The host Windows has 9P client to access the guest Linux files. The guest has a 9P server to process requests from the host to share the files[6].

9pfsPkg

As I described in the previous section, 9P is still widely used nowadays. I implemented a 9P client file system for UEFI: 9pfsPkg.

yabits / 9pfsPkg

9P Client File System for UEFI

9pfsPkg

9pfsPkg is a Plan 9 file system protocol (9P) client for UEFI. It provides EFI_SIMPLE_FILE_SYSTEM_PROTOCOL interface for network transparent file system operation.

License

9pfsPkg is released under the BSD-2-Clause Plus Patent License.

View on GitHub

9pfsPkg is a 9P client file system UEFI driver with a Simple File System Protocol interface. Because 9P is a network transparent file system, we can use existing non-network-aware tools (e.g. UEFI Shell) for file operations via networks without any modification by using 9pfsPkg. Another advantage of the file system is that it does not require dedicated servers (like TFTP in PXE Boot).

9P Boot

Let's take a look at a boot by 9P (9P Boot). The below shows the overview.

First of all, the 9P service runs on the server with an exported directory (e.g. /tmp/9). Next, the client loads the 9pfsPkg UEFI driver to create a new volume. The driver processes operations to the file system volume and communicates with the server via the UEFI network stack to handle the file operations.

The below video clip shows what it looks:

retrage

@retrage

I'm making 9pfsPkg, 9P client for UEFI in Simple File System Protocol manner. It can boot GRUB from the remote server. The code will be available later.

14:55 PM - 18 May 2020

6 12

When the UEFI startups, we can see the local file system FS0: only.

Mapping table
      FS0: Alias(s):HD0a65535a1:;BLK1:
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)/HD(1,MBR,0xBE1AFDFA,0x3F,0xFBFC1)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)

Load 9pfsPkg UEFI driver by load 9pfs.efi.

FS0:\> load 9pfs.efi
Image 'FS0:\9pfs.efi' loaded at 7E2E7000 - Success

The new file system FS1: has appeared.

Mapping table
      FS0: Alias(s):HD0a65535a1:;BLK1:                                          
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)/HD(1,MBR,0xBE1AFDFA,0x3F,0xFBFC1)
      FS1: Alias(s):F1:
          PciRoot(0x0)/Pci(0x2,0x0)/MAC(525400123456,0x1)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)

In contrast to local FS0:, the device path of remote FS1: is PciRoot(0x0)/Pci(0x2,0x0)/MAC(525400123456,0x1). It represents that the volume is on the remote server.

By executing fs1: and grubx64.efi, it boots the bootloader GRUB.

FS0:\> fs1:
FS1:\> grubx64.efi
                             GNU GRUB  version 2.02

   Minimal BASH-like line editing is supported. For the first word, TAB
   lists possible command completions. Anywhere else TAB lists possible
   device or file completions.


grub>

At this point, UEFI Shell and GRUB deal with the remote files in the same manner as local files. There is no network boot specific process.

Proxy Boot

9P Boot enables non-network-aware network boot. To take more advantages of the 9P, I propose Proxy Boot as an application of 9P Boot. It can boot from other servers via the direct server as a proxy. By using Proxy Boot, UEFI can boot from cloud storage without any effort. Following is the overview of Proxy Boot.

I used Google Cloud Storage (GCS) for the network boot. The storage bucket has boot images. The server mounts the bucket as a file system using gcsfuse[9]. The 9P server uses the gcsfuse's mount point (e.g. /mnt/gcs) as an exported directory. The client mounts the volume in the same manner as 9P Boot. The client UEFI can treat the cloud storage files as if local files.

The below is the demo:

retrage

@retrage

I confirmed network boot BitVisor (thin-hypervisor) from Google Cloud Platform Storage via the 9P server with less effort. The 9P client for UEFI, 9pfsPkg is available at github.com/yabits/9pfsPkg

10:18 AM - 02 Jun 2020

14 25

Create a GCS bucket and upload boot images. I used BitVisor (thin-hypervisor) as a practical boot image. loadvmm.efi is the loader, and bitvisor.elf is the actual BitVisor image. They are default build and no modification for the network boot.

Next, mount the bucket on the server using gcsfuse at /mnt/gcs mount point.

$ sudo -E gcsfuse proxy-boot /mnt/gcs
 Using mount point: /mnt/gcs
Opening GCS connection...
Opening bucket...
Mounting file system...
File system has been successfully mounted.

On the client, load the driver with load 9pfs.efi, move to fs1:, and call loadvmm.efi to boot BitVisor.

Shell> fs0:
FS0:\> load 9pfs.efi
FS0:\> map -u
FS0:\> fs1:
FS1:\> loadvmm.efi
Starting BitVisor...
Copyright (c) 2007, 2008 University of Tsukuba
All rights reserved.

Once again, the UEFI Shell and loadvmm.efi operates the cloud storage files as if local files, and there is no cloud-specific process.

Conclusion

In this blog post, I pointed out that the existing network boots are network-aware and less flexible. The 9P client file system for UEFI (9pfsPkg) enables network transparent network boot (9P Boot). As an application of the 9pfsPkg, I proposed a network boot from cloud storage via the server (Proxy Boot) without any effort.

References

Linux Kernel Library Nabla Containers Internals

Akira Moroo — Mon, 11 May 2020 09:49:46 +0000

This post describes the design and implementation of Linux Kernel Library Nabla Containers (LKL Nabla), Linux based unikernels as processes. The previous post introduces LKL Nabla and provides how to build and run. Since most of the unikernel work is done by frankenlibc LKL/musl, mainly focus on frankenlibc Solo5 port in this post.

You can find LKL Nabla code at:

Modifications to runnc

Before diving into frankenlibc code, let’s take a look at the modifications to runnc.

When runnc is executed, it initializes devices that will be used by the container. Then, the runtime builds arguments and launches a container as a process.

What kind of devices will be provided? On current runnc implementation, it can provide only one network device and block device correspondingly. This situation is the same in LKL Nabla.

A container manager like Docker pulls container an image and extracts to a rootfs as a directory. runnc creates a disk image from the rootfs directory. The disk image format is ISO in Rumprun, but the default file system is ext4 in LKL. Thus, it is switched to ext4 in LKL Nabla.
For the implementation, see CreateExt4().

Rumprun accepts JSON config from arguments on runtime. The original runnc builds config on container initialization. On the other hand, LKL also allows JSON config on runtime. However, the config format is quite different from Rumprun’s one. LKL Nabla’s runnc creates a config for LKL.
llruntimes/nabla/runnc-cont/lkl.go is the config builder for LKL.

After the initialization, runnc launches a unikernel process using Solo5 tender like:

    var args []string
    args = []string{r.NablaRunBin,
        "--mem=" + strconv.FormatInt(r.Memory, 10),
        "--net:tap=" + r.Tap,
        "--block:rootfs=" + disk,
        r.UniKernelBin}
    args = append(args, "__RUMP_FDINFO_NET_tap=4")
    args = append(args, r.Env...)
    args = append(args, "--config")
    args = append(args, unikernelArgs)
    args = append(args, "--")
  args = append(args, r.NablaRunArgs...)
  // snip
    err = syscall.Exec(r.NablaRunBin, args, newenv)

frankenlibc

Now, it’s time to dive into Solo5 port frankenlibc. It was a bunch of tools to run Rumprun unikernel on userspace. It was forked and added LKL/musl support. LKL Nabla uses this fork to run LKL on Solo5.

Below shows the architecture of frankenlibc.

frankenlibc Layers
Application
musl libc
LKL
librumpuser
franken
platform
Host

An application is the top of the 7 layers. The host is the bottom. The host-dependent layer is a platform. The code is located in platform directory. To port a new host, you will have to add the code to the platform.

The interfaces that platform code should provide are the same as Linux system calls. Here is the list.

int __libc_start_main(int (*main)(int,char **,char **), int argc, char **argv);
void _exit(int status);
int clock_getres(clockid_t clk_id, struct timespec *tp);
int clock_gettime(clockid_t clk_id, struct timespec *tp);
int clock_nanosleep(clockid_t clk_id, int flags, const struct timespec *request, struct timespec *remain);
int fcntl(int fd, int cmd, ...);
int fstat(int fd, struct stat *st);
int fsync(int fd);
int getpagesize(void);
int getrandom(void *buf, size_t size, unsigned int flags);
int kill(pid_t pid, int sig);
off_t lseek(int fd, off_t offset, int whence);
void *mmap(void *addr, size_t length, int prot, int nflags, int fd, off_t offset);
int munmap(void *addr, size_t length);
int mprotect(void *addr, size_t length, int prot);
int poll(struct pollfd *fds, nfds_t n, int timeout);
ssize_t pread(int fd, void *buf, size_t count, off_t offset);
ssize_t preadv(int fd, const struct iovec *iov, int iovcnt, off_t off);
ssize_t pwrite(int fd, const void *buf, size_t count, off_t offset);
ssize_t pwritev(int fd, const struct iovec *iov, int iovcnt, off_t off);
ssize_t read(int fd, void *buf, size_t count);
ssize_t readv(int fd, const struct iovec *iov, int iovcnt);
ssize_t write(int fd, const void *buf, size_t count);
ssize_t writev(int fd, const struct iovec *iov, int iovcnt);

It looks much larger than that of Solo5 as it provides only 13 hypercalls to the guest OS, but some of them are optional. We need to implement the platform code using the hypercalls for porting LKL/musl to Solo5.

Entry Point

solo5_start_main() is an entry point in Solo5 guest. A Solo5 tender starts the OS from this function. The argument is a pointer to struct solo5_start_info. It contains cmdline, heap_start and heap_size.

struct solo5_start_info {
    const char *cmdline;
    uintptr_t heap_start;
    size_t heap_size;
};

cmdline is an argument string passed when the unikernel process is launched. As frankenlibc expects envp and argv will be passed from the host, cmdline is parsed into envp and argv in the initialization.
rexec, a launch tool for frankenlibc, can pass a JSON config through a file descriptor. The FD value is shared using the environment variable (e.g. __RUMP_FDINFO_CONFIGJSON). However, this method cannot be applied to Solo5 port because any environment variable cannot be shared with Solo5 guests. Therefore, the JSON config is passed from cmdline as a string in the Solo5 port.

The other arguments heap_start and heap_size are information about heap provided by the tender. They are used for memory manager initialization. In this Solo5 port, the memory manager is a simple buddy allocator from mini-os. It is used in the mmap()/munmap() platform code.

Devices

In *nix system, most of the devices are represented as files and the operations are read/write to the file descriptor. frankenlibc also use this manner in platform code.
rexec opens devices and passes the FD numbers through environment variables (e.g. __RUMP_FDINFO_NET_tap). This behavior is the same as the JSON config. The franken layer registers devices using the FD info in fdinit().

In Solo5, devices attached at runtime must be specified at build time. When building a guest, a JSON format config called Application Manifest manifest.json must be supplied. It declares user-specified devices. In contrast to Solo5, frankenlibc rexec can specify devices at run time. As described before, current runnc can deal with one block device and one network device. Therefore, the Solo5 port uses fixed manifest.json that specifies one block device rootfs and one network device tap. Below is the config.

{
  "type": "solo5.manifest",
  "version": 1,
  "devices":
  [
    { "name": "rootfs", "type": "BLOCK_BASIC" },
    { "name": "tap", "type": "NET_BASIC" }
  ]
}

A device in Solo5 is represented by solo5_handle_t, not by the file descriptor. In the frankenlibc Solo5 port, as the devices are fixed, it assigns a virtual FD number to the Solo5 device handle.

Solo5 provides interfaces for reading/writing devices and console. In read()/write() platform code, it identifies the FD number and call appropriate hypercalls.

poll()/clock_nanosleep() are used for waiting network packets. Each network device has file descriptor pollfd to store polling state in frankenlibc. For Solo5 port, solo5_yield() is used to implement poll()/clock_nanosleep() The behavior is almost the same as the Linux port.

In clock_nanosleep(), it calls solo5_yield() and if the network handle is set on ready_set, it updates the pollfd.revents and wake the associated thread. In poll(), it sleeps until timeout and sets FD's revents if pollfd.revents is updated.

Conclusion

This post summarized LKL Nabla internals. The most of implementations are straight forward thanks to frankenlibc platform-independent interfaces and simple Solo5 hypercalls. However, since LKL has different interfaces with Rumprun, patches to runnc for LKL port is quite large. It will be better to have a switching option to change between Rumprun and LKL on runnc.

Porting Linux to Nabla Containers

Akira Moroo — Sat, 18 Apr 2020 08:20:05 +0000

This is an introduction of Linux Kernel Library ported to Nabla Containers.

runnc is an OCI runtime that runs process-level isolated unikernels. It is built on the top of Solo5, a sandbox for unikernels, and several unikernels (MirageOS, IncludeOS, Rumprun) run on it. The original runnc uses Rumprun, a NetBSD based unikernel. However, as Docker is started from Linux, it is needed to have system call level compatibility with Linux. Therefore, I ported Linux Kernel Library (LKL) and musl libc to Solo5 and put together with runnc.

frankenlibc on Solo5

frankenlibc is a set of tools to run Rump unikernels in various environments. It has a fork that ported LKL and some libraries. I used this frankenlibc fork and added Solo5 platform support.

https://github.com/retrage/frankenlibc/tree/solo5

Building frankenlibc

Clone the repository and checkout solo5 branch.

$ git clone https://github.com/retrage/frankenlibc.git
$ cd frankenlibc
$ git checkout solo5

Clone full Solo5 repository to avoid build failure and update submodules.

$ git clone https://github.com/Solo5/solo5.git
$ git submodule update --init

Apply some patches.

$ for file in `find patches/solo5/ -maxdepth 1 -type f` ; do patch -p1 < $file ; done

Finally, run the build script.

$ ./build.sh -k linux notests solo5

You can find libraries and toolchain wrappers in rump directory after building successfully.

Testing

Even if notests specified, build.sh builds simple tests to rumpobj/tests.

Create a tap100 tap device.

$ sudo ip tuntap add tap100 mode tap
$ sudo ip addr add 10.0.0.1/24 dev tap100
$ sudo ip link set dev tap100 up

Create disk.img disk image. As LKL/frankenlibc creates directories on initialization, some operations fail if read-only ISO image is used. To avoid this issue, we use the Ext4 file system image.

$ dd if=/dev/zero of=disk.img bs=1024 count=20480
$ mkfs.ext4 -F disk.img

Note that Solo5 requires an application manifest on build time, which is embedded in a unikernel binary. In current frankenlibc Solo5 support, the manifest is common across binaries and specifies rootfs block device and tap network device. We have to provide these devices even not used in the applications.

https://github.com/retrage/frankenlibc/blob/solo5/platform/solo5/manifest.json

Run hello test.

$ RUMP_VERBOSE=1 ./rump/bin/rexec rumpobj/tests/hello rootfs:disk.img tap:tap100

In the Linux platform, rexec provides a sandbox environment for unikernels using seccomp like Solo5's tenders. In the Solo5 platform, it is just a shell script wrapper for spt tender.

LKL Nabla Containers

Now, it's time to integrate with Nabla Containers. Since the original runnc imports older version of Solo5, I updated it and adapted the runnc code base.

https://github.com/retrage/runnc/tree/lkl-musl

Updating Supplied Arguments

Below is the original code that creates arguments for Solo5 tender.

    var args []string
    if mac != "" {
        args = []string{r.NablaRunBin,
            "--x-exec-heap",
            "--mem=" + strconv.FormatInt(r.Memory, 10),
            "--net-mac=" + mac,
            "--net=" + r.Tap,
            "--disk=" + disk,
            r.UniKernelBin,
            unikernelArgs}
    } else {
        args = []string{r.NablaRunBin,
            "--x-exec-heap",
            "--mem=" + strconv.FormatInt(r.Memory, 10),
            "--net=" + r.Tap,
            "--disk=" + disk,
            r.UniKernelBin,
            unikernelArgs}
    }

In the latest Solo5 (frankenlibc Solo5 platform uses), --net-mac option is removed and we can specify multiple block devices and network devices with --block: and --net: options. Ideally, it should support multiple devices. However, as described before, it can specify rootfs and tap only. So, the port ends up with the support of these devices like this.

    var args []string
    args = []string{r.NablaRunBin,
        "--mem=" + strconv.FormatInt(r.Memory, 10),
        "--net:tap=" + r.Tap,
        "--block:rootfs=" + disk,
        r.UniKernelBin}

Creating Disk Image

I added CreateExt4() function and llmodules/fs/ext4_storage.go to create Ext4 rootfs.

// CreateExt4 creates ext4 raw disk image from the dir argument
func CreateExt4(dir string, target *string) (string, error) {
    var fname string

    if target == nil {
        f, err := ioutil.TempFile("/tmp", "nabla")
        if err != nil {
            return "", err
        }

        fname = f.Name()
        if err := f.Close(); err != nil {
            return "", err
        }
    } else {
        var err error
        fname, err = filepath.Abs(*target)
        if err != nil {
            return "", errors.Wrap(err, "Unable to resolve abs target path")
        }
    }

    absDir, err := filepath.Abs(dir)
    if err != nil {
        return "", errors.Wrap(err, "Unable to resolve abs dir path")
    }

    cmd := exec.Command("virt-make-fs", "-F", "raw", "-t", "ext4",
        absDir, fname)
    err = cmd.Run()
    if err != nil {
        return "", errors.Wrap(err, "Unable to run virt-make-fs command")
    }

    return fname, nil
}

virt-make-fs, a part of libguestfs has similar interface with genisoimage.

It would be better to switch NewISOFsHandler() and NewExt4FsHandler() on run time.

Building and Installing runnc

Same as original.

$ git clone https://github.com/retrage/runnc.git
$ mkdir -p $GOPATH/github.com/retrage
$ ln -sf $PWD/runnc $GOPATH/github.com/retrage/runnc
$ cd runnc
$ git apply patches/0001-solo5-elf-segment-align-workaround.patch
$ make build
$ make install

Testing with Docker Images

I provided a set of Makefiles build LKL Nabla Container base Docker images. It builds Solo5 and frankenlibc, and Docker images.

https://github.com/retrage/lkl-nabla-base-build

I also pushed pre-built Docker images to Docker Hub.

You can use images like this.

$ sudo docker run --rm --runtime=runnc retrage/lkl-nabla-python3-base:
latest -c "print(\'hello\')"
[sudo] password for akira:
nabla-run arg [/opt/runnc/bin/nabla-run --mem=512 --net:tap=tap28157ba5950e --bl
ock:rootfs=/var/run/docker/runtime-runnc/moby/28157ba5950e3e84824bd843fd1dafb06eccc7de2020a0619d6a5b463e5f2c2b/rootfs.img /var/lib/docker/overlay2/3d36c19950e53eefded8e1933f3d7e51990fc4c7b065be6c00776eeab8fb3136/merged/python3.nabla __RUMP_FDINFO_NET_tap=4 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=28157ba5950e PYTHONHASHSEED=1 PYTHONHOME=/usr/local HOME=/ -- -c print(\'hello\')]
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.6.4-6-g756accf-dirty
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x889fff)
Solo5:     rodata @ (0x88a000 - 0xb4cfff)
Solo5:       data @ (0xb4d000 - 0xe7dfff)
Solo5:       heap >= 0xe7e000 < stack < 0x20000000
sleeping 50000 usec
hello
Solo5: solo5_exit(0) called

Conclusion

In this post, I introduced a brief of LKL Nabla Containers. It is still in an early stage and has room for improvement, but already runs practical applications like Python. I would like to measure the performance and evaluate the pros/cons.

Below is the TODO list:

Replace workaround for Solo5
Flexible manifest.json handling on build time
~~Pass lkl.json through run time arguments~~
Do not pass __RUMP_FDINFO_NET_tap=4 environment variable on run time

Update: May 1st, 2020

After wrote this post, I found that LKL must use network information created by the container runtime. Otherwise, the network does not work properly. I added the 3rd feature described in the above TODO list to frankenlibc and runnc.

The OCI runtime builds and passes JSON config for LKL at startup. LKL parses it along with environment variables and arguments.

Now, popular network applications Nginx and redis work on LKL Nabla Containers. They are available as base Docker Images.

Debugging OVMF with GDB

Akira Moroo — Mon, 09 Dec 2019 02:36:39 +0000

In this blog post, I will describe how to debug OVMF using GDB without any special tool unlike another post[1].

Code Mapping in UEFI

On x64 UEFI, it provides flat single address memory space and places the firmware itself and UEFI images on the space without any memory protection. In this way, we can do source code level debugging any UEFI code with a debugger. On OVMF, each feature is modularized
and the module is loaded as UEFI image. BootServices is included in DxeCore.efi, loaded at boot time.

Notify: PPI Guid: EE16160A-E8BE-47A6-820A-C6900DB0250A, Peim notify entry point: 836CA9
PlatformPei: ClearCacheOnMpServicesAvailable
DiscoverPeimsAndOrderWithApriori(): Found 0x0 PEI FFS files in the 1th FV
DXE IPL Entry
Loading PEIM D6A2CB7F-6A18-4E2F-B43B-9920A733700A
Loading PEIM at 0x00007EA8000 EntryPoint=0x00007EAB0BC DxeCore.efi
Loading DXE CORE at 0x00007EA8000 EntryPoint=0x00007EAB0BC

Debug Symbols in EDK2

EDK2 build system generates debug symbol information *.debug along with executables *.efi on debug build (-b DEBUG). If you use gcc (example: GCC5), it compiles source code to ELF object files, link with custom linker script, and convert to PE format. Thus, the debug info is for ELF
and can be recognized by GDB. On the other hand, Visual Studio and clang/lld (CLANG9)[2] generates PE/COFF directly and the debug info may be PDB.

To summarize, the points are:

OVMF code is placed on the flat single memory space.
GDB can debug EDK2 UEFI image built with gcc

Let's take a look at how to debug OVMF.

Building EDK2

Build EDK2 using gcc as usual.

$ git clone git@github.com:tianocore/edk2.git
$ cd edk2
$ git submodule update --init --recursive
$ make -C BaseTools
$ source ./edksetup.sh
$ build -p OvmfPkg/OvmfPkgX64.dsc -b DEBUG -a X64 -t GCC5

To make debugging easy, create a Makefile as follow. Note that we have to connect debugcon at 0x402 to dump debug information (debug.log) from OVMF[4].

#!/usr/bin/env make

SHELL=/bin/bash

LOG=debug.log
OVMFBASE=edk2/Build/OvmfX64/DEBUG_GCC5/
OVMFCODE=$(OVMFBASE)/FV/OVMF_CODE.fd
OVMFVARS=$(OVMFBASE)/FV/OVMF_VARS.fd
QEMU=qemu-system-x86_64
QEMUFLAGS=-drive format=raw,file=fat:rw:image \
          -drive if=pflash,format=raw,readonly,file=$(OVMFCODE) \
          -drive if=pflash,format=raw,file=$(OVMFVARS) \
          -debugcon file:$(LOG) -global isa-debugcon.iobase=0x402 \
          -serial stdio \
          -nographic \
          -nodefaults

run:
    $(QEMU) $(QEMUFLAGS)

debug:
    $(QEMU) $(QEMUFLAGS) -s -S

.PHONY: run debug

Before debugging, run the firmware to get debug.log. It may be better to provide startup.nsh script.

$ make run

Now, we have debug.log. It includes the addresses of loaded UEFI images like this:

Loading PEIM at 0x00007EA8000 EntryPoint=0x00007EAB0BC DxeCore.efi

Next, extract text section (.text) RVA from *.efi PE binaries. This can be done by readelf if it is ELF, but the images are PE format. Here we use retrage/peinfo[3].

$ git clone git@github.com:retrage/peinfo.git
$ cd peinfo
$ make

peinfo extracts section information from a binary. This time we want to know VirtualAddress in RVA.

Name: .text
VirtualSize: 0x000204c0
VirtualAddress: 0x00000240
SizeOfRawData: 0x000204c0
PointerToRawData: 0x00000240
PointerToRelocations: 0x00000000
PointerToLinenumbers: 0x00000000
NumberOfRelocations: 0x0000
NumberOfLinenumbers: 0x0000
Characteristics: 0x60000020

Run following bash script with debug.log and peinfo. This outputs a snippet of GDB script that adds symbol information (add-symbol-file). It calculates the address of UEFI image text section from base address and VirtualAddress.

#!/bin/bash

LOG="debug.log"
BUILD="edk2/Build/OvmfX64/DEBUG_GCC5/X64"
PEINFO="peinfo/peinfo"

cat ${LOG} | grep Loading | grep -i efi | while read LINE; do
  BASE="`echo ${LINE} | cut -d " " -f4`"
  NAME="`echo ${LINE} | cut -d " " -f6 | tr -d "[:cntrl:]"`"
  ADDR="`${PEINFO} ${BUILD}/${NAME} \
        | grep -A 5 text | grep VirtualAddress | cut -d " " -f2`"
  TEXT="`python -c "print(hex(${BASE} + ${ADDR}))"`"
  SYMS="`echo ${NAME} | sed -e "s/\.efi/\.debug/g"`"
  echo "add-symbol-file ${BUILD}/${SYMS} ${TEXT}"
done

$ bash gen_symbol_offsets.sh > gdbscript

The generated GDB script is like this:

add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/PcdPeim.debug 0x82c380
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/ReportStatusCodeRouterPei.debug 0x831080
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/StatusCodeHandlerPei.debug 0x833100
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/PlatformPei.debug 0x835100
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/PeiCore.debug 0x7ee8240
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/DxeIpl.debug 0x7ee3240
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/S3Resume2Pei.debug 0x7edf240
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/CpuMpPei.debug 0x7ed6240
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/DxeCore.debug 0x7ea8240
add-symbol-file edk2/Build/OvmfX64/DEBUG_GCC5/X64/DevicePathDxe.debug 0x7b8f240

Now we are ready.

$ make debug

Let's place a breakpoint at BootServices->HandleProtocol().

(gdb) source gdbscript
(gdb) b CoreHandleProtocol
(gdb) target remote localhost:1234
(gdb) c

The debugger stops, and we can do source code level debug.

   ┌──/home/akira/src/ovmf-debug/edk2/MdeModulePkg/Core/Dxe/Hand/Handle.c──────┐
   │933     CoreHandleProtocol (                                               │
   │934       IN EFI_HANDLE       UserHandle,                                  │
   │935       IN EFI_GUID         *Protocol,                                   │
   │936       OUT VOID            **Interface                                  │
   │937       )                                                                │
B+>│938     {                                                                  │
   │939       return CoreOpenProtocol (                                        │
   │940               UserHandle,                                              │
   │941               Protocol,                                                │
   │942               Interface,                                               │
   │943               gDxeCoreImageHandle,                                     │
   │944               NULL,                                                    │
   │945               EFI_OPEN_PROTOCOL_BY_HANDLE_PROTOCOL                     │
   └───────────────────────────────────────────────────────────────────────────┘
remote Thread 1 In: CoreHandleProtocol                      L938  PC: 0x7eb6ad4 



(gdb)

2019/12/05 Postscript

tnishinaga gave me the improved version of the script
to support multiple search paths. Thank you!

#!/bin/bash

LOG="debug.log"
BUILD="./Build"
SEARCHPATHS="./Build/OvmfX64/DEBUG_GCC5/X64/ ./Build/Edk2SamplePkgX64/DEBUG_GCC5/X64/"
PEINFO="peinfo/peinfo"

cat ${LOG} | grep Loading | grep -i efi | while read LINE; do
  BASE="`echo ${LINE} | cut -d " " -f4`"
  NAME="`echo ${LINE} | cut -d " " -f6 | tr -d "[:cntrl:]"`"
  EFIFILE="`find ${SEARCHPATHS} -name ${NAME} -maxdepth 1 -type f`"
  ADDR="`${PEINFO} ${EFIFILE} \
        | grep -A 5 text | grep VirtualAddress | cut -d " " -f2`"
  TEXT="`python -c "print(hex(${BASE} + ${ADDR}))"`"
  SYMS="`echo ${NAME} | sed -e "s/\.efi/\.debug/g"`"
  SYMFILE="`find ${SEARCHPATHS} -name ${SYMS} -maxdepth 1 -type f`"
  echo "add-symbol-file ${SYMFILE} ${TEXT}"
done

References

Librarizing Linux kernel for Unikernels

Akira Moroo — Mon, 09 Dec 2019 02:30:45 +0000

NOTE: This is a repost of my previous blog post on Medium.

Librarizing Linux kernel for Unikernels | by retrage | Medium

retrage ・ Jun 2, 2019 ・ 9 min read
Medium

I ported the Linux kernel to Unikraft as an external library. This makes it possible to reuse the rich functions of the Linux kernel for Unikernel with less functionality. In this blog post, I describe the overview of the library.

Background

Linux Kernel Library

The Linux kernel is a well-maintained mature open source OS kernel. Recently, there have been researches that propose reuse its components.The Linux Kernel Library (LKL) is one of them, which uses the Linux kernel as a form of Library OS with minimum modifications. LKL is not currently an official Linux project, but it is actively being developed (v4.19 is the latest). Below is the architecture of LKL.

LKL has a host-independent architecture named lkl, and the actual host-dependent code is separated from the arch. The independent and dependent code is placed under arch/lkl and tools/lkl correspondingly. For POSIX host environment, tools/lkl/lib/posix-host.c plays the role.

Unikraft

Unikraft is an experimental project by Xen Project which aims to build small and lightweight Unikernel images by providing divided Unikernel functions. The following is the architecture.

Unikraft itself consists of three parts: main libs, platform libs, and architecture libs. main libs contain architecture and platform independent libraries. platform libs provide platform dependent code as libraries. It currently supports Xen, KVM, and Linux userspace as platforms. architecture libs are libraries for architecture; x86, arm, and arm64. A user has to specify the target architecture and platform by Kconfig when building Unikraft application. The Unikraft build system generates a Unikernel image corresponding to each target.

Unikraft supports external libraries along with internal libraries. There are several official external libraries in public such as newlib, lwip, compiler-rt, eigen, libcxx, libcxxabi, libunwind, and libuuid.

For more details about porting external libraries to Unikraft, see External Library Development.

LKL on Unikraft

Since Unikraft is still at an early stage, it does not have a mature network stack or file system. To tackle this issue, we ported LKL as an external library for Unikraft. Here we introduce two types of the port.

LKL on Unikraft v1

First of all, we integrated LKL to Unikraft build system. The architecture is shown below.

In this version of the port, we added new architecture and platform libraries for LKL. They are just stub and only used when specified in Kconfig. The disadvantage of this design is that it is impossible to build LKL for other architectures or platforms. In addition, it can not cooperate with other Unikraft libraries.

The code is available at:

uk-lkl/unikraft:retrage/lkl

LKL on Unikraft v2

Next, we introduce v2 which can be used as an actual library. Below is the overview.

By this design, we can choose any architecture and platform in concept since LKL is separated from architecture and platform. In addition, other Unikraft libraries can use LKL functions.

For implementation, we added a new host-dependent code called uk-host.c to support Unikraft as a new LKL host environment. The LKL host-dependent code requires some primitives such as mutex, semaphore, thread, timer, etc. on the host side, however, Unikraft main libs cannot satisfy the requirements because of the lack of functionality. Therefore, we ported these primitives from littlekernel, an embedded kernel to LKL. The only LKL host has to do is that calling callback functions periodic time to run the preemptive scheduler. The port of littlekernel is independent of Unikraft. Here is the implementation.

retrage/linux:retrage/fiber

Implementation

The port of LKL supports only x86_64 architecture and KVM platform for some reason.

Unikraft has two types of libc implementation, newlib, and nolibc. newlib is an official external library that supports full libc functions. On the other hand, nolibc provides minimal libc functions so that general libc functions can be used even if newlib does not exist. This version of LKL port is designed to work with nolibc to reduce dependencies. However, since nolibc have enough functions and constants required by LKL, we added the following functions and constants to nolibc.

stdbool
fputc, putchar
STD{IN,OUT,ERR}_FILENO
strncat
strtok_r
setjmp/longjmp

The modified LKL expects the callback function is called periodically as mentioned above, but Unikraft does not provide an interface to register a callback function. Fortunately, it starts a periodic timer at startup, so we just added the interface.

In the KVM platform, a final image is generated using the custom linker script. Since the LKL binary liblkl.o has symbols that are not referred explicitly, they are deleted or hidden by linking using the linker script. For this reason, the linker script is modified so that the symbols from LKL are kept correctly.

The modified Unikraft and the port of LKL is here:

Demonstration

To demonstrate LKL on Unikraft, we ported tools/lkl/tests/boot from LKL as uk-lkl/boot.

$ mkdir unikraft && cd unikraft
$ git clone git@github.com:uk-lkl/unikraft.git --branch=retrage/lkl-v2
$ mkdir libs && cd libs
$ git clone --recursive git@github.com:uk-lkl/lkl.git
$ cd ..
$ mkdir apps && cd apps
$ git clone git@github.com:uk-lkl/boot.git
$ cd boot
$ make menuconfi

Select x86 architecture and KVM guest platform. Save and exit Kconfig. Then, run make.

$ make

To run the final image, run run.sh.

$ ./run.sh

It will output as follow:

[    0.721134] ERR:  [libukboot] boot.c @ 88   : Failed to initialize bus driver 0x56b060: -1
Welcome to  _ __             _____
 __ _____  (_) /__ _______ _/ _/ /_
/ // / _ \/ /  '_// __/ _ `/ _/ __/
\_,_/_//_/_/_/\_\/_/  \_,_/_/ \__/
                  Titan 0.2~ebcb42a
1..33 # boot
* 1 mutex
ok 1 mutex
 ---
 time_us: 0
 log: |
 ...
* 2 semaphore
ok 2 semaphore
 ---
 time_us: 0
 log: |
 ...
* 3 join
ok 3 join
 ---
 time_us: 1
 log: |
  joined 7909384
 ...
* 4 start_kernel
ok 4 start_kernel
 ---
 time_us: 9281
 log: |
  [    0.000000] Linux version 4.19.0+ (akira@akira-Z270) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)) #1 Thu May 30 23:10:09 JST 2019
  [    0.000000] bootmem address range: 0x1c001000 - 0x1d000000
  [    0.000000] On node 0 totalpages: 4095
  [    0.000000]   Normal zone: 56 pages used for memmap
  [    0.000000]   Normal zone: 0 pages reserved
  [    0.000000]   Normal zone: 4095 pages, LIFO batch:0
  [    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
  [    0.000000] pcpu-alloc: [0] 0 
  [    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 4039
  [    0.000000] Kernel command line: mem=16M loglevel=8
  [    0.000000] Dentry cache hash table entries: 2048 (order: 2, 16384 bytes)
  [    0.000000] Inode-cache hash table entries: 1024 (order: 1, 8192 bytes)
  [    0.000000] Memory available: 16088k/16380k RAM
  [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
  [    0.000000] NR_IRQS: 4096
  [    0.000000] lkl: irqs initialized
  [    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
  [    0.198000] lkl: time and timers initialized (irq1)
  [    2.093000] pid_max: default: 4096 minimum: 301
  [   15.926000] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
  [   18.024000] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
  [ 2722.817000] console [lkl_console0] enabled
  [ 2732.709000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
  [ 2760.874000] random: get_random_u32 called from bucket_table_alloc.isra.6+0x9b/0x250 with crng_init=0
  [ 2774.823000] NET: Registered protocol family 16
  [ 3121.319000] clocksource: Switched to clocksource lkl
  [ 3181.520000] NET: Registered protocol family 2
  [ 3261.648000] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
  [ 3263.125000] TCP established hash table entries: 512 (order: 0, 4096 bytes)
  [ 3264.198000] TCP bind hash table entries: 512 (order: 0, 4096 bytes)
  [ 3265.141000] TCP: Hash tables configured (established 512 bind 512)
  [ 3286.743000] UDP hash table entries: 128 (order: 0, 4096 bytes)
  [ 3287.830000] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes)
  [ 3456.331000] workingset: timestamp_bits=62 max_order=12 bucket_order=0
  [ 4041.077000] SGI XFS with ACLs, security attributes, no debug enabled
  [ 6032.987000] jitterentropy: Initialization failed with host not compliant with requirements: 2
  [ 6038.248000] io scheduler noop registered
  [ 6038.908000] io scheduler deadline registered
  [ 6067.383000] io scheduler cfq registered (default)
  [ 6068.093000] io scheduler mq-deadline registered
  [ 6068.764000] io scheduler kyber registered
  [ 7894.266000] NET: Registered protocol family 10
  [ 7979.259000] Segment Routing with IPv6
  [ 7988.492000] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
  [ 8097.859000] Warning: unable to open an initial console.
  [ 8106.456000] This architecture does not have kernel memory protection.
  [ 8107.119000] Run /init as init process
  lkl_start_kernel(&lkl_host_ops, "mem=16M loglevel=8") = 0 
 ...
* 5 getpid
ok 5 getpid
 ---
 time_us: 5
 log: |
  lkl_sys_getpid() = 1 
 ...
* 6 syscall_latency
ok 6 syscall_latency
 ---
 time_us: 126
 log: |
  avg/min/max: lkl:107822000/104000000/450000000 native:6788000/6000000/77000000
 ...
* 7 umask
ok 7 umask
 ---
 time_us: 0
 log: |
  lkl_sys_umask(0777) = 18 
 ...
* 8 umask2
ok 8 umask2
 ---
 time_us: 0
 log: |
  lkl_sys_umask(0) = 511 
 ...
* 9 creat
ok 9 creat
 ---
 time_us: 9
 log: |
  lkl_sys_creat("/file", access_rights) = 0 
 ...
* 10 close
ok 10 close
 ---
 time_us: 0
 log: |
  lkl_sys_close(0) = 0 
 ...
* 11 failopen
ok 11 failopen
 ---
 time_us: 9
 log: |
  lkl_sys_open("/file2", 0, 0) = -2 No such file or directory
 ...
* 12 open
ok 12 open
 ---
 time_us: 7
 log: |
  lkl_sys_open("/file", LKL_O_RDWR, 0) = 0 
 ...
* 13 write
ok 13 write
 ---
 time_us: 4
 log: |
  lkl_sys_write(0, wrbuf, sizeof(wrbuf)) = 5 
 ...
* 14 lseek_cur
ok 14 lseek_cur
 ---
 time_us: 0
 log: |
  lkl_sys_lseek(0, 0, LKL_SEEK_CUR) = 5 
 ...
* 15 lseek_end
ok 15 lseek_end
 ---
 time_us: 0
 log: |
  lkl_sys_lseek(0, 0, LKL_SEEK_END) = 5 
 ...
* 16 lseek_set
ok 16 lseek_set
 ---
 time_us: 0
 log: |
  lkl_sys_lseek(0, 0, LKL_SEEK_SET) = 0 
 ...
* 17 read
ok 17 read
 ---
 time_us: 1
 log: |
  lkl_sys_read=5 buf=test
 ...
* 18 fstat
ok 18 fstat
 ---
 time_us: 1
 log: |
  lkl_sys_fstat=0 mode=100721 size=5
 ...
* 19 mkdir
ok 19 mkdir
 ---
 time_us: 8
 log: |
  lkl_sys_mkdir("/mnt", access_rights) = 0 
 ...
* 20 stat
ok 20 stat
 ---
 time_us: 7
 log: |
  lkl_sys_stat("/mnt")=0 mode=40721
 ...
* 21 pipe2
ok 21 pipe2
 ---
 time_us: 9
 log: |
 ...
* 22 epoll
ok 22 epoll
 ---
 time_us: 5
 log: |
 ...
* 23 mount_fs_proc
ok 23 mount_fs_proc
 ---
 time_us: 18
 log: |
  lkl_mount_fs("proc") = 0 
 ...
* 24 chdir_proc
ok 24 chdir_proc
 ---
 time_us: 7
 log: |
  lkl_sys_chdir("proc") = 0 
 ...
* 25 open_cwd
ok 25 open_cwd
 ---
 time_us: 7
 log: |
 ...
* 26 getdents64
ok 26 getdents64
 ---
 time_us: 6
 log: |
  4 . .. fs bus irq net sys tty kmsg maps misc stat iomem crypto driver 
  mounts uptime vmstat cmdline cpuinfo devices ioports loadavg meminfo version 
  consoles kallsyms slabinfo softirqs zoneinfo buddyinfo diskstats interrupts 
  partitions timer_list 
 ...
* 27 close_dir_fd
ok 27 close_dir_fd
 ---
 time_us: 0
 log: |
  lkl_sys_close(dir_fd) = 0 
 ...
* 28 chdir_root
ok 28 chdir_root
 ---
 time_us: 7
 log: |
  lkl_sys_chdir("/") = 0 
 ...
* 29 umount_fs_proc
ok 29 umount_fs_proc
 ---
 time_us: 11
 log: |
  lkl_umount_timeout("proc", 0, 1000) = 0 
 ...
* 30 lo_ifup
ok 30 lo_ifup
 ---
 time_us: 44
 log: |
  lkl_if_up(1) = 0 
 ...
* 31 gettid
ok 31 gettid
 ---
 time_us: 0
 log: |
  7893000
 ...
* 32 many_syscall_threads
ok 32 many_syscall_threads
 ---
 time_us: 26
 log: |
 ...
* 33 stop_kernel
ok 33 stop_kernel
 ---
 time_us: 531
 log: |
  [145272.871000] reboot: Restarting system
  lkl_sys_halt() = 0 
 ...

Issues

The current LKL on Unikraft has the following issues.

It needs modifications to Unikraft.
It does not support disk operations and networks.
It hangs under some certain situations.

The first issue is described above in details. The second issue is because of the lack of file system and network support by Unikraft. The third issue is not only on Unikraft, but also with retrage/linux:retrage/fiber.

As a design issue, the use of Linux kernel does not match with Unikraft policy. Unikraft aims to build 'slim' Unikernel images by building the necessary libraries, but LKL reuses existing Linux kernel which is 'fat'. The size of the final image tends to be large if LKL is integrated.

References

LKL.js: Running Linux Kernel on JavaScript Directly

Akira Moroo — Mon, 09 Dec 2019 02:23:33 +0000

NOTE: This is a repost of my previous post on Medium.

LKL.js: Running Linux Kernel on JavaScript Directly | by retrage | Medium

retrage ・ Jul 25, 2018 ・ 10 min read
Medium

I ported Linux kernel directly on JavaScript. In other words, I translated the Linux kernel to JavaScript using Emscripten, and Unlike JSLinux, it runs without emulators.

The following is the working repository.

https://github.com/retrage/linux/tree/retrage/em-v2

I published a demonstration site for LKL.js. Please enable SharedArrayBuffer and try it out

LKL.js Demo

I also published slides about LKL.js.

https://speakerdeck.com/retrage/lkl-dot-js-running-linux-kernel-on-javascript-star-directly-star

Linux Kernel Library (LKL)

We use Linux Kernel Library (LKL) which makes the Linux kernel an anykernel. LKL is a fork of torvalds/linux. It is designed to put LKL specific code only in arch/lkl and runs without modifications of other code. By this design, it makes easy to follow the mainline. (Currently v5.3) Since LKL is anykernel, it runs on user space of various OS such as Linux, FreeBSD, and Windows, etc.

Emscripten

Emscripten is LLVM based C/C++ to
JavaScript/WebAssembly transpiler.
It also provides a Unix-like environment to run translated software
on web browsers.

Can we port LKL to JavaScript with Emscripten?

LKL runs on various OSes, Emscripten provides Unix-like
environment. So can LKL be ported to JavaScript with Emscripten?

Current Status of Linux Kernel Build with Clang

First of all, the Linux kernel is deeply dependent on gcc-extension, and there is a doubt that Clang can not compile it. Once upon the time, there was LLVMLinux project that aims to compile Linux kernel with Clang. However, through the efforts of the Google Android team, two LTS (4.4 and 4.9) can be built with Clang. Now, LKL can be built with Clang.

LKL Build Flow 101

Let's look at the build flow of LKL.
First, when

$ make -C tools/lkl

is performed, the build system determines which source code (*.c/*.S)
to be built from the Kconfig settings and compiles them. Object files (*.o) generated by compiling are once archived by ar to built-in.o. Next, it links all built-in.o files into vmlinux at once. For host side code, files under tools/lkl/lib compiled and linked to liblkl.o. Finally, link all files (vmlinux and liblkl.o) to liblkl.so.

This is a simple build flow of LKL.

Porting LKL with Emscripten

Next, we will take a look at how to port LKL with Emscripten.

Not limited to Emscripten, when using LLVM infrastructures, the compiler compiles source to target with the following flow.

Soruce -> LLVM IR -> Target

In this way, the source is once converted to LLVM IR (*.bc/*.ll) and then converted to the target. In Emscripten, the "linking" is the conversion from LLVM IR to JavaScript. Therefore, it is necessary to first convert all (including libc etc. provided by Emscripten) to LLVM IR.

Generating vmlinux.bc

The build using emcc (An Emscripten Clang wrapper) is:

make -C tools/lkl CC="$CC $CFLAGS" AR="$PY $PWD/ar.py" V=1

The two important things here are $FCFLAS and ar.py. I will explain each one. (Note that C="$CC $CFLAGS" is forced to pass $CFLAGS)

$CFLAGS is:

CFLAGS="$CFLAGS -s WASM=0"
CFLAGS="$CFLAGS -s ASYNCIFY=1"
CFLAGS="$CFLAGS -s EMULATE_FUNCTION_POINTER_CASTS=1"
CFLAGS="$CFLAGS -s USE_PTHREADS=1"
CFLAGS="$CFLAGS -s PTHREAD_POOL_SIZE=4"
CFLAGS="$CFLAGS -s TOTAL_MEMORY=1342177280"

The options are to pass to Emscripten. Please refer to the Emscripten manual for details.

Furthermore, the following definitions are specified.

CFLAGS="$CFLAGS -DMAX_NR_ZONES=2"
CFLAGS="$CFLAGS -DNR_PAGEFLAGS=20"
CFLAGS="$CFLAGS -DSPINLOCK_SIZE=0"
CFLAGS="$CFLAGS -DF_GETLK64=12"
CFLAGS="$CFLAGS -DF_SETLK64=13"
CFLAGS="$CFLAGS -DF_SETLKW64=14"

These values are originally obtained by compiling an empty file at the time of Linux kernel build. However, this time they can not be obtained directly. Therefore, we have to specify these values which come from when building with the x86_64 environment.

Next, I will explain ar.py. The following is a snippet of ar.py.

filename = "objs"

def main():
    if not os.path.exists(filename):
        with open(filename, "w") as fp:
            pass

    objs = []
    for i, arg in enumerate(sys.argv):
        if ".o" in arg and not "built-in" in arg and i > 2:
            objs.append(arg)

    with open(filename, "aw") as fp:
        for obj in objs:
            if not obj is "":
                fp.write(obj + " ")

    return 0

As explained above, the build system of Linux kernel gathers object files by ar and links them to get vmlinux.

To work with Emscripten we need to get vmlinux as a LLVM bitcode. LLVM has a linker called llvm-link that links multiple LLVM bitcode files to get one LLVM bitcode. To generate vmlinux.bc, we need to use llvm-link, but there is a problem. llvm-link can not take archive files as arguments like lds. Therefore, we have to record object files that are originally archived. In this case, ar.py will record them as file paths in objs.

Next, Let's look at the part of vmlinux.bc generation. I added following scripts to scripts/link-vmlinux.sh.

info CLEAN obj 
python "${srctree}/clean-obj.py"

info GEN link-vmlinux.sh
python "${srctree}/link-vmlinux-gen.py"

info LINK vmlinux
bash "${srctree}/link-vmlinux.sh"

clean-obj.py removes duplicated file paths from objs which is generated by ar.py. link-vmlinux-gen.py generates vmlinux-link.sh (not scripts/link-vmlinux.sh) which performs llvm-link. By performing vmlinux-link.sh, we can get vmlinux.bc.

This is the flow of generating vmlinux.bc.

Generating boot.js

Next, I will look at until JavaScript code is generated. As explained above, since LKL is one of Library OS, vmlinux does not work on its own, it works only when it has a part of an application. In this case, our target is tools/lkl/tests/boot which is LKL's Hello, world.

$LINK -o $LKL/tests/boot.bc \
    $LKL/tests/boot-in.o $LKL/lib/liblkl-in.o $LKL/lib/lkl.o

First, we have to link vmlinux.bc ($LKL/lib/lkl.o), host dependent part $LKL/lib/liblkl-in.o and applicatin part $LKL/tests/boot-in.o and get $LKL/tests/boot.bc.

$DIS -o $LKL/tests/boot.ll $LKL/tests/boot.bc
$CP ~/.emscripten_cache/asmjs/dlmalloc.bc js/dlmalloc.bc
$CP ~/.emscripten_cache/asmjs/libc.bc js/libc.bc
$CP ~/.emscripten_cache/asmjs/pthreads.bc js/pthreads.bc
$DIS -o js/dlmalloc.ll js/dlmalloc.bc
$DIS -o js/libc.ll js/libc.bc
$DIS -o js/pthreads.ll js/pthreads.bc
$PY rename_symbols.py $LKL/tests/boot.ll $LKL/tests/boot-mod.ll

First, it disassembles all LLVM bitcode files ($LKL/tests/boot.bc and libc.bc etc.) using llvm-dis. Next, it applies rename_symbols.py to boot.ll.

There is a reason for performing such operations. This is because function names used in the Linux kernel conflict with function names used in libcs. In normal LKL, this conflict is avoided by using ELF linker tricks. Meanwhile, since JavaScript generated by Emscripten does not have a namespace, such collisions occur. Therefore, by rewriting the functions names that would collide with rename_symbols.py, it can avoid collisions.

In addition, rename_symbols.py also performs operations such as converting inline assemblies in Linux kernel to Emscripten emscripten_asm_const_int.

From the boot-mod.ll,

EMCC_DEBUG=1 $CC -o js/boot.html $LKL/tests/boot-mod.ll $CFLAGS -v

generate HTML and JavaScript files.

Adding Workarounds

Although we generated the Linux kernel translated in "completely" JavaScript and the application boot.js, it will not work as it is. This is due to the fact that the architecture of computers and JavaScript is very different. So we have to make some modifications.

Replacing inline assemblies

In the Linux kernel, the architecture-dependent code is basically placed
under arch/$ARCH, and another code is architecture independent. However, an empty inline assembly may be inserted so that optimization by the compiler prevents meaningful code from being lost at compile time. Here is an example, set_normalized_timespec64 in kernel/time/time.c:

void set_normalized_timespec64(struct timespec64 *ts, time64_t sec, s64 nsec)
{
    while (nsec >= NSEC_PER_SEC) {
        /*
         * The following asm() prevents the compiler from
         * optimising this loop into a modulo operation. See
         * also __iter_div_u64_rem() in include/linux/time.h
         */
        asm("" : "+rm"(nsec));
        nsec -= NSEC_PER_SEC;
        ++sec;
    }
    while (nsec < 0) {
        asm("" : "+rm"(nsec));
        nsec += NSEC_PER_SEC;
        --sec;
    }
    ts->tv_sec = sec;
    ts->tv_nsec = nsec;
}

Such Inline assemblies cause a failure to convert from LLVM bitcode to
JavaScript. Therefore, we have to replace inline assemblies such as asm("" : "+rm"(nsec)) with emcsripten_asm_const_int which calls JavaScript code from C defined in Emscripten.

Fix early_param

In the Linux kernel, there is early_param. This is defined in include/linux/init.h as follows:

struct obs_kernel_param {
    const char *str;
    int (*setup_func)(char *);
    int early;
};
/* snip */
#define __setup_param(str, unique_id, fn, early)            \
    static const char __setup_str_##unique_id[] __initconst     \
        __aligned(1) = str;                     \
    static struct obs_kernel_param __setup_##unique_id      \
        __used __section(.init.setup)               \
        __attribute__((aligned((sizeof(long)))))        \
        = { __setup_str_##unique_id, fn, early }
/* snip */
#define early_param(str, fn)                        \
    __setup_param(str, fn, fn, 1)

early_param is a macro, taking str and fn as arguments, and obs_kernel_param structure placed in .init.setup.

By referring to arch/lkl/kernel/vmlinux.ldS which is generated in the build of LKL, we can see that .init.setup is arranged between __setup_start and __setup_end.

__setup_start = .; KEEP(*(.init.setup)) __setup_end = .;

These symbols will be used in init/main.c as follows. Here it compares one of boot parameter (param) of Linux kernel with str of obs_kernel_param in .init.setup. If it matches, it will execute (*setup_func)(char*) with argument val.

/* Check for early params. */
static int __init do_early_param(char *param, char *val,
                 const char *unused, void *arg)
{
    const struct obs_kernel_param *p;

    for (p = __setup_start; p < __setup_end; p++) {
        if ((p->early && parameq(param, p->str)) ||
            (strcmp(param, "console") == 0 &&
             strcmp(p->str, "earlycon") == 0)
        ) {
            if (p->setup_func(val) != 0)
                pr_warn("Malformed early option '%s'\n", param);
        }
    }
    /* We accept everything at this stage. */
    return 0;
}

In summary, do_early_param executes setup_func registered by early_param by referring boot parameters.

However, since it uses ELF symbols, it does not work correctly in JavaScript. For this reason, the function which will be called here is hard coded.

static int __init do_early_param(char *param, char *val,
                 const char *unused, void *arg)
{
        /* XXX: There is a lot of early_param, but hardcode in init/main.c */
        const char *early_params[MAX_INIT_ARGS+2] = { "debug", "quiet", "loglevel", NULL, };
        int i;

        for (i = 0; early_params[i]; i++) {
                if (strcmp(param, early_params[i]) == 0 ||
                    (strcmp(param, "console") == 0 &&
                     strcmp(early_params[i], "earlycon") == 0)
                ) {
                        switch (i) {
                                case 0: /* debug */
                                if (debug_kernel(val) != 0)
                                        pr_warn("Malformed early option '%s'\n", param);
                                break;
                                case 1: /* quiet */
                                if (quiet_kernel(val) != 0)
                                        pr_warn("Malformed early option '%s'\n", param);
                                break;
                                case 2: /* loglevel */
                                if (loglevel(val) != 0)
                                        pr_warn("Malformed early option '%s'\n", param);
                                break;
                                default:
                                pr_warn("Unknown early option '%s'\n", param);
                        }
                }
        }

    /* We accept everything at this stage. */
    return 0;
}

Fix initcall

Like early_param, initcall which are called in the initialization manages functions using ELF symbols. With JavaScript alone, we can not know which function should be called. Therefore, we have to generate inticall tables from System.map generated by a normal build of LKL.

    with open(sys.argv[1], "r") as fp:
        for line in fp:
            if SIG in line:
                symbol = line[:-1].split(" ")[2]
                try:
                    level = int(symbol[-1])
                    initcall = symbol[symbol.index(SIG)+len(SIG):len(symbol)-1]
                    initcalls[level].append(initcall)
                except ValueError:
                    pass

    for level, row in enumerate(initcalls):
        print("/* initcall{} */".format(level))
        print("EM_ASM({")
        for initcall in row:
            if initcall in blacklist:
                print("    /* _"+initcall+"(); */")
            else:
                print("    _"+initcall+"();")
        print("});")

The above is the initcall table generation script. We hard-code the code to do_initcalls. EM_ASM is an inline assembly that directly calls the JavaScript code in C.

static void __init do_initcalls(void)
{
        /* XXX: initcalls are broken, so hardcode here */
        /* initcall0 */
        EM_ASM({
            _net_ns_init();
        });
        /* initcall1 */
        EM_ASM({
            _lkl_console_init();
            _wq_sysfs_init();
            _ksysfs_init();
/* snip */
        });
}

Demonstration and the Results

As described at the top, LKL.js uses pthread, we have to enable SharedArrayBuffer. Although every modern web browsers are shipped with SharedarrayBuffer, it is disabled by default because of Spectre mitigation in Mozilla Firefox. Therefore, please enable it before executing the demo.

The following is the result of start_kernel. We can see that it shows dmesg on browsers.

  [    0.000000] Linux version 4.16.0+ (akira@akira-Z270) () #13 Tue Jul 17 23:01:19 JST 2018
  [    0.000000] bootmem address range: 0x675000 - 0x1674000
  [    0.000000] On node 0 totalpages: 4095
  [    0.000000]   Normal zone: 36 pages used for memmap
  [    0.000000]   Normal zone: 0 pages reserved
  [    0.000000]   Normal zone: 4095 pages, LIFO batch:0
  [    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
  [    0.000000] pcpu-alloc: [0] 0 
  [    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 4059
  [    0.000000] Kernel command line: mem=16M loglevel=8
  [    0.000000] Parameter  is obsolete, ignored
  [    0.000000] Parameter  is obsolete, ignored
  [    0.000000] Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
  [    0.000000] Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
  [    0.000000] Memory available: 16144k/16380k RAM
  [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
  [    0.000000] NR_IRQS: 1024
  [    0.000000] lkl: irqs initialized
  [    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
  [    0.000100] lkl: time and timers initialized (irq1)
  [    0.001100] pid_max: default: 4096 minimum: 301
  [    0.009400] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
  [    0.009900] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
  [    0.327100] console [lkl_console0] enabled
  [    0.329600] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
  [    0.329700] xor: automatically using best checksumming function   8regs     
  [    0.341199] NET: Registered protocol family 16
  [    0.388999] clocksource: Switched to clocksource lkl
  [    0.414100] NET: Registered protocol family 2
  [    0.437700] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes)
  [    0.438199] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
  [    0.439000] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
  [    0.439600] TCP: Hash tables configured (established 1024 bind 1024)
  [    0.443200] UDP hash table entries: 256 (order: 0, 4096 bytes)
  [    0.444000] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
  [    0.472100] workingset: timestamp_bits=30 max_order=12 bucket_order=0
  [    0.863100] SGI XFS with ACLs, security attributes, no debug enabled
  [    0.923700] jitterentropy: Initialization failed with host not compliant with requirements: 2
  [    0.924599] io scheduler noop registered
  [    0.924900] io scheduler deadline registered
  [    0.933099] io scheduler cfq registered (default)
  [    0.933500] io scheduler kyber registered
  [    1.633500] NET: Registered protocol family 10
  [    1.658400] Segment Routing with IPv6
  [    1.660800] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
  [    1.674200] ------------[ cut here ]------------
  [    1.675500] WARNING: CPU: 0 PID: 0 at arch/lkl/kernel/setup.c:188   (null)
  [    1.675899] Call Trace:
  [    1.676200] 
  [    1.676999] ---[ end trace 941dc55fe0966cff ]---
  [    1.684299] Warning: unable to open an initial console.
  [    1.685200] This architecture does not have kernel memory protection.
  pthread_join((pthread_t)tid, NULL): No such process
  lkl_start_kernel(&lkl_host_ops, "mem=16M loglevel=8") = 0

Limitations

From the above results, we confirmed that Linux kernel was booted directly in JavaScript. However, it just outputted dmesg and it is not suitable for practical use at all. This is because of the following problems:

It fails to create kernel threads.
It fails to mount rootfs.
It fails to execute init (PID 1).

Also, support for pthreads in Emscripten is not good. We extracted semaphore, mutex, and thread from Little Kernel (LK) and add them to LKL as green threads.

https://github.com/retrage/linux/tree/retrage/fiber

We plan to create LKL.js using this green threads.

Summary

We created a Linux kernel fully translated in JavaScript using LKL and Emscripten. It boots the Linux kernel and we confirmed that it shows dmesg. Although the architecture is greatly different between computers and JavaScript, we found that it works somewhat by adding some fixes and workarounds.

DEV Community: Akira Moroo

9pfsPkg: Network Boot from Bell Labs

yabits / 9pfsPkg

9P Client File System for UEFI

9pfsPkg

License

What is Network Boot?

UEFI is Extensible

Plan 9 File Protocol

9pfsPkg

yabits / 9pfsPkg

9P Client File System for UEFI

9pfsPkg

License

9P Boot

Proxy Boot

Conclusion

References

Linux Kernel Library Nabla Containers Internals

Modifications to runnc

frankenlibc

Entry Point

Devices

Conclusion

Porting Linux to Nabla Containers

frankenlibc on Solo5

Building frankenlibc

Testing

LKL Nabla Containers

Updating Supplied Arguments

Creating Disk Image

Building and Installing runnc

Testing with Docker Images

Conclusion

Update: May 1st, 2020

Debugging OVMF with GDB

Code Mapping in UEFI

Debug Symbols in EDK2

Building EDK2

2019/12/05 Postscript

References

Librarizing Linux kernel for Unikernels

Librarizing Linux kernel for Unikernels | by retrage | Medium

retrage ・ Jun 2, 2019 ・ 9 min read Medium

Background

Linux Kernel Library

Unikraft

LKL on Unikraft

LKL on Unikraft v1

LKL on Unikraft v2

Implementation

Demonstration

Issues

References

LKL.js: Running Linux Kernel on JavaScript *Directly*

LKL.js: Running Linux Kernel on JavaScript *Directly* | by retrage | Medium

retrage ・ Jul 25, 2018 ・ 10 min read Medium

Linux Kernel Library (LKL)

Emscripten

Can we port LKL to JavaScript with Emscripten?

Current Status of Linux Kernel Build with Clang

LKL Build Flow 101

Porting LKL with Emscripten

Generating vmlinux.bc

Generating boot.js

Adding Workarounds

Replacing inline assemblies

Fix early_param

Fix initcall

Demonstration and the Results

Limitations

Summary

Reference

retrage ・ Jun 2, 2019 ・ 9 min read
Medium

LKL.js: Running Linux Kernel on JavaScript Directly

LKL.js: Running Linux Kernel on JavaScript Directly | by retrage | Medium

retrage ・ Jul 25, 2018 ・ 10 min read
Medium