Getting started with distri on Azure

You might have come across Michael Stapelberg's Linux package managers are slow write-up earlier this month. His hypothesis is that indexes, hooks/triggers and archive operations are a big source of issues with conventional Linux package managers.

Concurrently, Michael announced distri, a Linux distro design to experiment with different approaches in these areas. I've been playing around with distri after getting it up and running on Azure. Here's what I learned.

What's in it?

In its current release, jackherer, distri ships Linux 5.1.9, systemd, and 170+ other packages including Python 2.7, Python 3, Perl, Golang and Docker, all the way to Xorg and i3.

distri is also a command-line tool used to build packages and the distro (images, repos) itself. The online repo contains said images and binary packages. The binary packages are SquashFS files, with metadata.

What makes it unique?

In addition to the distri tool and the packaging definitions for 400+ packages, the GitHub repo contains the code (Go) for a FUSE filesystem that assembles images from the local software store as a working, read-only filesystem for the running instance.

So if you inspect the root of the filesystem, you'll notice that /bin, /share and /lib (and their /usr counterparts) point to /ro. And, you guessed it, /ro is handled by this FUSE component.

Each package:version pair ships in a separate directory under /ro. Exchange directories are used for places where each package is expected to contribute files, such as /usr/include.

The part that makes distri unique is that this FUSE component lazily loads the images as needed. The SquashFS images, roughly equivalent to binary packages such as debs and rpms, live in a separate store under /roimg.

What all of this implies is that package manager operations are very fast, because they involve transporting an SquashFS file (and textproto metadata) to the local software store, and then letting the FUSE component work its magic by assembling it in the filesystem "view". Here's Michael showing how that works

Neither the concept of one folder per package or even the lazy loading with FUSE concepts are new (Nix, AppFS) but I don't think I have seen both put together in a modern way. Michael says that this method provides just enough FHS compatibility for 3rd-party applications to work, such as Chrome or Visual Studio Code.

Behind the scenes

About 80% of the packages in the distri repo use the cbuilder build method, but other available and popular methods include gomodbuilder and perlbuilder.

One of Michael's topics of interest is hooks and triggers, which he argues makes deserialization, idempotency, upstream standardization, etc., much more harder. Only one package in distri has a trigger, and that is openssh for host key generation.

The packaging is very streamlined (respective to upstream) with only about 60 patches, 130 manual extra build flags, 115 manually declared runtime dependencies and 295 custom build steps across over 400 packages.

Conceptually similar repos that you might want to compare include nixpkgs, Spack, Homebrew formulae or the Photon specs.

This is me building nano in distri. I use the distri scaffold tool to create a textproto file, declare my two build dependencies and go:

...this will end with something like:

2019/09/04 16:59:04   step 0: 2m31.51560212s (command: [${DISTRI_SOURCEDIR}/configure --host=x86_64-pc-linux-gnu --prefix=${DISTRI_PREFIX} --sysconfdir=/etc --disable-dependency-tracking])
2019/09/04 16:59:04   step 1: 18.112800177s (command: [make -j8 V=1])
2019/09/04 16:59:04   step 2: 2.581871913s (command: [make install DESTDIR=${DISTRI_DESTDIR} PREFIX=${DISTRI_PREFIX}])
[...]
2019/09/04 16:59:05 nano runtime deps: ["file-amd64-5.34-3" "glibc-amd64-2.27-3" "zlib-amd64-1.2.11-3" "ncurses-amd64-6.1-5"]
[...]
2019/09/04 16:59:05 package successfully created in /root/distri/build/distri/pkg/nano-amd64-4.3-1.squashfs

You can then place this package in your local store, where the FUSE component will pick it up, making the nano command avaiable in your $PATH. Neat!

Beyond packaging

distri runs on Azure pretty much out of the box. Images can be converted to the VHD format, and serial console can be enabled optionally, but because it doesn't let root SSH into it by default (which is good) you might want to consider the steps I documented which include uploading a disk and creating a VM with the Azure CLI.

I also wrote a basic /etc/os-release for distri, because one is currently not included. Here's what that looks like:

root@distri0:~# hostnamectl 
   Static hostname: distri0
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 201aa7a551c14f299bb973f1ce206503
           Boot ID: e511d5411c804a7e8f24288025219796
    Virtualization: microsoft
  Operating System: distri (jackherer)
            Kernel: Linux 5.1.9
      Architecture: x86-64

More demos and cool things to try are on the GitHub repo. If you're interested in distri, I suggest you join the mailing list or join the conversation on GitHub Issues for the repo.

If you're in Europe and attending All Systems Go I'd love to meet and talk more about Linux and package management. And if you're in the USA and attending All Things Open join me for a session on The Future of Linux Distros in the Cloud, October 14th, 2019 in Raleigh.