During downtime over the holidays, I like to tinker with computers. Given this habit and knowing how often I break my Linux system in the process, I was pretty excited when I came across Distrobox, which lets you run other Linux distros in a container, protecting your host system from whatever shenanigans you want to try. It recommended the Podman container engine over Docker and that was another exciting tool to learn about. This was going to be fun.
Then, ironically, while struggling to install a recent version of Podman on Ubuntu 24.04 (not available in ubuntu repos), I borked my system!
Well, shoot. Now what?
I had learned that Distrobox and Podman come from the Redhat / Fedora Linux community and are heavily used in so-called immutable / atomic distros. I had seen some articles and videos by Jorge Castro about Fedora Silverblue, an atomic Fedora distro, and his Ubuntu-flavored customizations. I'm a longtime Ubuntu user, too, so that made Fedora more possible in my mind. I had also learned about bootc, a tool promising to underpin future immutable distros by using container technology to build, distribute, install and upgrade OS images on bare metal hardware. As it turns out, Jorge is one of the developers of a new bootc distribution based on Fedora Silverblue called Bluefin.
At this point, I had to reinstall the OS, but I would run the same risk of borking the system trying to install Podman if I reinstalled Ubuntu. I decided to give Bluefin a shot.
Installation
Simply put, it was the easiest installation I've ever experienced. Up and running in about 10 minutes. No hardware issues. Everything just worked. As an aside, I decided to change up my dual-boot and move Windows to a separate drive. It took more than 3 hours from the start of the install process to having a working system (no bells and whistles ... just a functional display, a user folder matching my username, and Chrome installed).
It took about 5 more minutes to apply Secure Boot and whole-disk encryption with TPM unlock using user-friendly ujust commands included with Bluefin (after 20 minutes of googling and nervous flip flopping). Google TPM setup on Linux if you want a sense of the complexity that drives most of us to run less securely. This is the security I want if my travel laptop is stolen.
Of course, as a developer, I wanted to apply Bluefin's devmode that brings in a suite of modern development tools including Podman and Distrobox, the very tools that sent me down this path in the first place. One ujust devmode command and a reboot later and I had a powerful developer workstation.
Rules of the Road
Bluefin is opinionated and aims to keep system and user separate in order to ensure a stable system. This is where the immutability comes in. Only the /etc and /var directories are writeable. Before you write this off as a non-starter, know that most other system directories you might need to interact with to configure and run the system are linked to sub-directories in /etc and /var.
-
/home->/var/home -
/usr/local->/var/usrlocal -
/opt->/var/opt -
/mnt->/var/mnt -
/media->/run/media
This enables you to add systemd unit files and rules, install software in /usr/local, update /etc/fstab and /etc/hosts, for example.
Coming from Ubuntu and using the apt package manager, I expected to have to learn the Fedora dnf package manager, but as it turns out, Bluefin recommends the following alternatives, depending on what you want to install:
- Flatpak: For GUI applications
- Homebrew: For CLI and TUI apps
- Distrobox / Toolbox: For apps or binaries not available via Flatpak and Homebrew
- rpm-ostree: For low-level packages needed on the host system
This might seem like a lot of complexity compared with apt install this and apt install that, but the result is clean separation and ease of maintenance.
Flatpaks are isolated and can be easily removed without impacting the system.
Everything installed by Homebrew is under /home/linuxbrew/.linuxbrew, clearly separate from the system packages and easily removed without incident.
Containers exist to isolate filesystems and processes, so Distrobox is your ultimate tinkering playground. Install anything from any distribution and export it to make it run as if installed on the host system. If you break your container, or just don't like what you installed, you can delete it easy peasy and your system will still be working like new.
Finally, rpm-ostree gives you the means to install whatever packages you need as a layer on top of the host filesystem image managed by Bluefin. As an example, I installed the envycontrol package to be able to turn off the nvidia gpu in my laptop to save battery.
Now, rpm-ostree, in particular, gives you the ability to bork your system. And this is where the image-based nature of Bluefin is the ultimate backstop.
Updates and Restoring When Things Go Wrong
Bluefin uses OCI images and bootc to enable image-based updates that are staged and applied automatically after a reboot. That means packages aren't updated piecemeal with fingers crossed. Instead, a tested stable image replaces the prior tested stable image and you are, in most cases, blissfully unaware. Even more magical, it does this without interfering with your apps, configurations and customizations thanks to rpm-ostree and the clean separation described earlier.
Now, every user's combination of hardware, packages, apps and configuration is unique, so it's still possible that an update could break your system, or more likely that you add some ill-advised package or config via rpm-ostree and break it yourself. However, in that unlikely event, rolling back to a working system is as easy as selecting it from a menu at boot time. Amazing!
Now, restoring the immutable system doesn't restore your personal data if lost due to, for example, a drive failure. So, having a backup strategy is still important, but the built-in rollback for the host system makes having a workable backup strategy a lot easier.
Implementing a backup strategy short of block device replication is an exercise in hard choices. What do I backup? Where is all the important stuff located? How do I reapply what I backed up when I'm ready to restore the system? Is it a different process depending on how it got broke?
With Bluefin, getting back to a current working system is a reboot. After that, I keep a list of my flatpaks, homebrew installs, rpm-ostree layered packages, the few specific system customizations I've made to /etc/fstab and /etc/hosts, for example, and backup /home. (NOTE: This will get easier still with new tools and options that I'm still learning from Bluefin)
Tinkering
With so much thought having gone into the stability and recoverability of the system, Bluefin (and ublue, in general) gives me a lot more confidence about tinkering with new software.
Ideally, I'll find what I'm looking for available as a flatpak or via Homebrew, which are nearly certain to be problem-free. If not, I can launch a Distrobox container and install just about anything from any Linux distro. I might break the container, but I won't break my system.
If developing software, I'll use a devcontainer launched in VS Code and any system dependencies will be installed in the container, leaving my host system conflict-free.
Finally, infrequently, like when I'm experimenting with a whole new paradigm in operating systems, some things will need to be installed on the host system. As noted above, rpm-ostree gives me that option and the confidence of knowing that undoing any damage is a reboot away.
This is going to be fun!
Top comments (0)