DEV Community: Fran Torres

Pi-Hole setup using Quadlets

Fran Torres — Thu, 22 Feb 2024 14:25:59 +0000

I've had an old Raspberry Pi Model B running Pi-Hole at home for quite a long time, but I always wanted to dig a bit deeper into using containers to deploy applications. Because of that, I recently purchased a RPi 4 and this week found some time to figure out a way to deploy it using podman.

Since I started reading about containers, podman and such, it always looked a bit convoluted the way to run an application from the command line; many parameters basically, I didn't want to have one script launching one 100 characters command.
I then learnt docker-compose files, which are nice, but in my absolutely ignorant opinion seem to be only useful for local services (like on a development environment to spin a database); things you don't want to live forever as a service. And the second problem was: I wanted to use podman and, ideally, only native podman-related solutions.
Fast forward until recently. I read about systemd-units and quadlets and liked what I saw, so here we are.

Quadlets

I won't go too deep into all the details about podman, systemd and quadlets as Daniel Schier did it already way better than I will ever be able to.
Basically you can split a container deployment within its multiple dependant components (volumes, network, the container itself...) and generate systemd services that can be easily managed. You can either do it manually by running the container from the command line (via podman) and then generate the systemd unit with podman generate systemd OR you can use quadlets to have a single centralised set of files that systemd will use to automatically generate the service units for you.

Decissions and future goals

[x] Use rootfull containers for now (it felt easier not having to worry about port mapping).
[x] openSUSE MicroOS will be the host system. I want to automate maintenance as much as possible and using an immutable & self updatable system is a first step.
[ ] Automatically update the container.
[ ] Use unbound.
[ ] Use rootless containers.

Deploying Pi-Hole via Quadlets

Previous step

Without going into much details, first of all make sure there are no processes blocking at least ports 53 (tcp and udp) and 80 tcp if Pi-Hole will only be acting as the DNS server and not DHCP.
For example, here's the default installation of openSUSE MicroOS and, as you can see there's nothing listening on those ports. For that we'll use the command ss -tunlp.

By contrast, a default installation of Fedora shows the service systemd-resolved listening on ports 53.

I'll leave that up for research if you need to disable the service, but there's plenty of documentation on the topic.

Provisioning needed files

As we are going to use a rootfull container, we need to place our Quadlets in /etc/containers/systemd/. We will create a bunch of files:

Two .volume files for the volumes Pi-Hole needs.
One .network file to configure the network for the container.
One .container file for the main container.

pihole-dnsmasq.volume

[Unit]
Description=PiHole dnsmasq Container Volume

[Volume]
Label=app=pihole

pihole-pihole.volume

[Unit]
Description=PiHole Data Container Volume

[Volume]
Label=app=pihole

pihole.network

[Unit]
Description=PiHole Container Network

[Network]
Label=app=pihole
DisableDNS=true

pihole.container

[Unit]
Description=PiHole Service Container

[Container]
Label=app=pihole
ContainerName=pihole
HostName=pihole
Image=docker.io/pihole/pihole:latest
Network=pihole.network
Volume=pihole-pihole.volume:/etc/pihole
Volume=pihole-dnsmasq.volume:/etc/dnsmasq.d
AddCapability=NET_ADMIN
AutoUpdate=registry
PublishPort=53:53/tcp
PublishPort=53:53/udp
PublishPort=80:80/tcp
# Ignore the next 2 ones if not using pi-hole as dhcp server and not using https
PublishPort=67:67/udp
PublishPort=443:443/tcp

Environment=DNSMASQ_LISTENING=all
# Next variables to be customised
Environment=WEBPASSWORD="pihole"
Environment=TZ=Europe/Madrid
Environment=DNS1=208.67.222.222
Environment=DNS2=208.67.220.220
Environment=TEMPERATUREUNIT=c

[Service]
Restart=unless-stopped

[Install]
WantedBy=multi-user.target default.target

Change the webUI password to something better, set your timezone and temperature unit, and choose an upstream DNS provider (the one above is OpenDNS).

Generate and run service

Let's ask systemd to generate services from these files now. Run the following command as root:

systemctl daemon-reload

And lastly, run the service. As we said at the beginning this service is rootfull, so needs to be run as root. Just run the next command in a terminal and you should be done:

systemctl start pihole.service

If there are any failures you can check what happened with systemctl status pihole.service.

I have created a repository to hold this code for future use.

References

Linux Desktop: rolling releases and immutability

Fran Torres — Fri, 01 Apr 2022 14:51:55 +0000

It's been a long time since my last (and first!) post, but I felt like I had nothing to share. I would like to start writing more often but family obligations, work and impostor syndrome keeps pushing me away from it.
Anyway, after writing the btrfs and snapper setup for Fedora 33 and comparing it to openSuse's btrfs setup I realized that while it works, it's not as convenient as it could be.

I am lazy. I can follow complex processes for my work but not when dealing with my computer. I expect it to work with as little interaction as possible.

What should linux desktop look like (to me)?

I've been a linux desktop user for quite a while now and year by year I've added more requirements to my idea of an ideal linux desktop. Right now I could consider it something like this:

As stable as possible, with recovery options when an update goes wrong. This includes not having to look at the updates too much, just consider they will work. Otherwise having an integrated recovery system.
At least the kernel should be as up to date as possible, so that if I buy new hardware I don't need to consider switching to another distro.
Updated packages or not too old. Especially regarding desktop environments.
Closer to stock components if possible. I've got tired of customizations.

Now, the options I had:

Debian: stable is not an option unfortunately and when I've used testing or sid I've always had to be vigilant about the upgrades to avoid problems from time to time.
The Ubuntu team is doing a great job with the Enablement Stack, but you will still be running some software that's years old like the desktop. Running the latest version and upgrade every 6 months could be an option, but I was too used to Debian based distros and wanted a change.
OpenSuse Leap since version 15 is interesting. Basically every 2 years the new release will bring new kernel, desktops, etc; which will be a somewhat slow rolling experience. But that's still 2 years of waiting for major component changes. I had good experiences with openSuse in the past, their implementation of btrfs and snapshots is probably the best, so Tumbleweed could be an option.
Arch and Manjaro: I've ran Manjaro in the past, and having that testing repository where they basically hold arch updates for a while helps a lot. You have to delete customizations though.
Fedora: looked like a good balance between up-to-date components and stability, but lacked a polished recovery mechanism. Then I discovered Silverblue and made me consider it.

I've been running both systems for a while in the last year and here are my thoughts.

openSuse Tumbleweed

It is a great system, I can't deny it. The OpenQA is great and the amount of testing they do is amazing.
There's an article from Richard Brown explaining why regular release distributions are not an option for the desktop in his opinion, and while I subscribe pretty much all of his thoughts, because of the insane amount of software that can be considered part of any distro, it's almost impossible to ensure that everything will work.

For example, with the recent upgrade to Gnome 42 there are some broken components like the share section in configuration or cheese showing weird color artifacts. Are they deal breakers? Probably not, but that brings me uncertainty that maybe someday something I need breaks. The good thing is that they react quite quickly with fixes from upstream. The solution could have been just to not upgrade the system until Gnome 42.1 was released, but as I said I don't want to need to spend too much time checking what's being updated. I could have also rolled back and that would have helped because it's super easy in openSuse, but I was afraid that the next time I tried to upgrade, something else would break.

MicroOs looks super promising. They're not using any new technology, they're just changed the way the system does the updates and having the whole system as read only (over simplifying). But right now it's in early stages, the base image is incredibly small and requires more work in general, but once they have it, that is going to be the real Tumbleweed experience.

Fedora Silverblue

I'm not going to talk about immutable OSs now, there are quite a few talks in youtube, in blogs, etc. But basically the idea is that the core system will be read only, so if an update fails you can always go back to exactly the same state you were previously.

The fact that the base system can't be modified by the user brings some challenges mainly to developers, but also regular users. Where do you install your development libraries? How do I install applications?

For the first question toolbx and podman should be enough. Even more considering that most devs are getting used to containers nowadays.
The second question in general can be answered with flatpaks, which are a great way to have distribution independent applications that are always updated. In reality everyone is still trying to figure out how to put all these pieces together in the best way and sometimes you find a problem with a flatpak app that wouldn't happen with the distribution packaged version. Bear in mind that for basically every technology a flatpak app wants to use from the underlying system, some sort of bridge has to be built and not everything works right now.
Sometimes you'll have no other choice than to install the native packages into the system (what's called layering in Silverblue), but you shouldn't if possible.

I've been able to live with Silverblue up to the point where if I have to install linux to my parents I think I'll just use it. Flatpak experience has been good in general and it is improving every day. Hopefully developers will continue to adapt their flatpaks to build from source and not just use the packaged sources. Maybe this is just me being picky, but looks safer to me than just repackaging another format into flatpak.
Did I have to layer any packages? Yes. Unfortunately the Spanish ID smartcard requires 2 packages to be installed as RPMs directly and that forces you to have a browser installed natively (not as flatpak), so I have Chromium for that and Firefox as flatpak. Scanners don't work via flatpak, so your scanning application must be layered too. And lastly the nvidia drivers for my laptop. They are the only pain point of this setup because if there's a new kernel version coming I know I have to hold it for a week or so until a new version of the drivers is released. But only keeping an eye on the kernel I guess it's a price I'll have to pay. In my desktop with AMD hardware I can blindly install updates though.

Closing thoughts

I can't find a reason right now why I would need to change to a different distribution. The upgrade process even with regular Fedora has become incredibly safe, painless and nicely integrated with Gnome Software. Do you know what's the only problem I have with it? Being able to wait when the new version is released. I've done 33 to 34 and 34 to 35 and it went fine, but it's always better if you wait a month until things have settled a bit. You could even stay one version behind and you would still get kernel updates, but having an easy way to rollback I'm not considering it.
Having to layer some packages is not ideal definitely and you must adapt your workflow when developing, but I'm taking it as an opportunity to learn more about containers and use them more frequently professionally and in my free time.

I also have the feeling that I might have not been fair with Ubuntu and I could follow the same flow of upgrading every 6 months to the new release (waiting for a month for it to stabilize); and also that judging Tumbleweed because of 2 small issues with an update is not great and I'm sorry about it, but I always feel way more stressed than I should when something breaks and I need to find some balance and especially some peace of mind that I won't need to fix my computer a random day.

Fedora 33 & BTRFS Snapshots

Fran Torres — Mon, 25 Jan 2021 16:04:05 +0000

Update (27th June 2022): Since I wrote this I've switched to silverblue, which led me to forget a bit about btrfs snapshotting in Fedora because you get that for free with rpm-ostree. But if anybody reaches this I've found a more complete article describing the whole process for Fedora 36 and adding disk encryption that might be worth reading before mine.

Why Fedora and Btrfs?

Distrohopping has been a major pain when using GNU/Linux. Debian Stable has been my main choice for many years but it's not perfect when dealing with modern hardware. I came back to Fedora after many years not using it with Fedora 31 and found that the system has become way smoother and stable than I remembered. Even the upgrades have become an almost painless process.
As great as Fedora is right now, I still have that feeling that something might fail someday and I wanted to have a recovery system configured. Fedora Silverblue looks terrific and I want to be ready to use it, but I'm not yet unfortunately (I still have to get my head around the one purpose containers and podman).
Until then, btrfs and snapshots seem to be the best option for me.

Fedora 33 base install

The default Fedora 33 installation creates a disk structure like this:

One 1Gb ext4 partition for /boot
One Btrfs volume for the rest of the space
One Btrfs subvolume called "root" mounted at /
One Btrfs subvolume called "home" mounted at /home

[franute@localhost ~]$ df -h | grep vda
/dev/vda2        35G  7.1G   28G  21% /
/dev/vda2        35G  7.1G   28G  21% /home
/dev/vda1       976M  183M  727M  21% /boot

[franute@localhost ~]$ sudo btrfs sub list /
ID 256 gen 175 top level 5 path home
ID 258 gen 179 top level 5 path root
ID 265 gen 28 top level 258 path var/lib/machines

My main volume is called /dev/vda2 and the root btrfs subvolume has ID 258. Write down your volume path and ID as we'll need them later.

Install & configure snapper

Next I will install snapper and the dnf plugin to generate pre and post transactions every time something is installed on the system and create a configuration for root.

[franute@localhost ~]$ sudo dnf install -y snapper python3-dnf-plugin-snapper
...
Complete!
[franute@localhost ~]$ sudo snapper --config=root create-config /

This will create a subvolume called .snapshots in /, but we would rather have it at a different level to be able to mount it manually if needed and not nested under the "root" subvolume.

[franute@localhost ~]$ sudo btrfs sub list /
ID 256 gen 965 top level 5 path home
ID 258 gen 1038 top level 5 path root
ID 265 gen 966 top level 258 path var/lib/machines
ID 267 gen 1038 top level 258 path .snapshots

Delete that subvolume and create the new one.

franute@localhost ~]$ sudo btrfs sub delete /.snapshots
Delete subvolume (no-commit): '//.snapshots'
[franute@localhost ~]$ sudo mkdir /.snapshots
[franute@localhost ~]$ sudo mkdir /mnt/btrfs
[franute@localhost ~]$ sudo mount /dev/vda2 /mnt/btrfs
[franute@localhost ~]$ cd /mnt/btrfs/
[franute@localhost btrfs]$ ls
home  root
[franute@localhost btrfs]$ sudo btrfs sub create snapshots
Create subvolume './snapshots'
[franute@localhost btrfs]$ ls
home  root  snapshots
[franute@localhost btrfs]$ cd ..
[franute@localhost mnt]$ sudo umount /mnt/btrfs
[franute@localhost mnt]$ sudo rmdir /mnt/btrfs/

Now, let's modify /etc/fstab to mount the snapshots subvolume at boot.
Change it to look similar to this (do not change your UUID, just focus on the options after ext4 or btrfs):

UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6 /                       btrfs   defaults     0 0
UUID=1d095325-9db9-4ac9-af8f-54d698535a55 /boot                   ext4    defaults        1 2
UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6 /home                   btrfs   subvol=home     0 0
UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6 /.snapshots             btrfs   subvol=snapshots     0 0

I have added a new entry at the end and changed the subvol=root from the first entry to just defaults. The next change is important. Do you remember that subvolume ID from before (258)? We need to tell the system that we want that to be considered the default when mounting /dev/vda2 if no subvolume is specified. There's also a problem with how Fedora configures the bootloader options by default as it will try to boot from this very specific subvol, making the rollbacks useless.
Now, we will set that subvol as the default and remove a specific arg from the bootloader:

[franute@localhost ~]$ sudo btrfs sub set-default 258 /
[franute@localhost ~]$ sudo grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.8.15-301.fc33.x86_64"
args="ro rootflags=subvol=root rhgb quiet"
root="UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6"
initrd="/boot/initramfs-5.8.15-301.fc33.x86_64.img"
title="Fedora (5.8.15-301.fc33.x86_64) 33 (Workstation Edition)"
id="c6bb0926115440f9b9f721746626df37-5.8.15-301.fc33.x86_64"
index=1
kernel="/boot/vmlinuz-0-rescue-c6bb0926115440f9b9f721746626df37"
args="ro rootflags=subvol=root rhgb quiet"
root="UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6"
initrd="/boot/initramfs-0-rescue-c6bb0926115440f9b9f721746626df37.img"
title="Fedora (0-rescue-c6bb0926115440f9b9f721746626df37) 33 (Workstation Edition)"
id="c6bb0926115440f9b9f721746626df37-0-rescue"

[franute@localhost ~]$ sudo grubby --update-kernel=ALL --remove-args="rootflags=subvol=root"
[franute@localhost ~]$ sudo grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.8.15-301.fc33.x86_64"
args="ro rhgb quiet"
root="UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6"
initrd="/boot/initramfs-5.8.15-301.fc33.x86_64.img"
title="Fedora (5.8.15-301.fc33.x86_64) 33 (Workstation Edition)"
id="c6bb0926115440f9b9f721746626df37-5.8.15-301.fc33.x86_64"
index=1
kernel="/boot/vmlinuz-0-rescue-c6bb0926115440f9b9f721746626df37"
args="ro rhgb quiet"
root="UUID=279ddd2e-e753-483f-8ea9-3a8eab37f5e6"
initrd="/boot/initramfs-0-rescue-c6bb0926115440f9b9f721746626df37.img"
title="Fedora (0-rescue-c6bb0926115440f9b9f721746626df37) 33 (Workstation Edition)"
id="c6bb0926115440f9b9f721746626df37-0-rescue"

Now restart, if you now check the default subvolume for / you should see 258:

[franute@localhost ~]$ sudo btrfs sub get-default /
ID 258 gen 1093 top level 5 path root

Snapshots

By default there will be no snapshots in the system, but as soon as you use dnf to install something from the command line, it will generate 2 snapshots. Let's see what happens installing htop for example:

[franute@localhost ~]$ sudo snapper ls
 # | Type   | Pre # | Date | User | Cleanup | Description | Userdata
---+--------+-------+------+------+---------+-------------+---------
0  | single |       |      | root |         | current     |         
[franute@localhost ~]$ sudo dnf install -y htop
...
Complete!
[franute@localhost ~]$ sudo snapper ls
 # | Type   | Pre # | Date                         | User | Cleanup | Description                  | Userdata
---+--------+-------+------------------------------+------+---------+------------------------------+---------
0  | single |       |                              | root |         | current                      |         
1  | pre    |       | Sun 24 Jan 2021 18:58:37 WET | root | number  | /usr/bin/dnf install -y htop |         
2  | post   |     1 | Sun 24 Jan 2021 18:58:38 WET | root | number  | /usr/bin/dnf install -y htop |

You can see there are 2 new snapshots now: 1 is a snapshot with the state of the system before htop was installed and 2 has the current state (right after it is installed).

Rollback

Let's say that for some reason we want to rollback our system to right before installing htop. We can do it with the rollback operation from snapper:

[franute@localhost ~]$ sudo snapper rollback 1
Creating read-only snapshot of current system. (Snapshot 3.)
Creating read-write snapshot of snapshot 1. (Snapshot 4.)
Setting default subvolume to snapshot 4.

After restarting the system, you'll see that the default subvolume has changed now:

[franute@localhost ~]$ sudo btrfs sub list /
ID 256 gen 1212 top level 5 path home
ID 258 gen 1174 top level 5 path root
ID 265 gen 1060 top level 258 path root/var/lib/machines
ID 268 gen 1167 top level 5 path snapshots
ID 269 gen 1166 top level 268 path snapshots/1/snapshot
ID 270 gen 1110 top level 268 path snapshots/2/snapshot
ID 271 gen 1166 top level 268 path snapshots/3/snapshot
ID 272 gen 1212 top level 268 path snapshots/4/snapshot
[franute@localhost ~]$ sudo btrfs sub get-default /
ID 272 gen 1213 top level 268 path snapshots/4/snapshot
[franute@localhost ~]$ sudo snapper ls
 # | Type   | Pre # | Date                         | User | Cleanup | Description                  | Userdata     
---+--------+-------+------------------------------+------+---------+------------------------------+--------------
0  | single |       |                              | root |         | current                      |              
1  | pre    |       | Sun 24 Jan 2021 18:58:37 WET | root | number  | /usr/bin/dnf install -y htop |              
2  | post   |     1 | Sun 24 Jan 2021 18:58:38 WET | root | number  | /usr/bin/dnf install -y htop |              
3  | single |       | Sun 24 Jan 2021 19:01:22 WET | root | number  | rollback backup              | important=yes
4* | single |       | Sun 24 Jan 2021 19:01:22 WET | root |         | writable copy of #1          |

We have in fact booted from snapshot 4, which is just a writeable copy of snapshot 1. The previous state is retained in snapshot 3, so you can even go back to that state.
Now let's check if htop is installed:

[franute@localhost ~]$ htop
bash: htop: command not found...
Similar command is: 'top'

Great! It worked.
Setting up Fedora to work with Snapper is not very different from any other distro. The only real problem was the kernel arguments. Once you get rid of that, it works as expected.

Improvements and considerations

Nested subvolumes will not be considered in the snapshots, you can use that to ignore certain folders. /var/log is a good candidate in that case, here's my setup.

ID 256 gen 21503 top level 5 path root
ID 257 gen 21516 top level 5 path home
ID 261 gen 18514 top level 256 path var/lib/portables
ID 266 gen 18514 top level 256 path var/lib/machines
ID 269 gen 21515 top level 256 path var/log
ID 270 gen 17131 top level 5 path snapshots
ID 310 gen 17129 top level 270 path snapshots/1/snapshot

There's a problem with Gnome Software and apparently software upgrades performed with it will not generate pre and post snapshots, I'd like to dig into this and see how it works.
You can tweak this root configuration we created in several ways. For example you can enable or disable timed snapshots, define when are they cleaned up, how many to store in the system...