DEV Community

prodbymarcu
prodbymarcu

Posted on

A package upgrade on a full disk broke my boot path

One index, two trees

Two entries on my homelab PC's boot menu load byte-identical kernels, one into an emergency shell at 19:58:37 on Nov 8, 2026, the other into a working desktop from the same screen. The difference between them is one token on one line, and the file that holds that line opens with a bare integer. That's the half nobody writes up when they're chasing a dead boot, because all the attention goes to whatever the failed attempt logged.

The front door is an index into stanzas

/boot/limine.conf starts its default stanza with default_entry: 2. Limine resolves front-door entries by index into stanzas further down the same file, so the number tells you which block gets mounted, nothing tells you what that block is. Each stanza ends in its own mount flags, and those flags are where the two boots actually differ. The front door is the second of them.

The distro keeps its own copy beside the live config, limine.conf.old, sitting next to /boot/limine.conf before I touched either one. A diffable artifact exists on every bootloader edit whether you make one or not, and reading it beats re-deriving which entry changed.

Same binary, different tree

Both boots report the same UKI path and the same hash family, os-release reads 4.0.4 for BUILD_ID and VERSION_ID on both, and the running kernel is 7.2.5-3-omarchy in each. The cmdlines part company at one flag.

rootflags=subvol=@
rootflags=subvol=/@/.snapshots/6/snapshot

Same kernel bytes, same initrd bytes, two different userspaces underneath. That narrows the search before anyone opens a log. The kernel loaded fine on both attempts, and the tree it was pointed at held what the running kernel needed on one of them and not the other.

An entry name tells you how you got there, the flag string tells you where you land, and nothing in either one hints at which tree gets mounted.

Any tool that edits boot entries by label is editing the wrong axis. What matters is the flag string inside the stanza your index resolves to, and finding it means resolving that index against the rest of the file.

What your disk is actually made of

The root volume sits on /dev/mapper/root, LUKS holding btrfs, and the operating system isn't what fills it. Under one home directory live 99 GB of services, 64 GB of fish, and 51 GB of voicechat, all under my model weights. The pacman package cache carries another ~9.6 GB in its own @pkg subvolume.

The piles you own

So the room an upgrade needs doesn't come from the distro, it comes from folders you own, and a package manager asks for that headroom without checking whether the extraction fits before starting. Nothing reclaims the pile automatically either, snapshot pruning included, since the cleanup unit errors while the root is an overlay and the count just keeps climbing toward whatever update runs next.

Why this is worth checking before your next upgrade

A full-disk interruption mid-extraction leaves a userspace half-swapped against a kernel that installed fine, and the only record is a line at the tail of a log nobody opens. Two things decide whether that state is reachable from your front door: how much room your largest owned folders leave behind, and which stanza your default index points at.

Check both while the machine still boots, from whichever session survived. Every number above came off ssh on Nov 8, 2026, from inside the snapshot boot, with the broken subvolume mounted beside the live one, two separate trees on one machine. Nothing in that pass moved data, rebooted anything, or asked the box to prove itself.

df -h says what's claimed, lsblk -f and cat /etc/fstab say what the mounts expect, journalctl --list-boots turns your last reboot you didn't attend into a queryable session, and mounting a subvolume with -o ro,subvol=/@ beside the live system compares two trees without moving anything. All of it read-only, all of it over ssh, none of it needs a power button.

How many lines in your bootloader do you actually know? Mine has one flag per entry, and the difference between two of those flags decided whether I got a desktop or an emergency shell.

Top comments (0)