My homelab PC went to an emergency shell during its default boot at 19:58:37 on Nov 8, 2026, and every boot after that looked fine. Fine because I kept choosing the snapshot entry by hand on the same menu where the default entry drops you into that shell. Nothing downstream of the pick ever reported the failure, and my whole working day ran on top of a system whose normal path was already dead.
When your recovery route sits one keyboard pick from the front door, the broken path and the working path look identical to you, and only the journal separates them. That's the shape worth naming before any fix, because a boot that succeeds only proves the tree it booted from was good.
What the dead boot said
The failed entry's own kernel cmdline carried rootflags=subvol=@, the plain subvolume, and then the log ran in this order:
systemd-modules-load: Failed to insert module 'crypto_user'
error=No such file or directory for every /lib/modules/7.2.5-3-omarchy/**/*.ko.zst
boot.mount: Failed with result 'exit-code'
Failed to mount /boot.
Dependency failed for Local File Systems
local-fs.target failed
Started Emergency Shell
Seven log lines carry the chain, one missing file class causes all of it. The running kernel wanted 6,490 compressed modules and the tree it booted from had zero of them under that kernel version's directory.
Meanwhile llama-swap.service reports active since 19:59:42 and both RTX 3090s answer nvidia-smi at 19.6 GB and 23.3 GB used. Everything I use was up. The thing that was down was a path I wasn't using, and nothing in the desktop separates those two states.
Counting files inside the thing you mounted read-only
Finding the cause meant mounting the subvolume the broken entry points at, read-only, next to the live system, and comparing the same path across two trees. Same directory, two answers: 6,490 modules on the tree I actually booted from, 0 inside @. That directory still held 1.1 GB of other kernel versions, and the rest of the @ tree looked complete at 23 top-level entries and 334,618 files under usr.
That's the trap in snapshot tooling generally. A rescue tree is a full userspace, so it passes every casual glance you'd give it, and the one file class your running kernel needs is missing from it. You can list it, stat it, walk its parents, and never notice until you count.
Why nobody reported the death
The upgrade that emptied it quit mid-extraction. The last line of pacman.log reads [ALPM] error: could not extract /usr/lib/libwebkit2gtk-4.1.so.0.24.4 (Write failed) at 2026-11-06T16:30:39. Two snapper snapshots bracket that attempt, snapshot 6 stamped 16:26:42 and snapshot 7 stamped 16:27:12, thirty seconds apart, and the failed write landed three minutes and twenty-seven seconds after snapshot 7.
The disk had 12 GB free out of 930 GB formatted, with 917 GB used and 99% of that capacity claimed, and one weights directory holding 366 GB of the pile. So a package manager ran out of room mid-extraction on a filesystem already at 99%, left the userspace half-swapped against a kernel that was installed, and returned nothing to anything reading later. The entire record of the event is one line in pacman.log, and it's still the last line in that file.
Three units fail every day on a box that looks alive
The healthy system carries its own permanent noise, all of it caused by the same overlay root:
systemd-remount-fs exits 32 with mount: /: fsconfig() failed: overlay: No changes allowed in reconfigure.
The swap unit points at /swap/swapfile, which doesn't exist. zram0 at 62.7 GB covers real swapping, so hibernation is just quietly dead.
snapper-cleanup errors IO Error ... subvolume is not a btrfs subvolume, because / is an overlay.
That third line means snapshot pruning never runs. The count climbs toward whatever upgrade comes next, the disk sits at 99%, and the unit that was supposed to clean up after itself has been erroring since the root turned into an overlay. None of it changes your desktop, your prompt, or a single fan. That ledger lives in systemctl --failed.
The repair had to clear two rules
No package updates on this box, no teardown of the inference lanes, both from my standing rules for it. Repointing the main Limine entry's rootflags at the known-good snapshot subvolume satisfies both, does what my manual pick already did, moves nothing on the disk, and unwinds with one cp against a backup I kept at /boot/limine.conf.bak-20261108-bootfix.
Two other routes got rejected before anyone ran them. omarchy snapshot restore is the distro-native path, but its confirm needs a tty and dies off-console over ssh, which is how I work on this machine. Subvolume rename or send surgery would have invalidated every other snapshot entry's cmdline path, because each of those still points at the old name. On a boot path, an edit you can't undo from a terminal isn't an edit worth making.
The guard inside the patch script
The script asserts its precondition before touching the config: exactly one match of rootflags=subvol=@ rw rootfstype=btrfs in /boot/limine.conf. Zero matches or two means the file drifted while I was composing the edit, so it exits without writing, and it copies the original to the backup path first either way. It also arrived as a file over scp into sudo python3, never an inline heredoc through ssh, because quoting through two shells is where these kinds of edits get mangled.
Verification was grep counts on the live file after the write: 0 remaining occurrences of the old flag string, 2 lines carrying the snapshot path, and a diff between backup and current that showed only line 31 changed. Reading the config back is what makes a bootloader edit trustworthy; a silent successful write proves nothing about which entry the menu will pick.
What shipped, what didn't
Shipped: a default boot entry aimed at a tree holding the modules the running kernel needs, backed up byte-for-byte, checked by grepping the config after the write. Unshipped: nobody has rebooted that box since the edit. An ssh session can't press a power button, so the claim stays unproven until someone picks the default entry at a console and watches it come up without menu help.
The second open item is a trap worth handing to anyone else with a nearly full root filesystem: clear the space before an upgrade runs, because during one the room is already gone and the extraction dies where it stands. Package caches and old model tiers are the pile worth cutting first, and the blind pruning loop above is why that pile keeps growing.
How many things on your machine have a fallback one step deep? Mine had a boot path, a swap path, and a snapshot pruner all failing in silence while the desktop looked normal. journalctl -b -1 on the last reboot you didn't attend is the fastest way to find out. What does it say?
Top comments (0)