DEV Community: The Homelab Postmortem

llama-cli prints the error and exits 0. The same binary exits 1 when the model is missing.

The Homelab Postmortem — Mon, 31 Aug 2026 22:28:19 +0000

Point llama-cli at a file that isn't there:

$ llama-cli -m gemma-3-1b-it-Q4_K_M.gguf \
    --audio /tmp/does-not-exist.wav \
    -p test -n 1 --no-warmup --simple-io --single-turn
...
Error: file does not exist or cannot be opened: '/tmp/does-not-exist.wav'
...
$ echo $?
0

It printed the error and returned success. Both of those are true at the
same time, and only one of them is visible to the script that called it.

It is not that the exit status is meaningless here

That was my first assumption and it is wrong, which matters, because "this tool
doesn't do exit codes properly" is a much less useful thing to know than what is
actually happening. Three controls, same binary, same machine:

# Missing media file
$ llama-cli -m model.gguf --audio /tmp/does-not-exist.wav ... ; echo $?
Error: file does not exist or cannot be opened: '/tmp/does-not-exist.wav'
0

# Missing image, to check this isn't audio-specific
$ llama-cli -m model.gguf --image /tmp/nope.png ... ; echo $?
Error: file does not exist or cannot be opened: '/tmp/nope.png'
0

# Same missing file, through llama-mtmd-cli instead
$ llama-mtmd-cli -m model.gguf --audio /tmp/does-not-exist.wav ... ; echo $?
1

# Missing *model*, through llama-cli
$ llama-cli -m /tmp/nope.gguf ... ; echo $?
gguf_init_from_file: failed to open GGUF file '/tmp/nope.gguf'
1

The same binary returns 1 for a missing model and 0 for a missing media
file. So the machinery to fail is present and works; one specific path does
not use it. llama-mtmd-cli returning 1 on the identical input rules out the
other easy explanation — this is not a limitation of what the media loader can
express upward.

What the code actually does

From tools/cli/cli-context.cpp on master. The staging function reports failure
honestly and says nothing:

bool cli_context::stage_media_file(const std::string & fname, const std::string & type) {
    std::ifstream file(fname, std::ios::binary);
    if (!file) {
        return false;
    }

Its caller — the loop that handles media given on the command line — prints the
message I saw, and then leaves the loop:

for (auto & fname : params.image) {
    if (!stage_media_file(fname, media_type_from_ext(fname))) {
        ui::show_error(string_format("file does not exist or cannot be opened: '%s'", fname.c_str()));
        break;
    }
    ui::show_message(string_format("Loaded media from '%s'", fname.c_str()));
}
buffer = params.prompt;

break ends the media loop and nothing else. The next statement continues
into ordinary prompt handling, which is why the model still answered my prompt
in the run above — it answered it without the audio I asked for, having told me
so in a line that changed nothing.

And the function containing all this has exactly one exit:

int cli_context::run() {
    ...
    return 0;   // the only return in the function
}

There is no path through run() that returns non-zero. Not "there is one
and this case misses it" — there isn't one.

Where the upstream report and this disagree

The issue — open,
bug-unconfirmed, no comments, no fix — attributes this to run() discarding
the return value of generate_completion(). That is a real defect: the function
is declared bool, and its caller does not look at the result.

But it is not what fires here. A missing file on the command line is handled
before generation starts, breaks out of the media loop, and never reaches that
call. Generation then runs normally and succeeds.

So there are two independent routes to the same symptom — a staging failure
that breaks without propagating, and a generation failure whose return value is
dropped — and they converge on a run() that cannot report failure either way.
Fixing only the one named in the report would leave the case in the report's own
reproduction command still returning 0.

What I verified and what I didn't

Reproduced on Linux x86_64, build b10703, in a throwaway container, with a
text-only model — no multimodal model is needed, because the failure happens
while staging the file, before anything looks at what it contains. The upstream
report is from Darwin arm64, build 10706. Different OS, different
architecture, different build, same behaviour.

I did not verify the generation-failure route by triggering it; I read it in
the source. I did not run the reporter's zero-byte-PNG cases. And I have not
checked whether any of this differs on macOS beyond taking the report at its
word.

If you call this from a script

Until it is fixed, llama-cli's exit status does not tell you whether your media
was loaded. Check the thing you actually care about instead:

out=$(llama-cli -m model.gguf --image "$img" -p "$prompt" ... 2>&1) || exit 1
if grep -q 'does not exist or cannot be opened' <<<"$out"; then
  echo "media failed to load" >&2
  exit 1
fi

Grepping output is a bad interface and I would rather not recommend it. It is
the only signal that is currently correct.

llama-mtmd-cli returns 1 on the same input, so if your use is multimodal
anyway, it is the better call today — not because it is a workaround, but
because it is the tool that reports what happened.

The generalisable habit

The error text and the exit status came from two different places in the
program, and nothing keeps them consistent. ui::show_error is a print. return 0 is a claim. A tool can do both in the same run, and every layer above it —
your shell, your set -e, your CI step, your retry loop — is reading only the
second one.

So the habit is narrow: when you automate a tool, verify that its failure
actually reaches its exit status, using a case you know is broken. Point it at
a file that does not exist and read $?. It takes one command, and it is the
only way to find out whether the thing your automation depends on is load-
bearing or decorative.

The wider version is about which artefact you trust. A message on the terminal
is written for a human who is watching. An exit status is the only thing a
program downstream can see. When those two disagree, the human sees a failure
and the machine sees success — and the pipeline keeps running on the machine's
version.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

cloud-init calls your user-data valid. It will never read the ssh_import_id in it.

The Homelab Postmortem — Sat, 29 Aug 2026 04:21:25 +0000

TL;DR: On Raspberry Pi OS with cloud-init, a top-level ssh_import_id: in your user-data is only ever read for a user marked default. Raspberry Pi's own documented example creates a named user, which is not default, so the key is silently skipped — logged at debug, never at warning. cloud-init schema validates the file as correct, the ssh_authorized_keys you put next to it works fine, and the imported keys never arrive. On a headless box the first symptom is that you cannot log in with the key you expected.

The symptom

Raspberry Pi Imager 2.0 generates cloud-init configuration by default, and Raspberry Pi's own announcement shows this shape for creating your user:

users:
  - name: pi
    groups: users,adm,dialout,audio,netdev,video,plugdev,cdrom,games,input,gpio,spi,i2c,render,sudo
    shell: /bin/bash
    lock_passwd: false
    plain_text_passwd: mysecretpassword123
    ssh_authorized_keys:
      - ssh-ed25519 mykeystuff
    sudo: ALL=(ALL) NOPASSWD:ALL

That page does not mention ssh_import_id, so if you want your keys pulled from GitHub rather than pasted in, you go to cloud-init's documentation and add it — at the top level, which is where the schema defines it:

#cloud-config
users:
  - name: pi
    groups: users,adm,sudo
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyMaterialHere
    sudo: ALL=(ALL) NOPASSWD:ALL
ssh_import_id:
  - gh:mitoyosh

Check it before you flash, because that is the responsible thing to do:

$ cloud-init schema --config-file user-data
Valid schema user-data
$ echo $?
0

Boot the Pi. The static key in ssh_authorized_keys is in ~/.ssh/authorized_keys exactly as promised. The keys from gh:mitoyosh are not, and never will be. Nothing raises an error to change that: the skip is a LOG.debug and a continue, with no exception and nothing recorded as a failure, so there is no reason for the run's status to be anything but done — which is what cloud-init status reports on this machine.

Why it's easy to misdiagnose

Three separate things tell you the configuration is fine.

The schema says so. This is not a typo being tolerated — the shipped schema explicitly defines ssh_import_id as a top-level key:

$ python3 -c "import json;d=json.load(open('/usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json'));print(json.dumps(d['\$defs']['cc_ssh_import_id']))"
{"type": "object", "properties": {"ssh_import_id": {"type": "array", "items": {"type": "string", "description": "The SSH public key to import."}}}}

So cloud-init schema is correct to pass it. The key is valid cloud-config. It is simply not read on this path.

The other key in the same block works. ssh_authorized_keys is nested under the user, and nested keys are read. So you get a machine where one of the two SSH directives you wrote took effect. That is much more confusing than neither working, because it rules out the whole category of "my user-data wasn't applied."

Nothing is logged above debug level. The module runs — it is not skipped — and then declines to do anything, quietly:

for user, user_cfg in users.items():
    import_ids = []
    if user_cfg["default"]:
        import_ids = util.get_cfg_option_list(cfg, "ssh_import_id", [])
    else:
        try:
            import_ids = user_cfg["ssh_import_id"]
        except Exception:
            LOG.debug("User %s is not configured for ssh_import_id", user)
            continue

LOG.debug, not LOG.warning. It does not appear in the default log level, and cloud-init status has no opinion about it.

What's really going on

The top-level ssh_import_id is read inside exactly one branch: if user_cfg["default"]. Every other user falls to the else, which looks only at a key nested under that user.

A user created through the users: list is not the default user. Reproducing that on the machine, with the shipped cloud-init and the config shape above:

$ python3 -c "import cloudinit; print(cloudinit.version.version_string())"
25.2

config placement   user   default   branch taken          import_ids
top-level          pi     False     else (nested only)    None
nested under user  pi     False     else (nested only)    ['gh:mitoyosh']

Same user, same default=False, same cloud-init. The only difference is where the key sits. In the top-level case the module reaches LOG.debug(...) and continues, and there is no other user for the top-level value to apply to.

Note the version: Raspberry Pi OS ships 25.2 here, from their own build (25.2-1~bpo13+1+rpt20), not the 25.1.4 that Debian Trixie carries. Check the box rather than the distro's package page — the behaviour above is what is actually installed.

It is known upstream, and the fix in flight is for a different case

Two trackers are open and neither is fixed:

raspberrypi/trixie-feedback#98 — opened 2026-08-13, no comments.
canonical/cloud-init#4306 — open, labelled bug, eight comments, reported by a cloud-init contributor.

There is an open PR, #7050, and it is easy to assume it covers this. It does not. Its own commit message says what it changes:

Merge both sources for the default user. A top-level ssh_import_id keeps working on its own, and non-default users are unchanged.

That fixes a nested ssh_import_id being dropped for the default user. The case above — a top-level key with a non-default user — is explicitly left as it is. So merging that PR will not fix a Raspberry Pi Imager setup.

There is also a Raspberry Pi forum thread asking exactly this question. It has no replies.

The fix

Nest it under the user, alongside ssh_authorized_keys:

#cloud-config
users:
  - name: pi
    groups: users,adm,sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyMaterialHere
    ssh_import_id:
      - gh:mitoyosh

Both keys are then in the branch that gets read, and both apply.

Two things worth knowing before you rely on it. ssh-import-id must be installed or the module warns and stops — that one is at warning level, so it will show up. And the import needs working networking and reachable GitHub/Launchpad at first boot; if the machine comes up on WiFi that takes a moment, the keys can be a boot behind.

Verify on the machine rather than by asking cloud-init how the run went. The skip records no error, so a successful-looking status is exactly what you would expect in both cases — it is not evidence either way:

wc -l ~/.ssh/authorized_keys
grep -c 'ssh_import_id' /var/log/cloud-init.log

If the first number only accounts for the keys you pasted literally, the import did not happen — regardless of what anything else reported.

The generalisable habit

Schema validation answers "is this a well-formed document?" It cannot answer "will anything read this?", and the two feel like the same question when the validator is shipped by the same project as the code. Here they diverge completely: the key is in the schema, the schema is right, and the code that consumes it has a condition the schema knows nothing about.

So the habit is narrow: a validator passing tells you the file parsed, not that the field is live on your path. When a config option quietly does nothing, do not re-read the file looking for a typo — go find the code that consumes the key and check what it requires. Here it was one if.

The wider one is about partial success. One SSH directive worked and one did not, and that is the shape that costs the most time, because working evidence is louder than absent evidence. You look at authorized_keys, see a key, and conclude the mechanism works — when what you have confirmed is that a different mechanism works. The same trap as a WiFi profile that exists with the right SSID and no credentials: the thing you can see being right is not evidence about the thing you cannot.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

iptables says your kernel needs upgrading. Upgrading the kernel is what broke it.

The Homelab Postmortem — Thu, 27 Aug 2026 02:14:13 +0000

TL;DR: On Raspberry Pi's 6.18 kernel line, ip_tables.ko, iptable_nat.ko and iptable_filter.ko are not built. This is deliberate — legacy iptables was deprecated in favour of nftables. But CONFIG_IP_NF_IPTABLES=m is still set in the shipped kernel config, because the symbol that actually builds the legacy modules is now the separately-named CONFIG_IP_NF_IPTABLES_LEGACY, and that one is unset. So the config appears to promise a module that no longer comes from it, the module directory is still full of ipt_* files, and the error you get tells you to upgrade the kernel you just upgraded.

The symptom

A Pi 4B on the current kernel:

$ uname -r
6.18.34+rpt-rpi-v8

Anything that shells out to legacy iptables — a WireGuard container's PostUp, a Docker setup pinned to the legacy backend, Waydroid's install script — fails the moment it tries to write a NAT rule. Reproduced directly, with the same binary on both backends:

$ sudo iptables --version
iptables v1.8.11 (nf_tables)
$ sudo iptables -t nat -L -n
$ echo $?
0

Fine. Now switch the alternative, change nothing else, and run the identical command:

$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo iptables --version
iptables v1.8.11 (legacy)
$ sudo iptables -t nat -L -n
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.18.34+rpt-rpi-v8
iptables v1.8.11 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Read that last line again. The kernel is the newest one available, and upgrading to it is what caused this. Doing what the error suggests moves you further from a working system, and it is the first thing anyone will try.

Why it's easy to misdiagnose

The error offers two suggestions and both are dead ends. "Do you need to insmod?" points at a module that will never exist on this kernel; "your kernel needs to be upgraded" points backwards. Neither mentions the word nftables, which is the actual answer.

Then you go looking, and the evidence on disk agrees with the error rather than with reality.

The module directory is not empty. This is the part that really costs time:

$ ls -1 /lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/
arpt_mangle.ko.xz
ipt_ah.ko.xz
ipt_ECN.ko.xz
ipt_REJECT.ko.xz
ipt_rpfilter.ko.xz
ipt_SYNPROXY.ko.xz
nf_defrag_ipv4.ko.xz
nf_dup_ipv4.ko.xz
nf_nat_h323.ko.xz
nf_nat_pptp.ko.xz
nf_nat_snmp_basic.ko.xz
nf_reject_ipv4.ko.xz
nf_socket_ipv4.ko.xz
nft_dup_ipv4.ko.xz
nft_fib_ipv4.ko.xz
nf_tproxy_ipv4.ko.xz
nft_reject_ipv4.ko.xz

Seventeen files, five of them named ipt_*. It looks like a healthy legacy-iptables install. The ipt_* entries are matches and targets — REJECT, SYNPROXY, rpfilter — and they still ship. What is missing is the three modules that provide the tables those targets would go into, and nothing about the listing draws your eye to an absence.

$ sudo modprobe -n -v ip_tables
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.18.34+rpt-rpi-v8
$ sudo modprobe -n -v iptable_nat
modprobe: FATAL: Module iptable_nat not found in directory /lib/modules/6.18.34+rpt-rpi-v8
$ sudo modprobe -n -v iptable_filter
modprobe: FATAL: Module iptable_filter not found in directory /lib/modules/6.18.34+rpt-rpi-v8

And the kernel config says the module is built. This is where most people stop and conclude they have found a packaging bug:

$ grep CONFIG_IP_NF_IPTABLES /boot/config-$(uname -r)
CONFIG_IP_NF_IPTABLES=m

=m means "build as a loadable module." The module is not there. That looks exactly like a broken kernel build, it is a reasonable thing to file a bug about, and people have — against four separate projects, none of which own the problem.

What's really going on

The config is not lying. It is answering about a different thing than the one you are asking about, because the meaning of that symbol changed underneath the name.

Two more lines from the same file settle it:

$ grep -E 'CONFIG_IP_NF_IPTABLES_LEGACY|CONFIG_NFT_COMPAT' /boot/config-$(uname -r)
CONFIG_NFT_COMPAT=m

CONFIG_NFT_COMPAT=m is present. CONFIG_IP_NF_IPTABLES_LEGACY produces no output at all — it is unset, and that is the symbol that builds ip_tables.ko today. CONFIG_IP_NF_IPTABLES kept the historic name and now selects the nftables-backed path instead.

Upstream said so plainly, over six months ago. Phil Elwell answered this on raspberrypi/linux#7220 in February 2026:

ip_tables has been deprecated in favour of nf_tables. CONFIG_IP_NF_IPTABLES enables an ip_tables-like shim over nf_tables.

That issue is closed as completed, which is worth pausing on: on GitHub, closed as completed normally means the bug was fixed. Here it means the question was answered and the behaviour is intended. If you check only the state and not the thread, you will conclude this was fixed and stop looking — and the modules will still be missing.

The nftables side of the same kernel is fully populated — 28 modules matching nf_tables/nft_ under kernel/net/netfilter/ — so this is not a kernel with its firewalling removed. It is a kernel with exactly one way to do firewalling, and a userspace shim (iptables-nft) that makes the old commands work against it.

Who this actually hits

Not everyone, and the reports do not make that clear.

A stock Raspberry Pi OS Lite install ships no iptables binary at all — it arrives as a dependency of something else, usually ufw, and update-alternatives resolves it to iptables-nft. On that path nothing ever touches a legacy module and none of this happens. A default Docker install on a default image does not hit it either.

What hits it is anything that deliberately selects the legacy backend, which used to be sound advice:

Container images whose startup scripts call legacy iptables for MASQUERADE/FORWARD rules — this is why WireGuard front-ends broke.
Distributions and provisioning scripts that pin update-alternatives to iptables-legacy, a long-standing workaround for older Docker/nftables incompatibilities.

Every one of those was a reasonable decision when it was made. They break now because the thing they pinned to stopped existing, and the failure surfaces inside whichever tool made the call rather than at the pin.

That is also why the fix information is so hard to find. The canonical report is wg-easy#2614, which the maintainers closed as not planned — correctly, since it is not their bug. So the thread with twenty comments of people rediscovering the same thing sits in a project that has, accurately, disclaimed it.

The fix

If you control the alternative, point it at the shim. This is the whole fix for most cases:

sudo update-alternatives --set iptables /usr/sbin/iptables-nft
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft

Then restart whatever failed. Verify you are actually on the shim rather than trusting the command's exit code:

$ sudo iptables --version
iptables v1.8.11 (nf_tables)

The (nf_tables) in that output is the thing to check. (legacy) there means you are still pointed at a backend with no kernel behind it, whatever else looks right.

If the caller is inside a container image you do not control, the alternative is not yours to set — the container has its own userspace. There the answer is to replace the legacy calls with nft equivalents in whatever hook the image exposes, which is what the wg-easy threads eventually converged on, or to run the container with host networking so the host's own rules apply.

And before you rely on any of this on a Pi, the check costs nothing:

modprobe -n -v ip_tables 2>&1

-n is dry-run: it resolves the module without loading it. If that prints FATAL: Module ip_tables not found, then every legacy-iptables tool on that machine is going to fail, and you know it before you have installed one.

The generalisable habit

The trap here is a config symbol that kept its name while changing what it produces. CONFIG_IP_NF_IPTABLES=m was true and useful for years, it is still present, and it now means something else. Nothing in the file marks the change — there is no deprecated flag, no comment, no warning at build time. The only way to see it is to know that a second symbol exists and to look for its absence, and you cannot grep for the absence of a name you have never heard of.

So the habit: when a config says a thing exists and the thing does not exist, suspect the symbol before you suspect the build. A missing artifact with a present config is much more often a renamed or re-scoped option than a broken toolchain — the toolchain failing loudly is the common case, and this failed quietly.

The wider one is about reading closed issues. closed as completed on an upstream tracker is a strong signal and it is genuinely useful — it correctly killed an investigation for me two days earlier, on this same tracker, where a bcmgenet regression really had been fixed and shipped to apt while the reports about it were still circulating. Here the same state on the same tracker meant "answered, working as intended." The state field tells you the thread is over. It does not tell you which way it ended. That is in the comments, usually the last few, and there is no way to skip reading them.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

tar backed up your Pi's WiFi config and exited 0. The archive holds one empty directory.

The Homelab Postmortem — Wed, 26 Aug 2026 02:26:38 +0000

TL;DR: On a stock Trixie Pi OS image, the WiFi credentials for the connection this Pi actually uses are stored in /etc/netplan/90-NM-<uuid>.yaml, not in /etc/NetworkManager/system-connections/. That second directory — the one every guide, every backup snippet and every "copy your config to the new card" workflow names — is completely empty. tar archives it happily, exits 0, and produces a 10 KB file containing a single directory entry and no credentials. You find out after you have already reimaged.

The symptom

The machine is a Pi 4B on Raspberry Pi OS Lite, current image:

$ cat /etc/rpi-issue | head -1
Raspberry Pi reference 2026-06-18
$ nmcli --version
nmcli tool, version 1.52.1

WiFi works. NetworkManager is running and knows about the connection:

$ nmcli -t -f NAME,TYPE,DEVICE con show
netplan-wlan0-HOMEWIFI:802-11-wireless:wlan0
lo:loopback:lo
netplan-eth0:802-3-ethernet:

So you do the sensible thing before reimaging — save the network config:

$ sudo tar cf /tmp/wifi-backup.tar -C /etc/NetworkManager system-connections
$ echo $?
0
$ ls -l /tmp/wifi-backup.tar
-rw-r--r-- 1 root root 10240 Aug 26 11:06 /tmp/wifi-backup.tar

Ten kilobytes. Exit zero. No warning. Here is what is in it:

$ tar tvf /tmp/wifi-backup.tar
drwxr-xr-x root/root         0 2026-08-26 10:09 system-connections/

One directory entry. Nothing else. Because:

$ sudo ls -la /etc/NetworkManager/system-connections/
total 8
drwxr-xr-x 2 root root 4096 Aug 26 10:09 .
drwxr-xr-x 8 root root 4096 Aug 26 10:09 ..

The directory is empty, has always been empty on this image, and the connection that is currently carrying your SSH session is not in it.

Why it's easy to miss

Three things line up to keep you from noticing.

The archive isn't empty. A 0-byte file would make you look. tar pads to a 10240-byte block, so an archive of one empty directory is the same size as an archive of a few small files. It looks exactly like a successful backup of a handful of .nmconnection files, which is what you expected to get.

NetworkManager's own config points you at the wrong place. If you go looking for where connections are stored, this is what you find:

$ grep -vE '^\s*#|^$' /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile
[ifupdown]
managed=false

keyfile is the plugin that reads and writes .nmconnection files. It is enabled, and NetworkManager --print-config — the effective configuration, drop-ins included — agrees:

$ sudo NetworkManager --print-config | grep -A1 '^\[main\]'
[main]
plugins=ifupdown,keyfile

None of that is wrong, which is what makes it unhelpful. The keyfile plugin reads three directories — /etc/NetworkManager/system-connections, /run/NetworkManager/system-connections, and /usr/lib/... — and only the first is writable and persistent. Netplan renders its YAML into the /run one at boot, so the keyfile plugin is genuinely doing all the work, on files it did not write, in a directory the config never names. There is a netplan.conf drop-in, but it lives in /run/NetworkManager/conf.d/ and only assigns device management:

[device-netplan.wifis.wlan0]
match-device=type:wifi
managed=1

Nothing in any file under /etc tells you netplan is holding your PSK.

There are .nmconnection files, and finding them makes you think you had the wrong path, not the wrong idea:

$ sudo ls -l /run/NetworkManager/system-connections/
total 12
-rw------- 1 root root 314 Aug 25 00:56 lo.nmconnection
-rw------- 1 root root 174 Aug 26 10:09 netplan-eth0.nmconnection
-rw------- 1 root root 295 Aug 26 10:09 netplan-wlan0-HOMEWIFI.nmconnection

There they are, in the format you expected, with the names you expected. It is easy to conclude the directory just moved and get on with your day. It didn't:

$ findmnt -no FSTYPE /run
tmpfs

Those are runtime copies on a tmpfs. They are regenerated at every boot from something else, and if you back those up instead you have captured a derived artifact rather than the source.

What's really going on

The source is in /etc/netplan:

$ sudo ls -l /etc/netplan/
total 8
-rw------- 1 root root 275 Aug 26 10:09 90-NM-75a1216a-9d1a-30cd-8aca-ace5526ec021.yaml
-rw------- 1 root root 590 Aug 26 10:09 90-NM-be1282f3-d98b-3db6-9c1f-0cd80398f4f5.yaml

Two files, mode 600, named after connection UUIDs rather than after anything a human would search for. The second one is the WiFi:

network:
  version: 2
  wifis:
    wlan0:
      renderer: NetworkManager
      match: {}
      dhcp4: true
      access-points:
        "HOMEWIFI":
          auth:
            key-management: "psk"
            password: "<REDACTED>"
          networkmanager:
            uuid: "be1282f3-d98b-3db6-9c1f-0cd80398f4f5"
            name: "netplan-wlan0-HOMEWIFI"

The pre-shared key is right there in plaintext, which is why the file is 600. This is the file that matters, and it is the one your backup missed.

Both stores are live, and you cannot predict which one a connection is in

This is the part that makes the situation genuinely confusing rather than merely surprising: both stores are live at once, and nothing on the machine announces which connection is in which.

The wlan0 and eth0 profiles on this machine are netplan YAML. A connection created with nmcli is not — it lands in the keyfile directory, and that holds for real device types, not just a synthetic one:

$ sudo nmcli con add type wifi con-name pm-grill-wifi ifname wlan0 \
    ssid PM-GRILL-TEST autoconnect no wifi-sec.key-mgmt wpa-psk wifi-sec.psk ...
Connection 'pm-grill-wifi' (82da931e-...) successfully added.

$ sudo nmcli con add type ethernet con-name pm-grill-eth ifname eth0 \
    autoconnect no ipv4.method manual ipv4.addresses 10.98.98.2/24
Connection 'pm-grill-eth' (ce3c1462-...) successfully added.

$ sudo ls -1 /etc/NetworkManager/system-connections/
pm-grill-eth.nmconnection
pm-grill-wifi.nmconnection
$ sudo ls -1 /etc/netplan/
90-NM-75a1216a-9d1a-30cd-8aca-ace5526ec021.yaml
90-NM-be1282f3-d98b-3db6-9c1f-0cd80398f4f5.yaml

A WiFi connection and an ethernet connection — the same two types the netplan-backed profiles use — both went to the keyfile directory. The netplan files were untouched.

I am deliberately not going to tell you which tool put the wlan0 profile into netplan, because I could not establish it and I would rather say so. raspi-config sets WiFi by shelling out to nmcli — there is not one reference to netplan in it — and nmtui is part of network-manager and goes through the same library. Both of those should therefore land in the keyfile directory, exactly as my test connections did. They did not, for this machine's WiFi profile.

Take that as the practical finding rather than a loose end: a Pi that has been configured over time can have some of its network state in each store, and which is which is not something you can reason out from the tool you remember using. You have to look. Backing up either directory alone silently captures part of the picture, and the part it misses is not predictable.

Modifying an existing netplan-backed connection, on the other hand, edits the YAML in place rather than migrating it to a keyfile. Setting a static IP on the (unused, disconnected) eth0 profile:

$ sudo nmcli con modify netplan-eth0 ipv4.method manual \
    ipv4.addresses 10.99.99.5/24 ipv4.gateway 10.99.99.1
$ echo $?
0

rewrites the ethernets: block in 90-NM-75a1216a-....yaml and leaves system-connections/ empty. Worth stating plainly, because it is the thing people expect to be broken and it isn't: nmcli works. Setting a static IP on Trixie behaves exactly as documented, and nmcli -g ipv4.addresses con show reads the value back. The failure here is not in configuring the network. It is in finding the configuration afterwards.

The received answer is the wrong one

The standard answer to "where does Raspberry Pi OS keep my WiFi password" has been /etc/NetworkManager/system-connections/ since Bookworm moved off dhcpcd in 2023, and write-ups as recent as July 2026 still give that answer without qualification. On this image it is wrong, and nothing on the machine contradicts it loudly enough to notice — the directory exists, the plugin that owns it is enabled, and .nmconnection files really do exist a few paths away.

The fix

The obvious move is to stop guessing at paths and ask NetworkManager where the connection lives. It has a field for exactly that, and on this machine it does not help:

$ nmcli -f NAME,FILENAME con show
NAME                    FILENAME
netplan-wlan0-HOMEWIFI  /run/NetworkManager/system-connections/netplan-wlan0-HOMEWIFI.nmconnection
lo                      /run/NetworkManager/system-connections/lo.nmconnection
netplan-eth0            /run/NetworkManager/system-connections/netplan-eth0.nmconnection

FILENAME reports the file NetworkManager actually loaded the profile from, which is the truth — and for a netplan-sourced connection the truth is the tmpfs copy. Back up what that column names and you have backed up the derived artifact that disappears at the next boot.

What makes this genuinely treacherous rather than merely wrong is that the column is correct for connections you created yourself. Add one with nmcli con add and it reports the real, persistent path:

pm-grill-wifi  /etc/NetworkManager/system-connections/pm-grill-wifi.nmconnection

So FILENAME is reliable exactly for the connections you already know where to find, and misleading for the one that came with the image and holds your WiFi password.

There is no single field that names the persistent source. What works is mapping by UUID, which is stable across both stores:

nmcli -t -f NAME,UUID con show | while IFS=: read -r name uuid; do
  src=$(grep -rl "$uuid" /etc/netplan/ 2>/dev/null | head -1)
  [ -z "$src" ] && src=$(grep -rl "$uuid" /etc/NetworkManager/system-connections/ 2>/dev/null | head -1)
  printf '%-26s %s\n' "$name" "${src:-<runtime-generated, no persistent source>}"
done

On this box:

netplan-wlan0-HOMEWIFI     /etc/netplan/90-NM-be1282f3-d98b-3db6-9c1f-0cd80398f4f5.yaml
lo                         <runtime-generated, no persistent source>
netplan-eth0               /etc/netplan/90-NM-75a1216a-9d1a-30cd-8aca-ace5526ec021.yaml

It finds keyfile-backed connections too — adding one with nmcli con add makes it show up under /etc/NetworkManager/system-connections/ on the next run — and it correctly reports lo as having no persistent source, because it doesn't.

Run that before you reimage rather than after. To capture everything regardless of origin:

sudo tar czf wifi-backup.tar.gz \
  /etc/netplan \
  /etc/NetworkManager/system-connections

Then verify the archive contains files and not just directories — the whole point of this post is that the command succeeding proves nothing:

tar tzvf wifi-backup.tar.gz | grep -v '/$'

If that prints nothing, you have backed up nothing.

Size is not the signal either, and on this machine it points the wrong way. The correct archive — both netplan YAMLs, PSK included, gzipped — is 566 bytes. The one that captured nothing is 10,240 bytes, because tar pads to a 10 KB block whether or not it found anything to put in it. The bigger file is the empty one.

Two cautions on the archive itself. It contains your PSK in plaintext, so it inherits the 600 that the source files carry for a reason — don't leave it in /tmp or commit it anywhere. And when restoring onto a new image, put each file back where it came from: a netplan YAML dropped into system-connections/ is not read, and vice versa.

The generalisable habit

The failure here is a specific instance of a general one: an operation that has nothing to do can't tell you apart from an operation that succeeded. tar was asked to archive a directory. It did. Zero files matched, and zero matched is indistinguishable from "there was nothing that needed archiving" — the same shape as a clone tool whose search-and-replace matches nothing and reports success, and the same shape as a journal directory that exists to reassure you persistence is on when it isn't.

So the narrow habit, which costs one command: after any backup, list the archive. Not the exit code, not the file size — the contents. Both of those were fine here.

And the wider one, which is the part I got wrong on the first pass: asking the tool is better than asking the internet, but "the tool told me" is not the same as "this is the file I need." nmcli's FILENAME is accurate — it answers "where did I load this from?" precisely. My question was "what do I have to copy so this survives a reimage?", and those turn out to be different files. Every layer here answers its own question correctly. The gap is between the question the tool answers and the one you meant, and nothing in the output marks where that gap is.

That is the same failure as trusting tar's exit code, one level up. Both times the machine was right and the inference was wrong.

A ready-to-run version of this check — which finds every connection's real backing file across both stores, warns when /etc/NetworkManager/system-connections/ is empty, and verifies that a backup archive actually contains files — is in the toolkit that comes with this post.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Podman publishes the same port and your firewall still holds. Even as root.

The Homelab Postmortem — Mon, 24 Aug 2026 15:28:56 +0000

TL;DR: On the same box, same port, same ufw config where Docker published straight past the firewall, Podman does not — the port times out from another machine exactly as ufw status claims. Rootless Podman writes no firewall rules at all. But rootful Podman does write them, and the port is still blocked, so "it has no root" is not the explanation. Docker inserts an ACCEPT into FORWARD ahead of ufw's chains; netavark keeps its rules in its own table and never overrides ufw's policy drop. Also: Podman never needed iptables installed at any point.

Where this came from

The Docker writeup got a comment on r/selfhosted saying, roughly: this is why I use rootless Podman — it can't modify the system firewall if it doesn't have root.

That is a satisfying explanation, and I wanted it to be true, which is a good reason to go and check it rather than repeat it. It turns out to be right about the outcome and wrong about the mechanism, and the wrong part is the more interesting half.

Setup

Identical to the Docker test, deliberately. Raspberry Pi OS Lite (Trixie) on a Pi 4B, ufw active, default deny incoming, one rule for SSH:

$ sudo ufw status verbose
Status: active
Default: deny (incoming), allow (outgoing), disabled (routed)

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere

Baseline, before any container runtime: a plain listener on 8080 is blocked from another machine on the LAN.

$ python3 -m http.server 8080 --bind 0.0.0.0
# from the laptop:
192.168.128.154:8080 → Connection timed out

Then apt install podman — 26 packages, none of them iptables. It pulls netavark (Podman's network backend), aardvark-dns, and passt/slirp4netns for rootless networking.

Rootless: no firewall rules at all

$ podman run -d --name pmtest -p 8080:80 docker.io/library/nginx:alpine
$ podman ps --format '{{.Ports}}'
0.0.0.0:8080->80/tcp

That is byte-identical to what Docker printed. Bound to all interfaces, same port, same everything. Locally it answers:

localhost:8080 → HTTP 200

From another machine:

192.168.128.154:8080 → timed out

And the firewall is untouched:

$ sudo nft list ruleset | wc -l
387                      # unchanged from before the container started
$ sudo nft list ruleset | grep -c 8080
0
$ sudo nft list tables
table ip filter
table ip6 filter         # ufw's. No nat table, no netavark table.

Nothing. No DNAT, no ACCEPT, no new table. The commenter's description holds exactly here: rootless has no way to write these rules, so it doesn't, and ufw decides — correctly.

Rootful: rules appear, and the port is still blocked

This is the part that decides whether "no root" is really the explanation.

$ sudo podman run -d --name pmroot -p 8080:80 docker.io/library/nginx:alpine
$ sudo podman ps --format '{{.Ports}}'
0.0.0.0:8080->80/tcp

Now the firewall does change:

$ sudo nft list ruleset | wc -l
447                      # was 387
$ sudo nft list ruleset | grep -c 8080
4
$ sudo nft list tables
table ip filter
table ip6 filter
table inet netavark      # new

Rules written, DNAT in place. By the Docker precedent this is the moment the port becomes reachable. It doesn't:

192.168.128.154:8080 → timed out

So the privilege level is not what saved us. Something about which rules get written is.

What netavark actually writes

table inet netavark {
    chain FORWARD {
        type filter hook forward priority filter; policy accept;
        ct state invalid drop
        jump NETAVARK-ISOLATION-1
        ip daddr 10.88.0.0/16 ct state established,related accept
        ip saddr 10.88.0.0/16 accept
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        fib daddr type local jump NETAVARK-HOSTPORT-DNAT
    }

    chain NETAVARK-HOSTPORT-DNAT {
        tcp dport 8080 jump nv_2f259bab_10_88_0_0_nm16_dnat
    }
}

The DNAT is there and it works — more on proving that below. What matters is the FORWARD chain. netavark's accepts are scoped to its own container network, 10.88.0.0/16. It has policy accept, but it only accepts traffic that belongs to it.

ufw's FORWARD chain, in a separate table, is not so relaxed:

chain FORWARD {
    type filter hook forward priority filter; policy drop;
    counter packets 19 bytes 2309 jump ufw-before-logging-forward
    counter packets 19 bytes 2309 jump ufw-before-forward
    counter packets 19 bytes 2309 jump ufw-after-forward
}

policy drop, and a counter that is climbing — those 19 packets are the connection attempts from my laptop being dropped.

Both chains hang off the same forward hook at the same priority. In nftables that means both run, and a drop anywhere is final. netavark never tries to be first, never inserts anything into ufw's chains, and never short-circuits them. It manages its own traffic and leaves the host's policy alone.

Docker's -A FORWARD -j DOCKER with an ACCEPT inside is the outlier here, not the norm.

Proving the DNAT genuinely works

"It's blocked" could just mean the port forwarding never worked. It isn't that. Turn ufw off with the same rootful container still running:

$ sudo ufw disable
192.168.128.154:8080 → HTTP 200
$ sudo ufw enable
192.168.128.154:8080 → timed out

The redirect is fine. ufw is doing the blocking, which is the entire point — the firewall is back to being the thing that decides.

One more thing: iptables never appeared

Worth noting because it changes what a Podman box looks like. On the Docker test, iptables arrived as a dependency of ufw and Docker used it. Here, after installing Podman and running containers both rootless and rootful:

$ command -v iptables
$                        # nothing

netavark talks to nftables directly. There is no legacy-vs-nft backend question to get wrong, because there is no iptables layer at all.

What I'd take from this

The comment was right and worth acting on, and the reason it gave was wrong. If I'd repeated it without checking, I'd have published "rootless protects you because it can't write rules" — true as far as it goes, and it would have left people thinking rootful Podman is as dangerous as Docker. It isn't.

That distinction is practical. Plenty of people run rootful Podman because they need low ports, or systemd integration, or just inherited it that way. They get the firewall behaviour too.

The narrower lesson is about what "publishes a port" means. Docker, rootless Podman and rootful Podman all print 0.0.0.0:8080->80/tcp. Three different security outcomes behind one identical string. The output tells you the intent, not the result — same as ufw status telling you its rules rather than your exposure.

Which loops back to the habit from the Docker post, now with a second data point behind it: curl the port from another machine. It is the only check that has been right about every one of these configurations.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

UFW says the port is closed. Docker published it to your whole network anyway.

The Homelab Postmortem — Sat, 22 Aug 2026 01:59:07 +0000

TL;DR: ufw enable on a Pi does what it says — until Docker is installed. Docker inserts its own chains ahead of UFW's in FORWARD, and DNATs published ports before UFW is ever consulted. Same port, same firewall, same deny (incoming) default: a host process is blocked and a container is reachable from the entire LAN. ufw status reports the port as not allowed in both cases. Bind to 127.0.0.1 explicitly, or put a rule in DOCKER-USER — both verified below.

The demonstration

One Pi 4B, Raspberry Pi OS Lite (Trixie), ufw active with a single rule for SSH:

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)

Port 8080 is not in that list, so it is denied. Start a plain listener on it:

$ python3 -m http.server 8080 --bind 0.0.0.0

From another machine on the same LAN:

192.168.128.154:8080 → Connection timed out after 6010 milliseconds

Blocked, exactly as configured. Now stop that, and publish the same port from a container instead:

$ sudo docker run -d --name ufwtest -p 8080:80 nginx:alpine
$ sudo docker ps --format '{{.Ports}}'
0.0.0.0:8080->80/tcp, :::8080->80/tcp

ufw status is unchanged — still no rule for 8080, still deny (incoming). From the same other machine:

192.168.128.154:8080 → HTTP 200

Same port. Same firewall. Same policy. Opposite outcome, and the firewall's own status output cannot tell the two situations apart.

Why it happens

Docker writes iptables rules when it starts, and it puts them at the front of the chains that matter:

$ sudo iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j ufw-before-logging-forward
...

DOCKER-USER and DOCKER come first. ufw-before-logging-forward — the entry point to everything UFW manages — comes after them. Traffic that Docker accepts never reaches a UFW rule.

The actual redirection happens earlier still, in nat:

$ sudo iptables -t nat -S DOCKER
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80

The packet is DNAT'd to the container before filtering decides anything. By the time UFW's rules are consulted, the destination is no longer the host.

None of this is a bug in the strict sense. Docker documents that it manipulates iptables, and publishing a port is a request to make it reachable. The problem is that the tool you use to check whether a port is exposed does not model the mechanism that exposed it. ufw status reports UFW's rules, correctly, and those rules genuinely do not allow 8080.

What a stock Pi actually starts with

Worth knowing, because it changes who this applies to. On a fresh Raspberry Pi OS Lite (Trixie) install, iptables is not present at all:

$ ls /usr/sbin/iptables /sbin/iptables
ls: cannot access '/usr/sbin/iptables': No such file or directory
ls: cannot access '/sbin/iptables': No such file or directory

$ dpkg-query -W -f='${Status}' iptables
unknown ok not-installed

$ update-alternatives --list iptables
update-alternatives: error: no alternatives for iptables

nftables is installed but its service is disabled and the ruleset is empty. So the box ships with no firewall running and no iptables binary.

iptables arrives as a dependency of whatever you install first. apt install ufw pulls it in, and the alternative resolves to the nft backend:

$ update-alternatives --display iptables
iptables - auto mode
  link best version is /usr/sbin/iptables-nft
  link currently points to /usr/sbin/iptables-nft

$ sudo iptables --version
iptables v1.8.11 (nf_tables)

Installing Docker afterwards did not change that — the alternative stayed on iptables-nft, and Debian's docker.io package does not depend on iptables-legacy. Advice that says Docker forces the legacy backend did not hold here. It may have been true on earlier releases; on this image it wasn't, and the UFW-bypass problem happens regardless of which backend is selected.

The fix

Two things work. Both were tested against the same container and the same firewall state.

Bind the published port to localhost when you don't want it on the network. The -p flag takes an address, and most examples omit it, which means 0.0.0.0:

$ sudo docker run -d --name fixA -p 127.0.0.1:8080:80 nginx:alpine
$ sudo docker ps --format '{{.Ports}}'
127.0.0.1:8080->80/tcp

localhost:8080            → HTTP 200
192.168.128.154:8080      → timed out

The DNAT rule still exists; it's just scoped to loopback. This is the right default for anything behind a reverse proxy, which on a homelab box is most things.

Or filter in DOCKER-USER, which Docker evaluates first and never rewrites. That chain exists specifically so you have somewhere to put rules that survive Docker restarts:

sudo iptables -I DOCKER-USER 1 -p tcp --dport 80 -m conntrack --ctstate NEW ! -s 127.0.0.1 -j DROP

$ sudo iptables -S DOCKER-USER
-N DOCKER-USER
-A DOCKER-USER ! -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -j DROP
-A DOCKER-USER -j RETURN

192.168.128.154:8080      → timed out

Note the port in that rule is the container's port (80), not the published one (8080) — DOCKER-USER is in FORWARD, so it sees the packet after DNAT has already rewritten the destination. Writing --dport 8080 there matches nothing and silently does not protect you, which is its own small trap.

Rules added this way are not persistent across reboots on their own; iptables-persistent, a systemd unit, or a tool like ufw-docker handles that. Whichever route you take, verify from another machine rather than from the Pi — curl localhost will succeed in every configuration above and tells you nothing.

The generalisable habit

This is the same shape as the swap file that isn't swap and the memory limit that isn't enforced: a tool reports on its own model of the world, and something outside that model is what actually decides.

ufw status is not lying. It is answering "what rules does UFW have?" precisely. The question you had was "what can reach this machine?", and no single tool on the box answers that once two things are both writing firewall rules.

So the habit worth building is narrow and cheap: test exposure from a different machine. One curl from a laptop settles in three seconds what an hour of reading rule listings will not, because it exercises the whole stack instead of one layer's opinion of it.

And when you install something that manages its own firewall rules — Docker, Tailscale, libvirt, k3s — assume it did, and go look at the chain order once. The ordering is the entire behaviour, and it is visible in one command.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

A new NetworkManager property broke a decade of headless WiFi scripts

The Homelab Postmortem — Wed, 19 Aug 2026 03:27:08 +0000

TL;DR: NetworkManager 1.52 (in Trixie) added an autoconnect-ports property. That makes autoconnect-p — an abbreviation that worked for years — ambiguous against autoconnect-priority, and nmcli rejects it with exit 2. nmcli connection modify is atomic, so every other property in that same command is discarded too, including ones listed before it. If your provisioning script created the profile in an earlier command, you're left with a WiFi profile that exists, shows the right SSID, and has no key-mgmt and no psk. It will never connect, and nothing about the profile looks broken.

The symptom

A headless provisioning script that has worked for years stops working against a Trixie image. The Pi comes up, the WiFi profile is there, and it never associates.

Listing the profile shows nothing obviously wrong:

$ nmcli connection show pm-test | grep -E '^(connection.id|connection.type|802-11-wireless.ssid)'
connection.id:                          pm-test
connection.type:                        802-11-wireless
802-11-wireless.ssid:                   PM-TEST-SSID

Right name, right type, right SSID. But:

$ nmcli -g 802-11-wireless-security.key-mgmt connection show pm-test
$ nmcli -g 802-11-wireless-security.psk connection show pm-test
$

Both empty. The profile has no security configuration at all — no key management, no pre-shared key. On a WPA2 network that profile cannot work, and the profile itself gives no hint that it's incomplete unless you go looking for the specific fields that are missing.

What's really going on

nmcli accepts abbreviated property names as long as they're unambiguous. On this machine:

$ nmcli --version
nmcli tool, version 1.52.1

1.52 introduced autoconnect-ports. There was already autoconnect-priority. Both exist in nm-settings-nmcli(5) now, and both start with autoconnect-p:

$ nmcli connection modify pm-test conn.autoconnect-p 10
Error: invalid property 'autoconnect-p': 'autoconnect-p' is ambiguous: autoconnect-priority, autoconnect-ports.
$ echo $?
2

So the abbreviation that scripts have used since long before either property collided now resolves to nothing. That part is loud — it's an error on stderr with a non-zero exit.

The damaging part is what it does to the rest of the command.

The command is all-or-nothing, in both directions

The obvious assumption is that nmcli processes properties left to right and stops when it hits a bad one, so anything before the failure got applied. It doesn't work that way. Putting a perfectly valid property first:

$ nmcli connection modify pm-test wifi-sec.key-mgmt wpa-psk conn.autoconnect-p 10
Error: invalid property 'autoconnect-p': 'autoconnect-p' is ambiguous: autoconnect-priority, autoconnect-ports.
$ echo $?
2
$ nmcli -g 802-11-wireless-security.key-mgmt connection show pm-test
$

key-mgmt was listed before the ambiguous property and was still not applied. nmcli connection modify validates the whole argument list before committing anything, so one bad name discards all of it.

That's a reasonable design — a half-applied network config would be worse. But combined with the usual scripting pattern, it produces the failure above. Headless WiFi setup scripts overwhelmingly look like:

nmcli connection add type wifi con-name "$NAME" ifname wlan0 ssid "$SSID"
nmcli connection modify "$NAME" \
  conn.autoconnect-p 10 \
  wifi-sec.key-mgmt wpa-psk \
  wifi-sec.psk "$PASSWORD"

The add succeeds — that's a separate command and it doesn't mention autoconnect-p. The modify fails in its entirety. The result is a profile that exists with an SSID and no credentials, which is exactly the state above.

If the script checks exit codes it fails loudly, and you find this in a minute. Plenty of provisioning scripts don't — they run under set +e, or pipe output away, or run each step under a supervisor that only cares that the script finished. Those report success and hand you a Pi that won't join the network.

Why it's easy to misattribute

Everything about the symptom points somewhere else. The machine is headless, so the first evidence you get is "it didn't come up on the network," and the natural suspects are the password, the SSID, 2.4 vs 5 GHz, the regulatory domain, the router. All of those are more common than a CLI abbreviation collision.

The profile actively supports the wrong theory. It exists. It has the SSID you expect. nmcli connection show on it prints dozens of populated fields. Two empty ones in the middle of that output do not stand out, and if you don't already know that key-mgmt and psk are the two that matter, there is nothing to draw your eye to them.

And the change is invisible from the Pi side of things. This isn't in any Raspberry Pi changelog — it's an upstream NetworkManager property addition. The behaviour of abbreviations is documented (they must be unambiguous), but no documentation says "this specific abbreviation stopped working," because from NetworkManager's perspective nothing broke: a new property was added, and abbreviation resolution worked as specified.

The fix

Use full property names in anything non-interactive:

$ nmcli connection modify pm-test \
    connection.autoconnect-priority 10 \
    wifi-sec.key-mgmt wpa-psk \
    wifi-sec.psk testpassword123
$ echo $?
0
$ nmcli -f connection.autoconnect-priority,802-11-wireless-security.key-mgmt connection show pm-test
connection.autoconnect-priority:        10
802-11-wireless-security.key-mgmt:      wpa-psk

Abbreviations aren't broken in general — only ones that became ambiguous. A longer abbreviation still resolves:

$ nmcli connection modify pm-test conn.autoconnect-pri 20
$ echo $?
0
$ nmcli -g connection.autoconnect-priority connection show pm-test
20

autoconnect-pri is unique again, so it works. But that's the trap in miniature: autoconnect-pri is fine today, and it is fine for exactly as long as nobody adds autoconnect-primary. Every abbreviation in a committed script is a bet on a namespace you don't control.

The section prefixes (conn., wifi-sec.) are a different thing and are fine — those are documented setting-name aliases, not prefix matches.

If you maintain provisioning scripts, this greps them:

grep -rn "nmcli.*\b\(conn\|connection\)\.autoconnect-p\b" .

And after any scripted profile creation, verify the fields that actually matter rather than trusting the exit code alone:

nmcli -g 802-11-wireless-security.key-mgmt,802-11-wireless-security.psk \
  connection show "$NAME"

Empty output there means the profile will never associate, regardless of what the script reported.

The generalisable habit

Abbreviations are a convenience for humans at a prompt. They're a liability in a file, because the thing that makes an abbreviation valid isn't your input — it's the set of every other name that exists, and that set grows without your involvement. A script using autoconnect-p didn't change. Its meaning did, when someone upstream added a property that happened to share a prefix.

The narrower habit worth taking from this: when a tool validates a batch, find out whether it's atomic before you rely on ordering. The intuition that "the stuff before the error went through" is wrong here, and it's wrong in the safer direction — but if it had been right, this same bug would have produced a profile with a password and no key management, which is a stranger thing to debug than a profile with neither.

And the one that generalises furthest: a resource that exists is not a resource that is configured. The provisioning script's job wasn't to create a profile, it was to create a working connection. Checking for the former and reporting success is how you end up with a headless box on the bench and no idea why it's silent.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

sysctl -p says it worked. The next reboot disagrees.

The Homelab Postmortem — Wed, 19 Aug 2026 03:15:10 +0000

TL;DR: /etc/sysctl.conf doesn't ship on Raspberry Pi OS Trixie, and if you create it, the thing that runs at boot — systemd-sysctl — never reads it. The sysctl CLI from procps still does, so sudo sysctl -p applies your setting and prints it back at you like it worked. It did work, until the next reboot. Verified both directions on hardware: identical content in /etc/sysctl.conf and /etc/sysctl.d/99-local.conf, one ignored at boot, one applied. systemd-sysctl.service exits 0/SUCCESS either way.

The symptom

You want a kernel parameter to stick. Every guide written in the last fifteen years says the same thing: put it in /etc/sysctl.conf, run sysctl -p. So:

$ sudo sh -c 'printf "vm.swappiness = 42\n" > /etc/sysctl.conf'
$ sudo sysctl -p
vm.swappiness = 42
$ cat /proc/sys/vm/swappiness
42

The tool echoes the setting back. The kernel confirms it. This is as clear a success signal as Linux gives you.

Reboot, and the file is still sitting there, unchanged:

$ cat /etc/sysctl.conf
vm.swappiness = 42
$ cat /proc/sys/vm/swappiness
60

Back to the default. Nothing removed the file. Nothing rewrote it. It was simply never read.

Why it's easy to misdiagnose

The first thing you'd suspect is that something overrode you later in boot, so you'd go looking for a conflicting drop-in and find nothing relevant:

$ ls /etc/sysctl.d/
98-rpi.conf  README.sysctl
$ cat /etc/sysctl.d/98-rpi.conf
kernel.printk = 3 4 1 3
vm.min_free_kbytes = 16384
net.ipv4.ping_group_range = 0 2147483647

No vm.swappiness anywhere. Nothing competing with your setting.

The second thing you'd check is whether the service that applies sysctls failed. It didn't:

$ systemctl status systemd-sysctl.service
● systemd-sysctl.service - Apply Kernel Variables
     Loaded: loaded (/usr/lib/systemd/system/systemd-sysctl.service; static)
     Active: active (exited) since Wed 2026-08-19 12:11:49 JST
   Main PID: 323 (code=exited, status=0/SUCCESS)

$ journalctl -b -u systemd-sysctl.service
Aug 19 12:11:49 mitoyosh-pi4b systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.

Exit status 0. One log line, and it says "Finished". There is no warning that a file was skipped, because from systemd-sysctl's point of view no file was skipped — it read every file it knows about, and /etc/sysctl.conf is not one of them.

There's a third clue that isn't there either: on a stock Trixie image, /etc/sysctl.conf doesn't exist in the first place.

$ ls -la /etc/sysctl.conf
ls: cannot access '/etc/sysctl.conf': No such file or directory

procps ships /etc/sysctl.d/ and a README.sysctl, but not the file itself. So you don't edit an existing file with existing content — you create a new one, from scratch, at a path the boot process ignores. Nothing about that act feels like it needs verifying.

What's really going on

Two different programs are called "sysctl" here, and they disagree about which files matter.

At boot, systemd runs its own implementation:

$ systemctl cat systemd-sysctl.service | grep ExecStart
ExecStart=/usr/lib/systemd/systemd-sysctl

That binary contains exactly four config paths:

$ strings /usr/lib/systemd/systemd-sysctl | grep -E '^/(etc|run|usr)/.*sysctl'
/etc/sysctl.d
/run/sysctl.d
/usr/lib/sysctl.d
/usr/local/lib/sysctl.d

Four directories. /etc/sysctl.conf is not among them, and sysctl.d(5)'s SYNOPSIS lists the same four. The boot-time path has no concept of that file.

From your shell, sysctl is a different program entirely — /usr/sbin/sysctl, from procps. Its --system precedence list does include the file, last:

/etc/sysctl.d/*.conf
/run/sysctl.d/*.conf
/usr/local/lib/sysctl.d/*.conf
/usr/lib/sysctl.d/*.conf
/lib/sysctl.d/*.conf
/etc/sysctl.conf

And plain sysctl -p with no argument defaults to reading /etc/sysctl.conf specifically. So the interactive tool honours the file, the boot process doesn't, and the gap between them is exactly one reboot wide.

On many Debian systems this is papered over by a compatibility symlink at /etc/sysctl.d/99-sysctl.conf pointing back to /etc/sysctl.conf, which drags the file into a directory systemd does read. This image doesn't have one:

$ ls -la /etc/sysctl.d/99-sysctl.conf
ls: cannot access '/etc/sysctl.d/99-sysctl.conf': No such file or directory

Worth checking on your own machine before assuming either way — its presence or absence is the whole difference.

The demonstration

Run the boot-time binary by hand and you can watch it ignore the file, without waiting for a reboot. Start from the file in place and the value at its default:

$ cat /etc/sysctl.conf
vm.swappiness = 42
$ cat /proc/sys/vm/swappiness
60
$ sudo /usr/lib/systemd/systemd-sysctl
$ cat /proc/sys/vm/swappiness
60

No output, no error, no change. Now put byte-identical content in the directory it does read:

$ sudo sh -c 'printf "vm.swappiness = 42\n" > /etc/sysctl.d/99-local-test.conf'
$ sudo /usr/lib/systemd/systemd-sysctl
$ cat /proc/sys/vm/swappiness
42

Same setting, same syntax, same nineteen bytes. The only variable is which path it sits at.

And the end-to-end version — drop-in removed, /etc/sysctl.conf left in place, real reboot:

$ ls /etc/sysctl.d/
98-rpi.conf  README.sysctl
$ cat /etc/sysctl.conf
vm.swappiness = 42
$ cat /proc/sys/vm/swappiness
60

The fix

Put it in /etc/sysctl.d/ and give it a numeric prefix:

sudo sh -c 'printf "vm.swappiness = 42\n" > /etc/sysctl.d/99-local.conf'
sudo /usr/lib/systemd/systemd-sysctl        # apply now, same as boot will
cat /proc/sys/vm/swappiness                 # verify it took

Files across all the sysctl directories are sorted lexicographically by filename regardless of which directory they're in, and the last one to set a given key wins. 99- puts you after the vendor's 98-rpi.conf, which is what you want if you ever need to override one of its three settings.

Two things worth doing differently from the old habit:

Apply with systemd-sysctl, not sysctl -p. Not because sysctl -p is broken, but because running the boot-time binary tests the boot-time path. If it applies now, it will apply at boot. sysctl -p proves less than it appears to.
Verify by reading /proc/sys/..., not by trusting the command's output. sysctl -p echoing your setting back means it parsed your file, not that the value survives.

If you have an existing /etc/sysctl.conf from an older install or a config-management tool, the minimal migration is to make the file visible to systemd rather than move its contents:

sudo ln -s /etc/sysctl.conf /etc/sysctl.d/99-sysctl.conf

That's the same compatibility symlink other Debian systems ship, and it's the smallest change that makes both tools agree.

The generalisable habit

The trap here isn't the missing file. It's that the tool you use to apply a config and the code that applies it at boot were two different programs with two different opinions, and only one of them ever spoke to you.

sysctl -p isn't lying. It read the file you gave it and set the value, and it said so. It just has no authority over what happens at boot, and no reason to mention that. The success message is scoped to the current kernel, and you read it as scoped to the machine.

So the reflex: when you make a setting persistent, verify persistence specifically, not application. Those are different claims. The cheapest version is to run whatever the boot path actually runs — here systemd-sysctl, one command, no reboot — and confirm the value moves. If you can't identify what runs at boot, that's worth five minutes with systemctl cat, because it also tells you which files it reads, which is the question you actually had.

This is the third instance of the same shape on this platform in a week, after journald keeping logs in RAM and swap config needing a reboot. Correct config, no error, no effect. The common thread is a config file whose reader isn't the program you were talking to.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Your Pi accepts every memory limit you set and enforces none of them

The Homelab Postmortem — Wed, 19 Aug 2026 02:57:41 +0000

TL;DR: On stock Raspberry Pi OS the memory cgroup controller is disabled, so docker run --memory=512m, systemd's MemoryMax=, and k3s pod limits are all accepted without complaint and enforced not at all — the kernel has nowhere to write the limit. The disable comes from cgroup_disable=memory, which the firmware injects ahead of your cmdline.txt and which never appears in that file. Adding cgroup_enable=memory cgroup_memory=1 fixes it, costs no measurable RAM, and needs a reboot. And /proc/cgroups — the file a lot of guides tell you to check — prints byte-identical output before and after the fix.

The symptom

Nothing errors. That's the entire problem.

On a Pi 4B running Trixie (6.18.34+rpt-rpi-v8), ask systemd to cap a process at 100 MB:

$ systemd-run --user --scope -p MemoryMax=100M sleep 30
Running as unit: run-p1753-i1754.scope; invocation ID: f8ecc5cb8b884fd68fd9eb2cccbdb72d

No warning. Ask systemd whether the limit is in place, and it says yes:

$ systemctl --user show run-p1753-i1754.scope -p MemoryMax -p MemoryAccounting
MemoryAccounting=yes
MemoryMax=104857600

104857600 bytes. Exactly the 100 MB requested. Except the limit does not exist, because there is nowhere to put it:

$ ls /sys/fs/cgroup/user.slice/user-1000.slice/session-22.scope/
cgroup.controllers   cgroup.max.depth        cgroup.stat.local
cgroup.events        cgroup.max.descendants  cgroup.subtree_control
cgroup.freeze        cgroup.procs            cgroup.threads
cgroup.kill          cgroup.stat             cgroup.type

There is no memory.max file in that directory. There is no memory. anything. systemd reported a limit it had no way to apply, and reported it as successfully set.

The same applies to docker run --memory=512m and to k3s pod memory limits. This is not a Docker bug — it's one layer below Docker, which is why it hits every container runtime on the platform.

What's really going on

The memory controller isn't in the active set:

$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io pids

cpuset, cpu, io, pids — no memory. So the natural next step is to look at the kernel command line, and this is where it gets interesting:

$ cat /boot/firmware/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=b11c96d4-02 rootfstype=ext4 fsck.repair=yes rootwait ds=nocloud;i=rpi-imager-1786770245730 cfg80211.ieee80211_regdom=JP

Nothing about cgroups at all. No cgroup_disable, no cgroup_enable. If cmdline.txt is your model of the kernel command line, there is nothing here to explain the missing controller.

cmdline.txt is not the kernel command line. It's a fragment the firmware appends to its own. Here is what the kernel actually received:

$ cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_headphones=0 cgroup_disable=memory numa_policy=interleave nvme.max_host_mem_size_mb=32 ... console=ttyS0,115200 console=tty1 root=PARTUUID=b11c96d4-02 ... cfg80211.ieee80211_regdom=JP

cgroup_disable=memory, fourth parameter in, put there by the firmware. It is not in any file on the boot partition. You cannot grep for it in /boot/firmware/ and find it. The only place it is visible is /proc/cmdline, after boot.

This is a deliberate platform default, not an accident — the memory controller has a small per-page overhead, and Pi images have historically optimised for the machine having as much usable RAM as possible. The request to flip it for 64-bit Lite builds was filed as RPi-Distro/pi-gen#917 and closed not planned in April 2026. It isn't going to change upstream, so every Pi that runs containers has to fix it locally.

The check that lies to you

A lot of guides tell you to inspect /proc/cgroups. On this machine, before the fix:

#subsys_name    hierarchy   num_cgroups enabled
cpuset  0   44  1
cpu 0   44  1
cpuacct 0   44  1
blkio   0   44  1
devices 0   44  1
freezer 0   44  1
net_cls 0   44  1
perf_event  0   44  1
net_prio    0   44  1
pids    0   44  1

No memory row — consistent with the problem, so far so good. After the fix, rebooted, with the controller confirmed working:

#subsys_name    hierarchy   num_cgroups enabled
cpuset  0   75  1
cpu 0   75  1
cpuacct 0   75  1
blkio   0   75  1
devices 0   75  1
freezer 0   75  1
net_cls 0   75  1
perf_event  0   75  1
net_prio    0   75  1
pids    0   75  1

Still no memory row. Same ten subsystems, same order. The only thing that changed is a counter.

/proc/cgroups reports cgroup v1 subsystems, and this system runs a pure v2 unified hierarchy:

$ stat -fc %T /sys/fs/cgroup
cgroup2fs
$ mount | grep ' /sys/fs/cgroup '
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

So /proc/cgroups is answering a question about a hierarchy this machine doesn't use. It is not wrong, it's irrelevant — and it is irrelevant in a way that looks exactly like a failed fix. Check /sys/fs/cgroup/cgroup.controllers instead. That one changes.

The fix

Append two parameters to cmdline.txt. It must stay a single line — a stray newline here is a machine that doesn't boot, so back it up first and read it back before rebooting:

sudo cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.bak-$(date +%Y%m%d%H%M%S)
sudo sed -i 's/$/ cgroup_enable=memory cgroup_memory=1/' /boot/firmware/cmdline.txt
cat /boot/firmware/cmdline.txt          # one line, params at the end
awk 'END{print NR}' /boot/firmware/cmdline.txt   # must print 1

While you're in there, confirm root=PARTUUID= still matches the disk you actually boot from — blkid -s PARTUUID -o value /dev/sda2 — because if that's ever been wrong you'll find out at the same reboot and misattribute it to this change.

Then reboot, and verify against the right file:

$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory pids

memory is now present. The end-to-end check is that a limit lands somewhere real:

$ systemd-run --user --scope -q -p MemoryMax=100M bash -c \
    'CG=$(cut -d: -f3 /proc/self/cgroup); echo memory.max=$(cat /sys/fs/cgroup$CG/memory.max)'
memory.max=104857600

Before the fix that file did not exist. Now it holds the number.

Note what the fix is actually doing. cgroup_disable=memory is still on the kernel command line — the firmware still injects it:

$ cat /proc/cmdline | tr ' ' '\n' | grep cgroup
cgroup_disable=memory
cgroup_enable=memory
cgroup_memory=1

You are not removing the disable. You are overriding it by landing later on the command line, which works precisely because cmdline.txt is appended after the firmware's own parameters. That ordering is the whole reason a two-word edit is sufficient.

The RAM cost people worry about did not materialise here — free -m reported a total of 3794 MB both before and after.

The generalisable habit

The reflex worth building: the config file you edit is not always the config the system reads. cmdline.txt reads like the kernel command line — it has the right shape, the right contents, the right name. It is a fragment. The authoritative version lived at /proc/cmdline the whole time, and one look at it turns an unexplained missing controller into an obvious cgroup_disable=memory.

The narrower lesson is about verification. Two different files here will tell you about cgroups, and only one of them is about the hierarchy your machine actually runs. Picking the wrong one gives you a confident, stable, identical answer before and after a fix that worked. When a check shows no change after a change you believe in, consider that the check may be measuring the wrong thing before concluding the fix failed — the previous journald and swap posts on this site are the same shape, and that's three for three on this platform.

And the one that costs real money in production: a limit that is accepted is not a limit that is enforced. If you rely on --memory to stop one container taking down a box, verify once that the enforcement path exists. systemd-run will tell you in a single command, without installing anything.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Your Pi's 2 GB swap file isn't swap, and no swap command will tell you that

The Homelab Postmortem — Wed, 19 Aug 2026 02:05:53 +0000

TL;DR: Raspberry Pi OS Trixie ships rpi-swap instead of dphys-swapfile, defaulting to a zram+file hybrid. /var/swap is still there and still costs 2 GB of real disk, but it is zram's writeback target, not a swap area — so it never appears in swapon, free, or /proc/swaps. On the reference Pi that file was 43% of all disk in use, and every swap tool reported the identical output with it present and with it gone. The only way to see the link is /sys/block/zram0/backing_dev. Config moved to /etc/rpi/swap.conf, and changes need a full reboot — systemctl daemon-reload applies nothing and reports no error.

The symptom

There isn't a crash here. That's what makes it worth writing down: the numbers just quietly stop adding up.

A stock Trixie install on a Pi 4B with 3.7 GB of RAM, booting from a USB SSD:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.7Gi       220Mi       2.8Gi        35Mi       802Mi       3.5Gi
Swap:          2.0Gi          0B       2.0Gi

2 GB of swap. Fine. And there's a swap file where you'd expect one:

$ ls -lh /var/swap
-rw------- 1 root root 2.0G Aug 16 23:01 /var/swap

Also 2 GB. So that's the swap, sitting in a file, exactly as it's worked for the last decade.

Except it isn't, and those two 2 GB figures are not the same 2 GB.

$ swapon --show
NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition   2G   0B  100

$ cat /proc/swaps
Filename        Type       Size      Used  Priority
/dev/zram0      partition  2097148   0     100

The only swap device on the machine is /dev/zram0 — compressed RAM. The 2 GB file does not appear in the swap tables at all. But it is fully allocated on disk, not sparse:

$ du -h --apparent-size /var/swap && du -h /var/swap
2.0G    /var/swap
2.1G    /var/swap

On this machine, that's 43% of everything on the disk:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       219G  4.7G  203G   3% /

4.7 GB used, 2.1 GB of which is a file that no swap command acknowledges. On a 32 GB SD card the same file is 6.3% of the entire card.

What's really going on

Trixie replaced dphys-swapfile with a new package. The old one is not installed, not deprecated-but-present — simply gone:

$ dpkg -l | grep -E 'dphys-swapfile|rpi-swap'
ii  rpi-swap  1.2.2  all  early-boot swap configuration

$ command -v dphys-swapfile
$ ls /etc/dphys-swapfile
ls: cannot access '/etc/dphys-swapfile': No such file or directory

rpi-swap defaults to a mechanism called zram+file, and swap.conf(5) is direct about what that means:

zram+file: A compressed RAM-based swap device is created, with the file specified in File::Path used for writeback storage. The file is not used as a traditional swap device; instead, zram occasionally writes idle pages to it to free up RAM, reducing SD card wear through infrequent writes.

So the file's job is to be somewhere zram can evict cold pages to. You can watch the chain yourself:

$ cat /sys/block/zram0/backing_dev
/dev/loop0

$ losetup -a
/dev/loop0: []: (/var/swap)

zram0 → loop0 → /var/swap. That's the entire relationship, and /sys is the only place it's visible. It is a real mechanism doing a sensible thing — batching writes to spare the SD card — and the writeback timer is genuinely running on a 24-hour cycle:

$ systemctl list-timers rpi-zram-writeback.timer
NEXT                        LEFT LAST                        PASSED UNIT
Wed 2026-08-19 23:01:49 JST  15h Tue 2026-08-18 23:01:49 JST 8h ago rpi-zram-writeback.timer

The 2 GB zram size is also not arbitrary. With RamMultiplier=1 on a 3794 MiB machine you'd expect ~3.7 GB, but MaxSizeMiB defaults to 2048 and caps it. Both defaults are visible, commented out, in /etc/rpi/swap.conf.

Why it's easy to misread

Every tool that has ever answered "how much swap do I have and where does it live" answers this configuration wrongly, or at least uselessly.

free reports 2.0Gi of swap. True — that's zram's decompressed capacity. It says nothing about the 2 GB on disk. swapon and /proc/swaps list /dev/zram0 and stop. ls /var/swap shows a 2 GB file that looks exactly like the swap file that has been on Pis for years. Put those together and the obvious reading is "2 GB of swap, stored in /var/swap" — which is wrong in a way that no single command contradicts.

The strongest demonstration of this: I switched the machine to Mechanism=zram, rebooted, and compared. The file was removed and 2 GB of disk came back:

                    before      after
/var/swap           2.0G        (deleted)
df / used           4.7G        2.7G
backing_dev         /dev/loop0  none

And swapon --show:

NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition   2G   0B  100

Byte-for-byte identical output, before and after. The swap subsystem was reconfigured, 2 GB of disk changed hands, and the tool you'd use to check swap could not tell the two states apart.

None of this is undocumented — swap.conf(5) is clear and thorough. But nothing routes you to the man page for a package you didn't know was installed, replacing a package you assumed still existed.

The other half: nothing applies until you reboot

swap.conf(5) warns about this, and it's worth confirming because it is the part that will bite a provisioning script rather than a person:

WARNING: After modifying any swap configuration, you must reboot the system for changes to take effect.

While running systemctl daemon-reload after configuration changes will generate new swap units, existing swap units are not typically stopped by systemd, and new units will not be started automatically as they are pulled in by swap.target which has already been reached during boot.

Observed exactly as described. With a drop-in in place selecting Mechanism=zram, after systemctl daemon-reload:

$ cat /etc/rpi/swap.conf.d/90-test-zram-only.conf
[Main]
Mechanism=zram

$ sudo systemctl daemon-reload
$ ls -lh /var/swap
-rw------- 1 root root 2.0G Aug 16 23:01 /var/swap
$ cat /sys/block/zram0/backing_dev
/dev/loop0
$ df -h --output=used / | tail -1
 4.7G

Nothing changed. No error, no warning, no non-zero exit. The config was valid and correctly placed; it simply does not take effect until rpi-swap's generator runs during early boot. A script that writes a drop-in, reloads, checks the exit code and reports success will report success — and be wrong until something reboots the box, possibly weeks later.

This is the same shape as the journald volatile-storage problem on the same OS: a correct config change that produces no visible effect and no error until a second, separate step happens.

The fix

First, find out what's actually managing swap before trusting any tutorial:

dpkg -l | grep -E 'dphys-swapfile|rpi-swap'
swapon --show
cat /sys/block/zram0/backing_dev 2>/dev/null   # a path here means zram+file
losetup -a                                      # resolves that path to the real file

If backing_dev names a loop device, you have a writeback file consuming disk that won't show up in any swap listing.

To change it, write a drop-in rather than editing the main file — /etc/rpi/swap.conf.d/ is read after /etc/rpi/swap.conf, sorted lexicographically across all the config directories, last-one-wins:

sudo mkdir -p /etc/rpi/swap.conf.d/
sudo tee /etc/rpi/swap.conf.d/90-local-swap.conf > /dev/null <<'EOF'
[Main]
Mechanism=zram
EOF
sudo reboot

Mechanism= takes auto, swapfile, zram, zram+file, or none. zram keeps compressed-RAM swap and drops the writeback file. swapfile gives you the traditional pre-Trixie behaviour back. none removes swap entirely and deletes the file.

To resize rather than switch mechanism, FixedSizeMiB under [File] or [Zram] sets an exact size and overrides the RamMultiplier calculation. Undo is just deleting the drop-in and rebooting — that restored this machine to stock, /var/swap and all, in one boot.

The reboot is not optional and not a formality. Verify after it, not before.

The generalisable habit

The failure mode worth extracting isn't "Trixie changed swap." It's that a tool's output can stay identical across a change it doesn't model. swapon isn't broken or lying; it reports swap areas, and a writeback device isn't one. It answers its own question correctly while the question you actually had — "where did my disk go" — goes unanswered by every tool you'd naturally reach for.

That's the class of thing worth a reflex: when a number doesn't reconcile, don't re-run the tool that already gave you an answer you don't believe. Go to the layer underneath it. Here that's /sys/block/zram0/backing_dev and losetup -a — two commands nobody thinks of as swap commands, which is precisely why the link stays invisible.

The related habit: when a package you rely on is missing, check what replaced it before concluding your install is broken. dphys-swapfile: command not found reads like damage. It's a deliberate migration with a man page.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Raspberry Pi OS Trixie throws away your logs on reboot, and /var/log/journal exists anyway to reassure you it doesn’t

The Homelab Postmortem — Tue, 18 Aug 2026 00:00:00 +0000

TL;DR : Raspberry Pi OS Trixie ships /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf with Storage=volatile. Your journal lives in RAM and is gone after a reboot — exactly when you need it. /var/log/journal/ exists but stays empty, which makes it look like persistence is already working. Fixing it takes a drop-in and a flush; the drop-in alone silently does nothing until the next boot.

The symptom

Your Pi does something bad — crash-loops, drops off the network, wedges. You reboot it to get back in, then go looking for what happened:

journalctl -b -1


Specifying boot ID or boot offset has no effect, no persistent journal was found.

The logs from the boot you actually care about do not exist. They never made it to disk.

Why it’s easy to convince yourself this is already fine

Two things conspire here.

First, the directory exists. /var/log/journal/ is present on a stock Trixie install:

$ ls -la /var/log/journal/
total 8
drwxr-sr-x+ 2 root systemd-journal 4096 Jun 18 09:19 .
drwxr-xr-x 6 root root 4096 Aug 16 18:31 ..

For years, the canonical way to enable persistent journald has been “create /var/log/journal and it starts persisting”. The directory’s presence is normally the signal that persistence is on. Here it’s present and empty, and the emptiness reads as “nothing has been logged yet” rather than “nothing will ever be written here”.

Second, the default config file agrees with you. /etc/systemd/journald.conf — the file you’d naturally open to check — says:

#Storage=auto

Commented out, default auto. And auto means “persist if /var/log/journal exists”, which it does. So the config file you inspected and the directory you found both say persistence should be working.

Neither of them is where the decision is being made.

Where the decision actually is

$ ls /usr/lib/systemd/journald.conf.d/
40-rpi-volatile-storage.conf
syslog.conf

$ cat /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf
[Journal]
Storage=volatile

A vendor drop-in, shipped by the distro, overriding the default you read in the main config file. Drop-ins in /usr/lib/systemd/journald.conf.d/ take precedence over /etc/systemd/journald.conf, so volatile wins and the journal is written to /run/log/journal/ — a tmpfs — instead.

The reasoning behind the default is defensible: it protects SD cards from log write wear, which is a real failure mode on Pis. The problem isn’t the choice, it’s that the choice is invisible from every place you’d normally look.

Don’t guess at precedence. Ask systemd what it actually resolved:

systemd-analyze cat-config systemd/journald.conf

That prints every file in load order with its contents, so you can see exactly which line won.

The fix everyone posts, and why it isn’t enough

The advice you’ll find in forum threads is: add a higher-priority drop-in under /etc. That part is right:

sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nStorage=persistent\n' | sudo tee /etc/systemd/journald.conf.d/99-persistent-storage.conf
sudo systemctl restart systemd-journald

Then check:

$ journalctl --header | grep 'File path'
File path: /run/log/journal/<machine-id>/system.journal

Still /run. The setting is correct — systemd-analyze cat-config confirms Storage=persistent is winning — and the journal is still in RAM. Restarting the service was not enough.

What’s missing is the flush. Moving the journal from the runtime location to the persistent one is a distinct operation, normally performed at boot by systemd-journal-flush.service. Change the setting mid-session and nothing triggers it:

sudo journalctl --flush


$ journalctl --header | grep 'File path'
File path: /var/log/journal/<machine-id>/system.journal
File path: /var/log/journal/<machine-id>/user-1000.journal

Now it’s real.

This is the part that makes the bug expensive. Without the flush, the drop-in does take effect — at the next reboot. So if you apply the incomplete fix, verify it by checking journalctl --header, and see /run, you’ll reasonably conclude your drop-in didn’t work and start debugging precedence rules that are already correct.

On the filename everyone tells you to use

Forum threads specify the override must be named 99-something.conf, on the grounds that it has to sort after the vendor file to win.

The sorting rule is real. The number isn’t. On the image checked here (2026-06-18 Trixie arm64 Lite) the vendor file is 40- , not the 70- those threads describe. Anything above 40- wins, so 50- would do.

99- is still the right choice — not because 40- demands it, but because it survives the vendor renaming the file, which apparently already happened once. Just don’t take the specific number in a forum post as fact about your system. Check:

ls /usr/lib/systemd/journald.conf.d/

The trade-off you’re accepting

Turning this on means writing logs to your boot media continuously. The vendor default exists for a reason: on a Pi running from an SD card, that’s real wear on a device that’s already the most common hardware failure point.

Worth turning on if you boot from an SSD or NVMe, or if you’re debugging something that survives reboots. Worth thinking twice about on a plain SD card — and if you do enable it there, cap the size so it can’t grow without bound:

[Journal]
Storage=persistent
SystemMaxUse=200M

The generalisable habit

Three separate things pointed at “persistence is on”: the directory existed, the main config said auto, and after the first fix attempt the config resolution confirmed persistent. All three were true. None of them described where log bytes were actually being written.

journalctl --header was the only thing that reported the live state — the actual open file — and it’s the check worth building the habit around. Config tells you intent. Headers tell you reality. When they disagree, something in between hasn’t run yet.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n

Stripe retries a failed webhook for three days, and I found out by getting the same email three times

The Homelab Postmortem — Tue, 18 Aug 2026 00:00:00 +0000

TL;DR : Stripe webhook delivery is at-least-once, not exactly-once. If a webhook fails for any reason and you fix the cause later, every queued retry succeeds afterward — and if your handler isn’t idempotent, that means re-running your whole side effect (sending an email, granting access, provisioning something) once per retry. Keep a marker per event/session and check it before acting, not just on send.

The setup

A small toolkit sold as a one-time purchase: buy via a Stripe Payment Link, a webhook fires on checkout.session.completed, a Cloudflare Worker verifies the signature and emails a download link. Straightforward, and it had already been tested end-to-end with a real charge before going live.

Shortly after that real charge, three separate copies of the exact same download email arrived, roughly thirty minutes apart.

What actually happened

Nothing was resending on my end. Nothing in the Worker was looping. The explanation was upstream: the very first delivery attempt for that charge had failed — a webhook secret had been misconfigured minutes earlier — so Stripe’s signature check on my side rejected it with a 400.

Stripe’s webhook delivery isn’t “try once and give up.” A non-2xx response is treated as a delivery failure, and Stripe retries on a backoff schedule for up to three days. The first few retries land roughly every few minutes to half an hour depending on how long the endpoint’s been failing.

The secret got fixed a few minutes after the first failure. From Stripe’s side, nothing about that mattered — it just kept retrying the same event on its normal schedule. Once the secret was correct, the next retry succeeded. So did the one after that. Each one was a completely valid, correctly signed, legitimate-looking webhook for a real completed checkout — because it was. Nothing distinguished retry three from delivery one except that my endpoint had already acted on delivery one’s twin.

Why this is easy to miss until it costs you

The natural mental model when building a webhook handler is “an event happens once, so my handler runs once.” That model is wrong in a way that doesn’t show up in testing, because a clean test run typically doesn’t have a failed first attempt — you write the handler, it works, you ship it. The retry behavior only becomes visible the first time something fails at exactly the wrong moment: mid-deploy, during a secret rotation, during any transient error on either side.

And the failure mode isn’t a crash or an error someone will report. It’s a customer quietly getting the same email two or three times, mildly annoyed, possibly assuming something is wrong with the business rather than realizing the delivery layer is precisely doing what it’s specified to do.

The fix

Stripe’s Checkout Session ID is a stable identifier for the same real-world event across every retry. That’s the natural idempotency key: before acting, check whether that session has already been handled; if it has, acknowledge and stop.

const marker = `delivered/${session.id}`;
if (await env.TOOLKIT_BUCKET.head(marker)) {
  console.log(`Session ${session.id} already delivered; skipping duplicate send.`);
  return new Response('ok (already delivered)', { status: 200 });
}

try {
  const downloadUrl = await createDownloadUrl(url.origin, env.DOWNLOAD_SIGNING_SECRET);
  await sendDownloadEmail(env, email, downloadUrl);
  // Written only after a confirmed send, so a failed attempt stays retryable.
  await env.TOOLKIT_BUCKET.put(
    marker,
    JSON.stringify({ email, deliveredAt: new Date().toISOString(), eventId: event.id }),
  );
  return new Response('ok', { status: 200 });
} catch (err) {
  // 500 so Stripe retries — a transient send failure shouldn't lose a sale.
  return new Response('delivery failed', { status: 500 });
}

The ordering matters as much as the check. The marker is written after the email send succeeds, not before it and not unconditionally. Writing it earlier would mean a transient failure in the email provider permanently marks the session as handled, and the customer never gets their download at all — trading a duplicate-email bug for a much worse silent-failure bug. The retry mechanism that caused this problem is also the thing you want protecting you against provider outages; the fix has to keep both properties.

How to check it without waiting for another real charge

You don’t need to make another purchase to verify this. Forge a correctly signed request against your own secret and send it directly:

SESSION="cs_idempotency_test_$(date +%s)"
# ...compute t=<timestamp>,v1=<hmac-sha256 of "<timestamp>.<body>" with the webhook secret>...

curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok
curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok (already delivered)
curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok (already delivered)

Same technique as forging a signature to test rejection — construct the real thing yourself instead of guessing at what production will do.

The generalisable habit

“At-least-once” is the standard delivery guarantee for webhooks, queues, and most event systems, precisely because “exactly-once” is expensive or impossible to guarantee end-to-end. Any time you’re consuming events from a system that documents at-least-once delivery, the assumption to design against isn’t “this fires once” — it’s “this will eventually fire more than once, probably at the worst possible time, and my handler needs to be safe either way.”

The cost of skipping that isn’t a crash you’ll notice in a log. It’s a side effect running twice, silently, on exactly the request path most likely to be running for a real customer at the exact moment you were still shaking out the rest of the system.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n