DEV Community: Guatu

Tailscale Exit Node Logic: Why Auto-Select Fails for Security and How to Fix It

Guatu — Mon, 12 Oct 2026 16:15:48 +0000

The Tailscale tray icon says "Connected," and that tells you nothing about where your web traffic is going. If no exit node is active, Tailscale only carries traffic to your tailnet. Everything else (browsing, DNS lookups to non-tailnet resolvers, the background update checks your OS runs) goes out through whatever Wi-Fi you're on, just as it would without Tailscale. On a hotel or airport network, that's the traffic you wanted protected.

So you turn on an exit node. That leaves one question: should Tailscale pick the node for you (auto), or should you pin one and enforce it? Tailscale's docs present auto-select as the convenient default, and on a desktop that never moves it's usually fine. On a laptop that roams between networks you don't control, auto-select is solving a different problem from yours.

This post compares the two approaches, explains why auto-select falls short as a security control, and shows the setup I'd use instead.

Why you'd have to choose

The decision shows up as soon as you have more than one exit node. Common setups:

A home exit node running on a small VM or LXC (I covered the unprivileged-container version in Tailscale Kernel TUN in Unprivileged LXC)
A cloud VPS exit node for when home upload bandwidth is bad
Mullvad exit nodes through the Tailscale add-on
Exit nodes shared with you from someone else's tailnet

With one exit node, there's nothing to decide. With several, the client has to pick one. You can let it pick (--exit-node=auto:any) or you can pick yourself (--exit-node=exit-home). The two options look like a convenience question. They're really a question about what you're optimizing for.

Underneath that sits a second decision most people never make on purpose: what should happen when the chosen exit node is unreachable? Should traffic stop (fail closed), or fall back to the local network (fail open)? The two approaches answer this differently, and the answer matters more than which node gets picked.

Option A: Auto-select (`auto:any`)

What it is

Recent Tailscale clients can choose the exit node themselves. You can ask what the client would pick:

# Shows the exit node Tailscale would suggest right now
tailscale exitnode suggest

# Tell the client to keep choosing automatically
tailscale set --exit-node=auto:any

The suggestion is based on reachability and latency (and region, for Mullvad-style location-based nodes). As conditions change, the client can switch to a different node. Admins can also push this through MDM/system policy (ExitNodeID set to auto:any) so the user can't turn it off.

Strengths

No hostname to keep current. Add or retire an exit node and clients adjust on their own.
Good latency. If you travel between regions and have exit nodes in several of them, auto picks the close one. That's useful if you mostly care about speed.
Survives a dead node. When one exit node goes offline, auto moves to another one rather than leaving you stuck on a dead route.
Fleet-friendly. For an org with dozens of regional exit nodes and centrally managed laptops, auto plus MDM policy takes almost no ongoing work.

Weaknesses

The issue is what auto doesn't know, and there are two gaps.

It doesn't know which exit nodes you trust. "Any" means any exit node your ACLs let this device use. If a friend's shared node, a test VPS you forgot about, or a Mullvad node in a country you'd rather not appear from is eligible, it's in the pool. Auto won't treat your home node, which you patch and control, differently from a node someone else runs. Every eligible exit node can see your plaintext egress traffic after it leaves the WireGuard tunnel. That makes "who runs this box" a security question, and auto answers it by measuring latency.

It doesn't know anything about your local network. Auto has no idea whether you're on the home LAN you trust or on airport Wi-Fi. Tailscale doesn't have a built-in "trusted SSID" setting. Whether the exit node gets enabled at all is up to you or your MDM policy. Auto only decides which node to use after something else has decided to use one.

There's a smaller operational point too. The egress IP can change during a session when auto moves you to another node. Sites that tie sessions to IPs (banking, some SSO providers, anything doing geo-velocity checks) may log you out or show a challenge page. That doesn't make you less secure, but it's annoying enough that people turn exit nodes off, which does.

When it shines

Auto-select works well when every eligible exit node belongs to one operator you trust, when ACLs keep the pool tight, and when policy forces the exit node on. That describes a well-run corporate fleet. It doesn't describe a homelab tailnet where your shared-node experiments live next to your actual egress box.

Option B: Pinned exit node, enforced by the OS

What it is

You choose a specific exit node by its MagicDNS name and make the operating system enforce it, so it doesn't depend on you remembering to click the toggle. The naive version looks like this:

# The naive pin: works until it doesn't
sudo tailscale up --exit-node=100.64.0.10 --exit-node-allow-lan-access=false

There are two problems with this. Pinning by tailnet IP is fragile. If you rebuild the exit node and it gets a new address, every client stays pinned to an address that no longer exists. Use the hostname instead (--exit-node=exit-home). Also, tailscale up with flags will complain if you leave out settings you set earlier. tailscale set changes only what you pass, which makes it the right tool for automation.

The second problem is more interesting. A pinned exit node fails closed. If exit-home is unreachable (home ISP outage, the VM is rebooting, a NAT traversal problem after you roam to a new access point), the client keeps sending your default route into a tunnel that goes nowhere. In practice that looks like half-loaded pages, DNS timeouts, and a browser stuck on "Resolving host…" forever. From a security standpoint, that's the correct behavior. It's also exactly what makes people turn the exit node off at the airport gate and forget to turn it back on.

Making the pin hold up: a NetworkManager dispatcher

The fix is to stop treating the exit node as a setting you flip by hand and treat it as policy that runs on network events. NetworkManager runs every executable in /etc/NetworkManager/dispatcher.d/ with the interface and action as arguments. That's enough to say: "whenever a Wi-Fi connection comes up, enforce the exit node, preferring my own node and falling back to a second trusted one."

#!/usr/bin/env bash
# /etc/NetworkManager/dispatcher.d/50-wifi-safe
# Enforce a trusted exit node whenever Wi-Fi comes up. Idempotent.
IFACE="$1"; ACTION="$2"
PREFERRED="exit-home"     # MagicDNS names, not IPs
FALLBACK="exit-vps"

case "$ACTION" in up|connectivity-change) ;; *) exit 0 ;; esac
[ "$(nmcli -g GENERAL.TYPE device show "$IFACE" 2>/dev/null)" = "wifi" ] || exit 0

# Give tailscaled a moment after roaming before probing peers
for _ in 1 2 3 4 5; do
  tailscale status >/dev/null 2>&1 && break; sleep 2
done

pick=""
for node in "$PREFERRED" "$FALLBACK"; do
  if tailscale ping --c 1 --timeout 3s "$node" >/dev/null 2>&1; then
    pick="$node"; break
  fi
done

# No trusted node reachable: still pin the preferred one. Fail closed, never open.
tailscale set --exit-node="${pick:-$PREFERRED}" --exit-node-allow-lan-access=false
logger -t wifi-safe "iface=$IFACE action=$ACTION exit-node=${pick:-$PREFERRED}"

sudo chown root:root /etc/NetworkManager/dispatcher.d/50-wifi-safe
sudo chmod 755 /etc/NetworkManager/dispatcher.d/50-wifi-safe   # NM skips files that aren't root-owned

A few details here are deliberate:

No SSID allowlist. The script treats every Wi-Fi network as hostile, including your home network. That's the "trust the tailnet, not the network" model from the earlier post on dispatcher hardening. Allowlists rot: the coffee shop renames its SSID, someone sets up an evil twin of your home network name, and suddenly you're trusting something you shouldn't. If you really want home traffic to skip the exit node, match on something harder to spoof than an SSID.
The fallback list is short and curated. That's the actual difference from auto:any. You still get failover, but only among nodes you've decided to trust.
Total failure still fails closed. If neither node responds, the script pins the preferred one anyway. You lose connectivity instead of quietly sending traffic in the clear. The logger line means you can check journalctl -t wifi-safe and see why.

Locking down the local side

An exit node protects egress. It does nothing about other devices on the hotel network probing your laptop. Two pieces cover that, and both have persistence gotchas I've written about separately.

Bind Wi-Fi connections to the restrictive firewall zone through NetworkManager, so the binding survives reboots (runtime-only firewall-cmd changes vanish, as covered in Firewall Zone Persistence):

# Every saved Wi-Fi profile goes to the public zone
for c in $(nmcli -g NAME,TYPE connection show | awk -F: '$2=="802-11-wireless"{print $1}'); do
  sudo nmcli connection modify "$c" connection.zone public
done

# tailscale0 isn't NM-managed, so assign it permanently via firewalld
sudo firewall-cmd --permanent --zone=trusted --add-interface=tailscale0
sudo firewall-cmd --reload

(The for loop splits on whitespace, so SSIDs with spaces need a while read loop instead. Most of mine don't have spaces, but yours might.)

Then turn off the name-resolution protocols that answer to anyone on the LAN. Use a drop-in rather than editing resolved.conf directly. On recent Fedora the vendor defaults live elsewhere, and editing the main file can shadow them in ways that are hard to notice (Systemd Config Merges has the details):

# /etc/systemd/resolved.conf.d/50-untrusted-lan.conf
[Resolve]
LLMNR=no
MulticastDNS=no

sudo systemctl restart systemd-resolved
resolvectl status | grep -E 'LLMNR|mDNS'   # both should read "no"

Diagnosing "half-loaded pages"

When the pinned approach breaks, the symptom is always the same: some pages load from cache, others hang, and DNS looks flaky. The cause can be on either end, and the fix depends on which one it is.

# Is the local daemon healthy? Look for health warnings at the bottom.
tailscale status

# Which exit node is set, and does the client think it's online?
tailscale status --json | jq '.ExitNodeStatus'

# Can we actually reach it, and via DERP or direct?
tailscale ping exit-home

# Is the problem DERP/UDP reachability from this network?
tailscale netcheck

How to read it: if tailscale status shows health warnings about DNS forwarding failing or no DERP connection, the problem is local. The network you're on is blocking UDP, a captive portal hasn't been accepted, or tailscaled is wedged. If status looks clean but ExitNodeStatus.Online is false or tailscale ping times out, the exit node itself is down, and that's what the fallback in the dispatcher exists for.

Captive portals deserve a mention. Until you accept the portal, nothing gets through, including the tunnel to your exit node. Most portals intercept HTTP, which never reaches them because your traffic is pointed at a dead route. The practical fix is to temporarily clear the exit node with tailscale set --exit-node=, accept the portal, and then reconnect Wi-Fi so the dispatcher puts the pin back. Tailscale's captive portal detection helps here on recent clients, but I wouldn't count on it at every hotel.

Weaknesses of Option B

More moving parts: a script, file permissions, firewall zones, a resolved drop-in.
Linux/NetworkManager-specific. macOS and Windows need different mechanisms (MDM policy is the realistic choice there).
Captive portals need a manual step.
If both trusted nodes go down, you're offline until one comes back. That's the point of the design, but it's still a cost.

Decision framework

Criterion	Auto-select (`auto:any`)	Pinned + dispatcher
Who chooses the egress operator	Latency and reachability	You, from a short trusted list
Awareness of local network trust	None	Treats every Wi-Fi as hostile
Behavior when the chosen node dies	Moves to any other eligible node	Moves to the next trusted node, else fails closed
Egress IP stability	Can change mid-session	Stable unless failover fires
Setup effort	One command or MDM key	Script, permissions, firewall, resolved drop-in
Ongoing maintenance	Near zero	Update the node list when you rebuild exit nodes
Platform coverage	All Tailscale clients	Linux with NetworkManager
Best fit	Managed fleet, single-operator exit pool	Personal laptops on a mixed-trust tailnet
Main risk	Egress through a node you didn't mean to trust	Offline when every trusted node is down

My pick and why

On a roaming Linux laptop, I'd use the pinned exit node with dispatcher enforcement every time.

My reasoning: the exit node operator sees your traffic after decryption. That makes the choice of exit node a trust decision, and trust decisions shouldn't come from a latency measurement. Auto-select is a good availability feature, and it was built as one. Once your tailnet holds nodes with different owners (shared nodes, experiments, third-party providers), "the fastest eligible node" and "a node I trust with my traffic" stop meaning the same thing, and auto has no setting that tells them apart.

The pinned approach without automation has a real failure mode, though. It fails closed, which means it's annoying, which means people disable it. The dispatcher takes care of that. It reapplies the pin on every network change, handles failover inside a list you control, and only cuts connectivity when no trusted option exists. You get most of auto's convenience and keep the trust decision for yourself.

If you run a fleet where every exit node is yours and ACLs keep the pool clean, auto plus MDM enforcement is a reasonable choice. I'd still audit the eligible pool regularly with tailscale exit-node list to check that it contains only what you expect.

Things I'd watch out for:

Pin by MagicDNS name, never by tailnet IP. Rebuilt nodes get new addresses, and clients pinned to the old one fail with no useful error.
Use tailscale set in scripts, not tailscale up. up wants the full flag set restated and will refuse or reset settings you didn't mention.
Make firewall and resolver changes permanent. A hardening setup that disappears after a reboot is worse than none, because you believe it's still there.
"Connected" isn't "protected." Check ExitNodeStatus and an external IP check (curl -s https://ifconfig.me) before trusting a hostile network. The tray icon only reports whether the tunnel is up.

If you're building the exit node side of this, Tailscale Subnet Routers covers the router configuration that pairs with it. If you want help designing this kind of zero-trust egress for a team instead of a single laptop, that's the kind of infrastructure work I do.

Systemd Config Merges: Avoiding Silent Shadowing with Drop-ins on Fedora 43

Guatu — Fri, 09 Oct 2026 16:15:48 +0000

You edit a unit file, run systemctl daemon-reload, restart the service, and nothing changes. Or you add one Environment= line in a drop-in, and a variable you never touched ends up with a different value. In neither case does systemd raise an error. It merges the files it finds using its own rules, then runs the result.

This is a gotcha post about how those merges work. A drop-in can shadow a unit file or another drop-in without any warning, and list-type settings cause the worst of it. My examples come from Fedora 43, but the merge logic has been stable for years, so all of this applies to any modern systemd distro.

The Symptom

Most versions of this bug look like one of these:

You edit /usr/lib/systemd/system/foo.service (or /etc/systemd/system/foo.service) and the change has no effect.
You add a drop-in to "just add one thing" to an ExecStartPre= chain or an environment, and something else in the service breaks.
You change a variable in an env file, restart, and the process still sees the old value.
You add a drop-in with a new ExecStart= and the unit refuses to start with:

foo.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

That last one is the lucky case, because at least systemd complains. The others fail silently. The service starts, reports active (running), and does something slightly different from what you think you configured.

What I Expected

A unit feels like a file. One file, one config. Drop-ins feel like patches: you put override.conf next to the service, and its lines "win." I worked from that mental model for a long time, and I see it constantly in forum answers and copy-pasted setup scripts.

Under that model, editing the main unit file should change behavior. Adding a line to a drop-in should add exactly that line and nothing else. Changing an env file should change the environment. All three assumptions are wrong in specific, predictable ways.

What Actually Happens

systemd builds every unit from a stack of files spread across several directories. It then applies merge rules that depend on the type of each setting. Four separate mechanisms interact here, and each one can hide a change.

1. Whole-file shadowing by directory priority

Unit files are searched in a fixed order. For system units, these are the directories that matter:

Path	Who owns it	Priority
`/etc/systemd/system/`	You (admin)	Highest
`/run/systemd/system/`	Runtime, generators	Middle
`/usr/lib/systemd/system/`	Packages (RPM)	Lowest

If foo.service exists in both /etc and /usr/lib, the /etc copy wins entirely. systemd doesn't merge the /usr/lib file. It ignores it. That's why systemctl edit --full foo.service is a trap: it copies the vendor unit into /etc, and from then on every package update to the vendor unit is invisible. Months later the RPM ships a fix to ExecStart=, dnf installs it cleanly, and your service keeps running the frozen copy.

It cuts the other way too. You edit the file under /usr/lib directly, but a full copy is sitting in /etc from some earlier experiment. Your edit does nothing, and the next package update will overwrite it anyway.

2. Drop-in shadowing by filename

Drop-ins live in foo.service.d/ directories under each of those same paths. systemd collects every *.conf file from all of them, sorts them by filename (not by directory), and applies them in that order. For scalar settings, later files win.

Here's the catch. If two drop-ins have the same filename in different directories, systemd only uses the higher-priority one. A /etc/systemd/system/foo.service.d/override.conf completely masks a /usr/lib/systemd/system/foo.service.d/override.conf. That vendor drop-in may have set a timeout or a hardening option you never knew about, and now it's gone.

Fedora adds another layer because it ships top-level type drop-ins. A directory named service.d/ applies to every service on the box, and Fedora uses one (10-timeout-abort.conf) to change what happens when a service hits its stop timeout. Prefix drop-ins work too: foo-.service.d/ applies to foo-bar.service, foo-baz.service, and so on. As a result, settings in your unit can come from files whose names never mention it.

3. List settings append, scalars replace

This one causes the most damage. The behavior changes from setting to setting, and nothing in the unit file syntax tells you which kind you're dealing with.

Scalar settings (User=, TimeoutStartSec=, Restart=, WorkingDirectory=): last assignment wins. Drop-ins behave the way you'd expect.
List settings (Environment=, EnvironmentFile=, ExecStartPre=, ExecStartPost=, ExecStart= for oneshot, ReadWritePaths=, and many more): each assignment appends to the list. An empty assignment (ExecStartPre= with nothing after it) resets the list to empty.

That gives you two opposite failure modes from the same drop-in.

First, accidental accumulation. You want to replace a pre-start step, so you write a new ExecStartPre= in a drop-in. systemd appends it. Now both the old and the new step run in order, and if the old one fails, the unit never starts.

Second, accidental truncation. Once people learn about the empty-reset trick, some start every drop-in with an empty assignment "to be safe." Every entry that came before gets wiped, including entries from vendor files they never read.

Truncation is the sneakier of the two. Here's the vendor unit:

# /usr/lib/systemd/system/agent-worker.service
[Service]
Type=simple
ExecStartPre=/usr/libexec/agent-worker/check-config
ExecStartPre=/usr/libexec/agent-worker/clean-sockets
ExecStart=/usr/bin/agent-worker --serve

And here's a drop-in written to "add one migration step":

# /etc/systemd/system/agent-worker.service.d/override.conf
[Service]
# Reset "just in case" -- this deletes check-config and clean-sockets
ExecStartPre=
ExecStartPre=/usr/libexec/agent-worker/migrate

After daemon-reload, the merged unit runs only migrate before start. The config check is gone, and so is the socket cleanup. The service starts fine on the first boot, and on the next unclean restart it trips over a stale socket that clean-sockets used to remove. When you go looking, the drop-in looks innocent and the vendor file looks correct. Neither file shows the problem on its own. You only see it in the merged result.

I see the same shape in AI agent setups, which is part of why I'm writing this up. Many agent runners take their allowed-tool or safe-binary list from an environment variable or a list setting in the unit. A drop-in meant to add one tool resets the list, and the agent quietly loses every tool except the new one. If you've read how I structure agent skills, this is the systemd version of a registry losing its entries: nothing crashes, and the agent just reports that it "can't" do things it could do yesterday.

4. Environment precedence across sources

Environment variables add their own ordering. Variables loaded from EnvironmentFile= override variables set with Environment=, no matter which file declared them. Multiple env files are read in the order they're listed, and later files win.

So if the main unit sets Environment=LOG_LEVEL=info, and some drop-in adds EnvironmentFile=/etc/agent-worker/legacy.env containing LOG_LEVEL=debug, your process runs at debug. A drop-in that sets Environment=LOG_LEVEL=warn won't change that, because env files take precedence. There's a timing difference to keep in mind as well. Environment= lines are part of the unit definition and need a daemon-reload. Env files are read from disk each time the service starts, so editing one only needs a restart. Mix up the two and you'll restart a service expecting a change it hasn't loaded yet.

The Fix

None of this is a bug. It's documented behavior, spread across systemd.unit(5), systemd.service(5), and systemd.exec(5). The fix is to stop reasoning about individual files and start checking the merged result.

Inspect the merge, not the files

Start with these commands. Each one answers a different question:

# Every file that contributes to the unit, in merge order
systemctl cat agent-worker.service

# Which file is the "main" unit, and which drop-ins were applied
systemctl show agent-worker.service -p FragmentPath -p DropInPaths

# The final, merged value of a list setting
systemctl show agent-worker.service -p ExecStartPre

# Final environment config (Environment= entries plus the env file list)
systemctl show agent-worker.service -p Environment -p EnvironmentFiles

# System-wide: every overridden, extended, or masked unit
systemd-delta --type=overridden,extended

systemctl cat is where I start, because it prints file paths as comments above each section. A stray full copy in /etc shows up right away. systemctl show -p ExecStartPre is the definitive answer for list settings. It prints one { path=... ; argv[]=... } block per entry, so you can count the entries and confirm which ones survived the merge.

systemd-delta doesn't get used enough. Run it after any distro upgrade. It lists every unit where /etc overrides /usr/lib, and that's exactly the list of services that have stopped receiving vendor fixes.

systemctl show -p Environment only tells you what the unit declares. It doesn't expand env files. To see what the running process actually received, check one named variable and leave the rest alone:

pid=$(systemctl show -p MainPID --value agent-worker.service)
# Check one known, non-secret variable. Don't dump the whole environ,
# it usually contains credentials loaded from env files.
sudo tr '\0' '\n' < /proc/"$pid"/environ | grep '^LOG_LEVEL='

Write drop-ins that state their intent

Two rules cover most of the damage. Never use a reset unless you mean to replace the whole list. When you do mean it, restate the full list in the same file so a reader can see the final value without opening another file.

Appending one step, the correct way:

# /etc/systemd/system/agent-worker.service.d/20-migrate.conf
[Service]
# Appends after the vendor's check-config and clean-sockets
ExecStartPre=/usr/libexec/agent-worker/migrate

Replacing the list on purpose:

# /etc/systemd/system/agent-worker.service.d/20-prestart.conf
[Service]
# Intentional reset: this file owns the complete pre-start chain
ExecStartPre=
ExecStartPre=/usr/libexec/agent-worker/check-config
ExecStartPre=/usr/libexec/agent-worker/clean-sockets
ExecStartPre=/usr/libexec/agent-worker/migrate

For ExecStart= on a non-oneshot service, the reset isn't optional. systemd only allows one entry, so the reset-then-set pattern is the only valid way to override it.

Name your drop-ins. override.conf is the default name systemctl edit uses, which makes collisions with vendor drop-ins of the same name more likely. Use numbered, descriptive names instead, so the sort order is obvious and nothing gets masked by accident. On Fedora 43's systemd you can do that straight from systemctl edit:

sudo systemctl edit --drop-in=20-migrate agent-worker.service

Prefer drop-ins over full copies, and revert the copies you have

If systemd-delta shows a full unit override you didn't mean to keep, systemctl revert removes the /etc copy and every admin drop-in for that unit, so it falls back to the vendor version:

sudo systemctl revert agent-worker.service
sudo systemctl daemon-reload
# Then re-apply only what you actually need, as a named drop-in
sudo systemctl edit --drop-in=20-migrate agent-worker.service

Before reverting, read the /etc copy with systemctl cat and copy any real customizations somewhere. Revert doesn't ask what's in there.

One environment file per service

Environment shadowing goes away when there's one source of truth. My pattern is a single drop-in that resets the env file list and points at one managed file:

# /etc/systemd/system/agent-worker.service.d/10-env.conf
[Service]
# One source of truth; the leading "-" would make it optional, so omit it
EnvironmentFile=
EnvironmentFile=/etc/agent-worker/agent-worker.env

Here the reset is deliberate and earns its place: it guarantees no vendor or legacy env file comes along for the ride. The .env file is generated from version control or a secrets manager, not edited by hand in three different places. This is the same discipline I use for agent credentials in two-tier service accounts. A secret should live in exactly one file, and the unit should name that file explicitly.

User units deserve extra care. A systemd --user service inherits from the user manager's environment, which gets assembled from environment.d files, systemctl --user import-environment calls, and whatever your login session pushed in. That's another layer of shadowing, one you can't see in the unit at all. Check it with systemctl --user show-environment. For anything that matters, set the variables in the unit's env file instead of relying on inheritance. The execution-context surprises in headless Claude sandboxing come from the same root problem: a process gets a different context than the shell you tested in.

Fixing the config doesn't fix the service

This step catches people even after they've fixed the merge. Correcting the drop-in fixes what systemd starts. It doesn't fix what the application remembers. Services that keep state between runs (session files, caches, agent conversation histories) can carry the effects of the broken config into the fixed one.

Agents are the worst case. An agent that ran for a while with a truncated tool list has a history full of failed tool calls and notes like "this tool is unavailable." Restore the tools, restart the service, and the agent can still reload that history and keep avoiding tools that work fine now. The config is correct and the behavior isn't, because the state learned from the broken run.

So my reset procedure has a step after the restart:

#!/usr/bin/env bash
set -euo pipefail
svc=agent-worker.service
state_dir=/var/lib/agent-worker/sessions  # example path

sudo systemctl daemon-reload
# Confirm the merge before trusting it
systemctl show "$svc" -p ExecStartPre -p EnvironmentFiles
sudo systemctl stop "$svc"
# Clear state that was learned under the broken config
sudo find "$state_dir" -mindepth 1 -delete
sudo systemctl start "$svc"
systemctl status "$svc" --no-pager

Stop before you clear state, not after. Otherwise the running process can write a fresh copy of the stale session back to disk during shutdown.

Why This Matters

You'll hit this in a few predictable situations: after a distro upgrade where vendor units changed underneath a full /etc copy, when you inherit a box with years of override.conf files, when a config management tool and a human both write drop-ins for the same unit, and any time you add to a list setting without checking what it already holds. Fedora's frequent release cadence makes the first one more likely than on slower distros, because vendor units change often.

This is the same class of problem as firewalld runtime changes that vanish on reload. The file you edited and the configuration that's actually running are two different things, and the tooling won't point out the gap.

What I'd do differently, and what I now do by default:

Never use systemctl edit --full on a packaged unit. Use a named drop-in.
Never start a drop-in with an empty assignment unless the same file restates the complete list.
Check systemctl show -p <Setting> after every change to a list setting, not just systemctl status.
Run systemd-delta after every major Fedora upgrade and look at each override.
Keep one env file per service, and treat any second EnvironmentFile= as a bug.
Treat "config fixed" and "service fixed" as two separate checks, and clear learned state when a broken config ran long enough to leave it behind.

If you're running agent workloads or long-lived services on hosts where this kind of drift has piled up for years, that's work I do through GuatuLabs consulting. Most of the time, though, systemctl cat and five minutes of reading turn up the problem. The merged config is always available if you ask systemd for it.

Firewall Zone Persistence: Why firewalld Runtime Changes Vanish and How to Pin Them

Guatu — Wed, 07 Oct 2026 16:15:47 +0000

You run firewall-cmd --zone=public --change-interface=wlp2s0, firewalld answers success, and --get-active-zones shows the change. After the next reboot, a firewall-cmd --reload, or a Wi-Fi reconnect, the interface is back in whatever zone it was in before. firewalld doesn't log anything about it.

This isn't a bug. firewalld keeps two separate configurations, and on a NetworkManager system a third component also gets a say in which zone an interface belongs to. If you don't know about all three, any change you make by hand will eventually be undone.

What I Expected

My mental model was the iptables one: you run a command, the rule exists, and if you want it to survive a reboot you save it. I knew firewalld had a --permanent flag, so I assumed the fix was to tack it on and move on.

There are two problems with that assumption. First, --permanent on its own doesn't change the running firewall at all. Second, for interfaces managed by NetworkManager, the zone binding isn't really firewalld's to keep. NetworkManager pushes a zone to firewalld every time a connection activates. If you've only told firewalld, your binding gets replaced the next time the connection comes up.

What Actually Happens

Runtime and permanent are two separate configs

firewalld keeps a runtime configuration (what the kernel is enforcing right now) and a permanent configuration (XML files in /etc/firewalld/ that get loaded at startup and on reload). Every firewall-cmd call writes to exactly one of them:

# Runtime only: takes effect now, gone on reload/restart/reboot
sudo firewall-cmd --zone=public --add-service=ssh

# Permanent only: written to disk, NOT active until reload
sudo firewall-cmd --permanent --zone=public --add-service=ssh

# Both: the boring, correct pattern
sudo firewall-cmd --permanent --zone=public --add-service=ssh
sudo firewall-cmd --reload

That gives you two ways to fail, and they mirror each other. If you use runtime only, the rule works until the next reload, and a reload is exactly what someone else's Ansible run or package update will trigger. If you use permanent only, the rule doesn't exist yet, so you test it, see it fail, and then "fix" it with a runtime command. Now the two configs disagree and nothing tells you.

The --reload itself is the destructive step that people overlook. A reload throws away the runtime state and rebuilds it from the permanent config. Every runtime-only change you made since the last reload is gone, including the one you made five minutes ago to debug something.

NetworkManager owns the zone of the interfaces it manages

This part is less well known. On Fedora, RHEL, and most desktop distros, NetworkManager manages your Ethernet and Wi-Fi interfaces. Each NM connection profile has a connection.zone property. When the connection activates, NetworkManager calls firewalld over D-Bus and assigns the interface to that zone. If connection.zone is empty, which is the default for every profile, NM puts the interface in firewalld's default zone.

So the sequence goes like this:

You run firewall-cmd --zone=drop --change-interface=wlp2s0. firewalld moves the interface. Runtime only.
Your laptop sleeps, wakes up, and reconnects to Wi-Fi.
NetworkManager activates the connection and tells firewalld to put wlp2s0 in the zone from the profile (empty, so the default zone).
Your interface is back in public. Your drop binding is gone.

Writing the binding into the permanent zone XML (--permanent --zone=drop --add-interface=wlp2s0) doesn't reliably fix this, because NM reasserts its own zone on every activation. Newer firewall-cmd builds detect NM-managed interfaces and print a message about the interface being under NetworkManager's control, and in some cases they forward the change to NM for you. That behavior has changed across versions, so I don't count on it. For an NM-managed interface, I treat NM as the source of truth and set the zone there.

Zone bindings are per connection, not per interface

There's one more consequence. The zone is attached to the connection profile, not to the device. wlp2s0 on your home network and wlp2s0 at an airport are two separate NM connections with two separate connection.zone values. Pinning the zone for the home SSID tells NM nothing about the next network you join. A new SSID creates a new profile with an empty zone, so it lands in the default zone.

That turns out to be the most useful behavior in the whole stack, as long as the default zone is set correctly.

The Fix

1. Set zones on the NetworkManager connection, not in firewalld

For any NM-managed interface, put the zone in the profile:

# See which connection is active on which device
nmcli -f NAME,UUID,DEVICE connection show --active

# Pin the zone on the connection profile (persists in the NM keyfile)
sudo nmcli connection modify "HomeWiFi" connection.zone home

# Re-activate so NM pushes the new zone to firewalld now
sudo nmcli connection up "HomeWiFi"

# Verify from firewalld's side
firewall-cmd --get-active-zones

The setting is stored in the connection's keyfile under /etc/NetworkManager/system-connections/, so it survives reboots and reconnects. It also survives firewall-cmd --reload, because NM reapplies it the next time the connection activates. Since it's stored per connection, it applies only to that SSID.

If you'd rather not bounce the connection, sudo nmcli device reapply wlp2s0 usually applies the change without a full disconnect. I still run connection up when I'm making a change by hand, because the result is unambiguous.

2. Make the default zone the untrusted one

Every new network falls back to firewalld's default zone. That makes the default zone the most important security decision on a laptop, and most people never change it from what the distro shipped. Fedora Workstation ships FedoraWorkstation as the default zone, and that zone allows a wide high-port range. That's a reasonable choice for a desktop on a home LAN. It's a bad one for a café.

# Check what new networks will land in
firewall-cmd --get-default-zone

# Make unknown networks untrusted by default
sudo firewall-cmd --set-default-zone=public

--set-default-zone is one of the few firewalld commands that changes runtime and permanent configuration together. It writes DefaultZone= to /etc/firewalld/firewalld.conf and applies the change immediately, so you don't need --permanent or a reload.

You end up with an allowlist built on inheritance. Every network is untrusted unless its NM profile explicitly says otherwise. You don't have to remember to lock down a new network. You only have to remember to unlock the networks you trust. If you forget, the failure is that something like a file share doesn't work at home. That's annoying, but it's a much safer way to fail than accidentally running a home-level ruleset on hotel Wi-Fi.

I build the rest of the mobile hardening on this rule (exit-node activation, DNS lockdown, captive portal handling), and I covered how the dispatcher scripts fit around it in Trust the Tailnet, Not the Network. The firewalld side is the foundation for all of it: if the zone doesn't persist, nothing built on top of it holds.

3. Bind interfaces NM doesn't manage in firewalld's permanent config

Not every interface goes through NetworkManager. tailscale0 is the one that matters to me. Check how NM sees it first:

nmcli device status | grep -E 'DEVICE|tailscale'

If it shows as unmanaged, NM won't assert a zone for it, and firewalld's permanent config is the right place for the binding:

sudo firewall-cmd --permanent --zone=trusted --add-interface=tailscale0
sudo firewall-cmd --reload
firewall-cmd --get-zone-of-interface=tailscale0

Bind the interface, not the source range. A common suggestion is --add-source=100.64.0.0/10 on the trusted zone. That trusts any packet carrying a CGNAT-range source address arriving on any interface, including the café Wi-Fi, where someone on the same L2 segment can put whatever source IP they like on a packet. Binding to tailscale0 means the trust applies only to traffic that came out of the WireGuard tunnel and was authenticated by the tailnet. It's the same least-privilege thinking as scoping Kubernetes service accounts, applied to a network interface instead. It's also what lets subnet routing into a homelab work without opening the LAN-facing side.

4. If you experimented in runtime, promote it on purpose

Sometimes the right workflow really is to try things in runtime and keep what works. firewalld has a command for that:

# Snapshot the current runtime state into permanent config
sudo firewall-cmd --runtime-to-permanent

It copies everything in runtime to permanent, including the throwaway rule you added while debugging and meant to remove. Diff the two before you run it:

zone=public
diff <(firewall-cmd --zone=$zone --list-all) \
     <(firewall-cmd --permanent --zone=$zone --list-all)

No output means the runtime and permanent configs agree. Any output shows exactly what a reload would throw away, or what --runtime-to-permanent would make permanent. I run this diff before every reload on a machine I didn't configure myself.

Runtime does have one feature I like: --timeout. When you're testing a rule over SSH and might lock yourself out, a time-limited runtime rule removes itself after the timeout:

# Rule expires automatically after 5 minutes
sudo firewall-cmd --zone=public --add-port=8443/tcp --timeout=300

It's the firewall equivalent of a dead-man's switch. If the rule works, add it with --permanent and reload.

The Shadowing Problem You Create Along the Way

Even when you use --permanent correctly, it has a side effect people don't notice. Zone definitions ship in /usr/lib/firewalld/zones/. The first time you permanently modify a zone, firewalld writes a full copy of that zone to /etc/firewalld/zones/<zone>.xml. From then on, the /etc copy completely replaces the vendor version. If a later firewalld update changes the shipped definition of public, you won't get that change, because your local copy wins.

This is the same trap as editing /etc/systemd/resolved.conf directly instead of using a drop-in under resolved.conf.d/: one local edit and you've forked the whole vendor config. firewalld zones don't support drop-ins, so you can't avoid the copy. You can keep an eye on it, though:

# Which zones have you forked from vendor defaults?
ls /etc/firewalld/zones/

# Compare your copy against what the package ships
diff /usr/lib/firewalld/zones/public.xml /etc/firewalld/zones/public.xml

# Reset a zone back to the vendor definition
sudo firewall-cmd --permanent --load-zone-defaults=public
sudo firewall-cmd --reload

I keep the list of forked zones short and track the /etc/firewalld/zones/ files in configuration management. If a zone XML sits there unmanaged, the next person who looks at the box has no way of knowing whether it was a deliberate choice or someone's debugging session that got saved.

Why This Matters

You'll run into this in a few predictable places.

Laptops and anything that roams. Every reconnect is a NetworkManager activation, and every activation reapplies the zone from the profile. Runtime zone changes on Wi-Fi don't last long. If your security model depends on "I moved the interface to drop before I connected," it lasts until the first sleep/wake cycle.

Servers managed by config management. An Ansible role that runs firewall-cmd --reload wipes out every runtime-only change since the last reload. Hand-applied emergency rules disappear at the next scheduled run, and the incident comes back. Whatever your config management writes should be the only permanent state on the box, and emergency changes should go into that config management afterward, not just into runtime.

Debugging sessions that turn into production config. The opposite mistake is just as common. Someone runs --runtime-to-permanent to "save the fix" and also saves three experimental port openings. Diff before you promote.

The habits I'd recommend:

Pair every --permanent with a --reload, or use commands like --set-default-zone that write both configs.
On NM-managed interfaces, set zones with nmcli connection modify ... connection.zone, never with firewall-cmd --change-interface.
Treat the default zone as your policy for unknown networks, and set it to something restrictive.
Bind trust to interfaces like tailscale0, not to source ranges that can be spoofed from the local segment.
Diff runtime against permanent before every reload and every promotion.
Know which zones you've forked into /etc/firewalld/zones/, and keep those files under version control.

The underlying issue is that firewalld, NetworkManager, and the kernel's nftables ruleset each hold their own piece of state, and firewall-cmd reports success while updating only one of them. Once you know which component owns which setting, persistence becomes a matter of writing each setting to its owner. If you're dealing with this across a fleet of edge boxes or mixed Linux hosts and want a second set of eyes on the design, that's the kind of infrastructure work I take on.

Invisible Text on GNOME: Fixing Qt/KDE Flatpak Theme Conflicts in Sandboxed Environments

Guatu — Mon, 05 Oct 2026 16:15:47 +0000

The app launched, the tray icon showed up, the window opened, and every label in it was dark gray on a dark gray background. Nothing crashed and nothing printed an error. The text just wasn't readable.

I ran into this with KTailctl, the Kirigami-based Tailscale GUI, installed as a Flatpak on a Fedora 43 GNOME 50 Wayland session. That combination makes it worse than a normal cosmetic bug. When a VPN client's window comes up blank, your first instinct is that the VPN is broken, not the color palette. I spent the first few minutes checking tailscale status before I noticed I could select the "invisible" text and it showed up highlighted.

That's the tell, by the way. If you drag-select across an empty-looking area and words appear, you don't have a rendering bug or a dead backend. You have a theme mismatch.

What I Expected

On a GNOME desktop with dark mode enabled, a Qt app should do one of two things:

Follow the system preference and render dark, with light text.
Ignore the preference and render in its default light theme, with dark text.

Either is fine. Both give you readable text. What I didn't expect was a third option where the app picks half of each: the background from one source and the foreground colors from another.

Flatpak's pitch is that the app brings its own runtime and doesn't care what the host has installed. For libraries, that's great. For theming, it means the app is sitting in a sandbox trying to guess what the desktop outside looks like, and it doesn't always guess consistently.

What Actually Happened

The short version: a KDE app on GNOME inside a Flatpak has two separate places it can get colors from, and they disagreed.

Here's the longer version. A Kirigami/QML app like KTailctl doesn't render with a single palette. Roughly, it has:

The Qt palette (QPalette), which comes from the Qt platform theme plugin. On GNOME, with QT_QPA_PLATFORMTHEME unset, Qt sees XDG_CURRENT_DESKTOP=GNOME and loads its built-in GNOME theme. Recent Qt 6 releases read the dark/light preference through the XDG desktop portal (org.freedesktop.appearance → color-scheme), so this side says "dark."
The KDE color scheme (KColorScheme), which Kirigami's org.kde.desktop style uses for a lot of its text and control colors. That comes from kdeglobals. Inside the sandbox, the app's config lives under ~/.var/app/<app-id>/config/, and on a GNOME machine there's usually no kdeglobals there at all. No file means KDE's defaults, and the defaults are Breeze light, which means dark text.

Put those together and you get a dark window background from the Qt palette with dark text from the KDE color scheme. Invisible.

On a Plasma desktop this never happens. Plasma ships the kde platform theme plugin and a real kdeglobals, so both sources agree because they're the same source. On GNOME outside a sandbox, distro packages often patch around it or pull in integration plugins. Inside a Flatpak you get neither. The host's GTK settings don't apply, the host's ~/.config/kdeglobals (if you even have one) isn't visible, and the only bridge to the host preference is the portal, which only one of the two color sources reads.

Which side "wins" depends on the exact runtime version. I saw it on org.kde.Platform 6.10. It isn't a new class of bug. Qt 5-era Flatpaks had the same problem, and the old fix was the QGnomePlatform theme extension, which has since been archived upstream. Qt 6 is supposed to handle GNOME natively, and for plain Qt Widgets apps it mostly does. Kirigami apps that also read KDE color schemes are where it falls apart.

Libadwaita doesn't help. GNOME's move to Libadwaita means there's less of a GTK theme for anything to imitate, and the color-scheme portal key is now the only reliable signal of what the user wants. If an app's theming stack doesn't fully consume that one key, it ends up with a split palette.

Diagnosing It Before Changing Anything

Before you start throwing environment variables at it, confirm what you're dealing with. First, which runtime is the app on:

# Which runtime and version the app is built against
flatpak info --show-runtime org.fkoehler.KTailctl
# org.kde.Platform/x86_64/6.10

# Any overrides already applied (Flatseal writes here too)
flatpak override --user --show org.fkoehler.KTailctl

Next, check what the host is actually advertising through the portal. This is what the sandboxed app sees, not what GNOME Settings shows you:

# 0 = no preference, 1 = prefer dark, 2 = prefer light
gdbus call --session \
  --dest org.freedesktop.portal.Desktop \
  --object-path /org/freedesktop/portal/desktop \
  --method org.freedesktop.portal.Settings.ReadOne \
  org.freedesktop.appearance color-scheme

If that returns <uint32 1>, the app is being told to go dark. Now look at which platform theme plugins the runtime actually ships, because you can only select ones that exist inside the sandbox:

# List platform theme plugins visible inside the sandbox
flatpak run --command=sh org.fkoehler.KTailctl -c \
  'find / -path "*platformthemes*" -name "*.so" 2>/dev/null'

Then check whether the app has a KDE config of its own:

ls -la ~/.var/app/org.fkoehler.KTailctl/config/
# no kdeglobals here = KColorScheme falls back to Breeze light

Finally, you can make Qt tell you which platform theme it actually loaded. It's noisy, but grep makes it usable:

flatpak run --env=QT_DEBUG_PLUGINS=1 org.fkoehler.KTailctl 2>&1 \
  | grep -i platformtheme

Dark mode from the portal, no kdeglobals, and the GNOME platform theme loaded: that's the split-palette setup described above.

The Fix

Whatever you do, the goal is to give the app one source of truth for colors. You can make both sides agree on dark, or make both sides agree on light. What you can't do is leave them reading from different places.

Test every option with a one-off flatpak run --env=... first. It doesn't persist anything, so a bad guess costs you nothing.

Option 1: Use the KDE platform theme and give it a color scheme

This is the fix I prefer for Kirigami apps. It makes the Qt palette and KColorScheme both come from kdeglobals, so they can't disagree.

First, test it without persisting anything:

flatpak run --env=QT_QPA_PLATFORMTHEME=kde org.fkoehler.KTailctl

If the runtime includes the kde platform theme plugin (the find above will tell you), the app should now render consistently. On GNOME that will probably be consistently light, since there's still no kdeglobals. Light and readable is already a win. If you want dark, give the app a dark scheme. The KDE runtime usually ships the Breeze color scheme files, and their format is compatible with kdeglobals:

APP=org.fkoehler.KTailctl
mkdir -p ~/.var/app/$APP/config

# Copy Breeze Dark from inside the runtime into the app's private kdeglobals
flatpak run --command=cat $APP /usr/share/color-schemes/BreezeDark.colors \
  > ~/.var/app/$APP/config/kdeglobals

If that path doesn't exist in your runtime version, run ls /usr/share/color-schemes/ inside the sandbox to see what's there. If you have Plasma color schemes installed on the host, you can also copy one over from there. Once it renders the way you want, persist the environment variable:

flatpak override --user --env=QT_QPA_PLATFORMTHEME=kde org.fkoehler.KTailctl

The tradeoff is that the app stops following GNOME's dark/light toggle. It uses whatever scheme you put in its kdeglobals. For a tray utility I open a few times a day, I'm fine with that. If you switch modes on a schedule, keep reading.

Option 2: Force a single Qt Quick style

If the app is Kirigami/QML, the split usually comes from the org.kde.desktop style pulling colors from KColorScheme. Forcing a style that only reads the Qt palette removes the second source:

# Test first
flatpak run --env=QT_QUICK_CONTROLS_STYLE=Fusion org.fkoehler.KTailctl

# Persist if it looks right
flatpak override --user --env=QT_QUICK_CONTROLS_STYLE=Fusion org.fkoehler.KTailctl

Recent Qt 6 releases make Fusion generate a dark palette when the platform reports a dark color scheme, so this keeps the portal-driven dark/light behavior. The downside is cosmetic: it won't look like Breeze. Some Kirigami components also look a bit off outside their native style. Readable and slightly plain beats native-looking and blank.

Option 3: Force the GTK platform theme

The research floating around for this issue often suggests QT_QPA_PLATFORMTHEME=gtk. For Qt 6 the plugin is named gtk3, not gtk, and it only works if the runtime actually ships it. Most of the time the find command above won't list it in the KDE runtime, and Qt quietly falls back to the default when you name a plugin that doesn't exist. That's one more silent failure on top of the first one. If your runtime does include it:

flatpak run --env=QT_QPA_PLATFORMTHEME=gtk3 org.fkoehler.KTailctl

Check with QT_DEBUG_PLUGINS=1 that it actually loaded before you trust it.

Quick fix vs. permanent fix

Approach	Persists	Scope	Undo
`flatpak run --env=VAR=value <app>`	No	That one launch	Close the app
`flatpak override --user --env=...`	Yes	Your user, that app	`flatpak override --user --reset <app>`
Flatseal "Environment" section	Yes	Same as above (writes the same override file)	Delete the entry in Flatseal
`flatpak override --user --env=...` with no app ID	Yes	Every Flatpak for your user	`flatpak override --user --reset`

Don't use the last row casually. Setting QT_QPA_PLATFORMTHEME=kde globally will also apply to Qt apps on runtimes that don't ship that plugin, and you'll be debugging a different app next week. Scope overrides to the app ID.

To undo everything and get back to a clean state:

flatpak override --user --reset org.fkoehler.KTailctl
rm ~/.var/app/org.fkoehler.KTailctl/config/kdeglobals

Why This Matters

When you'll hit this, specifically: a Qt or KDE app (especially a Kirigami/QML one) installed from Flathub, running on GNOME or another non-Plasma desktop, with dark mode on. Wayland makes it more likely, because the old X11-era hacks of reading X resources or the host's GTK settings files are even further out of reach. Every runtime bump is a chance for the behavior to change, so an app that rendered fine last month can turn invisible after a flatpak update with no change on your side.

The part that stays with me isn't the fix. It's how much time a failure like this takes when nothing tells you it's a failure. A crash gives you a stack trace. A blank window that responds to clicks gives you nothing, and it pushes you toward the wrong subsystem. With a VPN client, "the UI shows nothing" sounds a lot like "it isn't connected," so you end up debugging networking that works fine. I wrote about the same pattern in a very different context in the blog pipeline fence corruption post: output that looks plausible and is quietly wrong is worse than output that fails loudly, because none of your error detection fires.

The sandbox angle is the other lesson. Flatpak isolates the app from host libraries, which is the point. It also isolates the app from host configuration, which people forget. A theme file, a /tmp path, a GPU driver: each one is something the app assumes about its environment, and the sandbox can quietly invalidate that assumption. I hit the /tmp version of this with Claude running headless in a sandbox, and the driver version shows up with the NVIDIA container toolkit. Different layers, same shape: the host and the container each have a version of the truth, and the bug lives in the gap between them.

What I'd do differently, and what I'd tell anyone running Qt Flatpaks on GNOME:

Select-all first. Ctrl+A or a drag-select on a blank-looking window tells you in two seconds whether the content is there. Do it before you touch the network stack.
Check the portal, not the settings panel. The gdbus ReadOne call shows you what the sandbox actually receives. GNOME Settings shows you what GNOME thinks.
Pick one color source on purpose. Either the KDE platform theme plus an app-local kdeglobals, or a Qt Quick style that only reads the Qt palette. Mixing them is the bug.
Verify the plugin exists before you name it. QT_QPA_PLATFORMTHEME pointing at a missing plugin fails silently. find inside the sandbox plus QT_DEBUG_PLUGINS=1 takes thirty seconds and saves you from stacking a second silent failure on the first.
Scope overrides per app. A global override works until it hits a runtime that doesn't match, and then you're back to staring at a blank window wondering what changed.

None of this is in the Flatpak docs or the app's README, because technically nothing is broken. Each component is doing what it was designed to do. The bug only shows up when you put them together.

Trust the Tailnet, Not the Network: Automating Wi-Fi Hardening with NetworkManager Dispatchers

Guatu — Fri, 02 Oct 2026 16:15:47 +0000

A stock Fedora Workstation laptop on coffee shop Wi-Fi accepts inbound TCP and UDP on every port from 1025 to 65535, plus SSH. That's the default FedoraWorkstation zone, and every new Wi-Fi profile lands in it unless you tell NetworkManager otherwise. Tailscale shows "Connected" in the tray the whole time, which makes it easy to assume you're covered. Unless you've set an exit node, though, your DNS queries and web traffic go straight out the local access point like everyone else's.

This post is for anyone who runs Tailscale on a Linux laptop and carries it onto networks they don't control: hotels, airports, conference Wi-Fi. The idea is simple. No Wi-Fi network is trusted. The Wi-Fi is a dumb pipe, the tailnet is the security boundary, and a NetworkManager dispatcher script enforces that on every connect so you don't have to remember to.

The default state is worse than you think

Here's what an untouched Fedora Workstation looks like after joining a new network. The interface name and SSID are examples:

$ nmcli -g connection.zone connection show "CoffeeShop"
                                  # empty: falls back to the default zone
$ firewall-cmd --get-zone-of-interface=wlp0s20f3
FedoraWorkstation
$ firewall-cmd --zone=FedoraWorkstation --list-all
FedoraWorkstation (active)
  target: default
  interfaces: wlp0s20f3
  services: dhcpv6-client samba-client ssh
  ports: 1025-65535/udp 1025-65535/tcp

An empty connection.zone means "use whatever the default zone is." On Workstation, the default zone was designed so local dev servers and casual LAN sharing just work. That's a reasonable desktop default at home. On a network shared with 200 strangers it means any dev server you left running on 0.0.0.0:8000, any Jupyter kernel and any debug port is reachable by everyone on the same access point.

Tailscale doesn't change any of this. It adds a tailscale0 interface and routes 100.64.0.0/10 (plus whatever subnet routes you accept) through it. Everything else, including your DNS lookups unless MagicDNS is answering them, still exits via wlp0s20f3. If you've read my post on Tailscale subnet routers, this is the other half of that story: the subnet router gets you into your LAN, but it does nothing for traffic that isn't headed there.

What I tried first

Per-SSID allowlists

The first instinct is to classify networks: home is trusted, everything else gets the strict zone. You set connection.zone on each profile by hand.

This falls apart for two reasons. The first is that the set of "everything else" is unbounded. Every new hotel, every airport lounge, every "Guest-5G" creates a new profile, and each new profile inherits the permissive default zone. One forgotten profile and you're wide open. That's configuration drift with a nasty failure mode, because nothing tells you it happened.

The second is that it's backwards. An allowlist of safe networks means the default for unknown networks is unsafe. You want the inverse: everything is hostile, and the exceptions are explicit and small.

Runtime-only fixes

The other trap is fixing things live and calling it done:

sudo firewall-cmd --zone=public --change-interface=wlp0s20f3
sudo tailscale set --exit-node=exit-a

Both of these work immediately. The firewall change is runtime-only, so it evaporates on the next reconnect when NetworkManager reapplies the profile's zone (empty, so default). The exit node setting does persist in Tailscale's prefs, which sounds good until you get home and wonder why your LAN printer vanished and every request is hairpinning through a remote node. You end up toggling it by hand, and then forgetting to toggle it back on the next trip.

Tailscale's built-in auto exit node

Recent Tailscale releases have tailscale exit-node suggest and --exit-node=auto:any. These pick an exit node for you, optimized for latency. What they don't do is know anything about the network you're on. There's no "use an exit node only on untrusted Wi-Fi" setting, and as far as I can tell SSID-aware routing still lives in long-running feature requests on GitHub rather than in the client. So the policy decision has to happen locally, and NetworkManager is the thing that knows when you joined a network.

Trusting the CGNAT range

For the "allow SSH from my tailnet" half, a common suggestion is:

# Don't do this
sudo firewall-cmd --permanent --zone=trusted --add-source=100.64.0.0/10

Firewalld source bindings match on source address regardless of which interface the packet arrived on, and source zones take precedence over interface zones. A host on the same Wi-Fi can put 100.64.1.1 in the source field of a packet and send it to your wlp0s20f3 address. The return path for a TCP handshake would go out tailscale0, which limits what an attacker gets, but UDP services and anything stateless are fair game, and "the routing table will probably save me" isn't a security model.

Tailscale's default netfilter mode on Linux does install a rule dropping 100.64.0.0/10 sources that don't arrive on tailscale0. That's good defense in depth. I still don't want my firewall policy to be correct only because another daemon happened to insert a rule I didn't write. Bind to the interface instead.

The actual solution

There are three pieces: two firewalld zones, a dispatcher script, and a tiny opt-out list.

Step 1: A zone for hostile Wi-Fi

Fedora's built-in public zone is the usual recommendation, but on Fedora it still allows ssh and mdns. I'd rather not expose sshd on an airport network, so I build a dedicated zone with almost nothing in it:

sudo firewall-cmd --permanent --new-zone=hostile-wifi
sudo firewall-cmd --permanent --zone=hostile-wifi --add-service=dhcpv6-client
sudo firewall-cmd --reload
sudo firewall-cmd --zone=hostile-wifi --list-all

The zone target stays at default, which rejects unsolicited inbound traffic. Established and related return traffic is accepted by firewalld's base rules, so outbound browsing works normally. DHCPv4 doesn't need a service entry because NetworkManager's DHCP client uses a raw socket. If you'd rather drop silently than reject, --set-target=DROP works, though I'd test IPv6 on a few networks before committing to it.

Step 2: A zone for the tailnet, bound to the interface

sudo firewall-cmd --permanent --new-zone=tailnet
sudo firewall-cmd --permanent --zone=tailnet --add-interface=tailscale0
sudo firewall-cmd --permanent --zone=tailnet --add-service=ssh
sudo firewall-cmd --reload

NetworkManager doesn't manage tailscale0, so a permanent firewalld interface binding sticks here (unlike Wi-Fi interfaces, where NM owns the zone assignment). Now SSH is reachable only on packets that actually arrived through the WireGuard tunnel, which means they were authenticated by a tailnet peer key. That's the whole point: the trust decision is cryptographic, not based on an address someone can type into a packet. Pair it with Tailscale ACLs to decide which tailnet peers can reach port 22, the same least-privilege thinking I use for Kubernetes service accounts.

Step 3: The dispatcher script

NetworkManager runs every executable in /etc/NetworkManager/dispatcher.d/ on network events, in alphabetical order, passing the interface as $1 and the action as $2. I care about two actions: up (a connection activated) and connectivity-change (NM's connectivity check moved between states like portal and full).

The script is split into three parts here so each stays readable. First, the setup and filtering:

#!/usr/bin/env bash
# /etc/NetworkManager/dispatcher.d/50-wifi-safe
# Every Wi-Fi profile is hostile unless its UUID is explicitly listed.
set -u

ACTION="${2:-}"
EXIT_NODE="exit-a"                            # MagicDNS name or 100.x IP of your exit node
HOSTILE_ZONE="hostile-wifi"
TRUSTED_LIST="/etc/wifi-safe/trusted-uuids"   # one connection UUID per line

log() { logger -t wifi-safe -- "$*"; }

case "$ACTION" in
  up|connectivity-change) ;;
  *) exit 0 ;;
esac

# connectivity-change isn't tied to an interface, so find the active Wi-Fi profile ourselves
read -r UUID DEV < <(nmcli -t -f TYPE,UUID,DEVICE connection show --active \
  | awk -F: '$1=="802-11-wireless" {print $2, $3; exit}')
[ -z "${UUID:-}" ] && exit 0

I deliberately don't use set -e. A dispatcher script that dies halfway through on a non-zero grep leaves the machine in a half-hardened state with no log line explaining why. I'd rather check each step and log it.

Second, the trust check and zone pinning:

if [ -f "$TRUSTED_LIST" ] && grep -qxF "$UUID" "$TRUSTED_LIST"; then
  log "trusted profile $UUID on $DEV, clearing exit node"
  tailscale set --exit-node= || log "failed to clear exit node"
  exit 0
fi

# Persist the zone on the profile so reconnects and reboots keep it...
if [ "$(nmcli -g connection.zone connection show "$UUID")" != "$HOSTILE_ZONE" ]; then
  nmcli connection modify "$UUID" connection.zone "$HOSTILE_ZONE" \
    && log "pinned $UUID to zone $HOSTILE_ZONE"
fi

# ...and apply it now, because 'modify' doesn't touch the live device
firewall-cmd --zone="$HOSTILE_ZONE" --change-interface="$DEV" >/dev/null \
  || log "firewall-cmd failed for $DEV, interface may be in the default zone"

The two-step zone handling is the runtime-vs-permanent fix from earlier. nmcli connection modify writes the zone into the profile, so the next activation of that network comes up hostile before the dispatcher even runs. firewall-cmd --change-interface covers the current session. On the first connection to a brand-new network, there's a short window where the interface sits in the default zone before this runs. More on that in the lessons section.

Third, the captive portal check and the exit node:

state=$(nmcli -t networking connectivity)
if [ "$state" != "full" ]; then
  log "connectivity=$state on $UUID, deferring exit node"
  exit 0      # a later connectivity-change event will re-run us
fi

backend=$(tailscale status --json 2>/dev/null | jq -r '.BackendState')
if [ "$backend" != "Running" ]; then
  log "tailscaled state=$backend on $UUID: NOT PROTECTED"
  exit 1
fi

if [ -z "$(tailscale status --json | jq -r '.ExitNodeStatus.ID // empty')" ]; then
  tailscale set --exit-node="$EXIT_NODE" --exit-node-allow-lan-access=false \
    && log "exit node $EXIT_NODE enabled for $UUID" \
    || log "FAILED to set exit node for $UUID"
fi

The captive portal check matters more than it looks. If you force all traffic through an exit node while the hotel network is still intercepting HTTP for its login page, the portal never loads. Your browser's request goes through the tunnel, reaches the real internet, and you sit there with "Connected, no internet" forever. NetworkManager's connectivity check reports portal in that state. The script backs off, you log in, NM's next check flips to full, a connectivity-change event fires, and the script runs again and sets the exit node.

The script needs jq. You could parse tailscale status text output instead, but the text format isn't a stable interface and the JSON one is close enough to it.

Step 4: Install it with the right ownership

NetworkManager is picky about dispatcher scripts, and it's quiet about it. The script must be owned by root, executable, and not writable by group or others. If any of that is wrong, the dispatcher skips it and the only trace is at debug log level.

sudo install -o root -g root -m 0755 50-wifi-safe /etc/NetworkManager/dispatcher.d/
sudo restorecon -v /etc/NetworkManager/dispatcher.d/50-wifi-safe   # fix SELinux label on Fedora
systemctl is-enabled NetworkManager-dispatcher.service

# Opt out your home network by UUID, not by SSID
sudo mkdir -p /etc/wifi-safe
nmcli -g connection.uuid connection show "HomeWifi" | sudo tee -a /etc/wifi-safe/trusted-uuids

The restorecon line is the one people skip. If you write the script in your home directory and mv it into place, it keeps the user_home_t label, and SELinux will stop the dispatcher from executing it. install creates a new file, which usually gets the right label from the directory, but running restorecon costs nothing. If the script runs but tailscale or firewall-cmd calls fail inside it, check sudo ausearch -m avc -ts recent for denials before you start rewriting logic.

Step 5: Verify, and know where to look

nmcli connection down "CoffeeShop" && nmcli connection up "CoffeeShop"
journalctl -t wifi-safe -f                      # our own log lines
journalctl -u NetworkManager-dispatcher -f      # whether NM ran the script at all
firewall-cmd --get-zone-of-interface=wlp0s20f3  # expect: hostile-wifi
tailscale status --json | jq '.ExitNodeStatus'  # expect: non-null, Online: true

If journalctl -t wifi-safe shows nothing at all, the script didn't run, and the problem is ownership, permissions, SELinux, or the dispatcher service. If it shows lines but the state is wrong, the problem is inside the script. Splitting the logs this way cuts the debugging space in half on the first look.

Why it works

The design rests on moving trust from the network to the tunnel. Three properties make that hold up.

Deny by default, opt out by exception. Every Wi-Fi profile that isn't on a short, explicit list gets the hostile zone and the exit node. A new network can't slip through because you forgot to configure it. Forgetting is the safe outcome. The worst case of a missing trusted entry is that your home traffic takes a detour through the exit node, which is annoying and harmless.

Interface-bound policy instead of address-bound policy. Packets on tailscale0 have already been decrypted by WireGuard using a peer's key, so the interface itself carries an authentication guarantee that a source address doesn't. A --add-source=100.64.0.0/10 rule asks "does this packet claim to be from the tailnet?" An --add-interface=tailscale0 rule asks "did this packet come out of the tunnel?" Only the second question has an answer an attacker on the same Wi-Fi can't forge.

Event-driven reconciliation, idempotently. The script doesn't assume anything about previous state. It checks the profile's zone and the exit node status each time and only changes what's wrong. That's the same shape as a Kubernetes controller: observe, compare, act. It's why I'm comfortable having it fire on every up and every connectivity-change without worrying about it stacking up side effects.

The exit node closes the egress half. With --exit-node set, Tailscale installs a default route through tailscale0 and the local network only sees encrypted WireGuard UDP to the exit node (or a DERP relay). DNS goes along for the ride: with --accept-dns on, the system resolver points at Tailscale's 100.100.100.100, and non-tailnet lookups get forwarded via the exit node rather than handed to whatever resolver the coffee shop's DHCP offered. Setting --exit-node-allow-lan-access=false matters on hostile networks, because the alternative carves the local subnet back out of the tunnel and lets you talk to (and be talked to by) your neighbors.

When the exit node itself goes offline, my understanding is that the Tailscale client keeps the route pinned and traffic stops instead of silently falling back to the local network. That's fail-closed behavior, and it's the right call for this setup, but it means your exit node is now a dependency for internet access whenever you travel. I run mine on always-on hardware at home and I'd suggest the same. A second exit node in a different location is cheap insurance.

Lessons learned

There's a leak window, and you should know its size. Between Wi-Fi association and the dispatcher completing, the machine is on the network with whatever zone the profile had, and egress isn't tunneled yet. For a known profile, the zone is already pinned from last time, so it's only the egress window. For a brand-new network, the first few seconds are in the default zone. If that matters for your threat model, change the default zone itself (firewall-cmd --set-default-zone=hostile-wifi) and put your home profile in a more permissive zone explicitly. That flips the defaults at the firewall layer too, and it's what I'd do on a laptop that travels a lot.

Captive portals expire mid-session. Hotel networks commonly re-prompt after 24 hours. With the exit node already active, the portal can't intercept you, so you'll see a dead connection instead of a login page. The fix is manual: tailscale set --exit-node=, log in, and let the next connectivity-change re-enable it. I haven't found a clean way to automate the reverse transition without the script turning the exit node off whenever connectivity flickers, which would defeat the purpose.

UUIDs are better than SSIDs, but they aren't magic. I key the trusted list on the NM connection UUID so that a network called "HomeWifi" in a hotel lobby is a new profile, not a trusted one. That stops casual collisions. It doesn't stop an evil twin that clones your home SSID and knows your WPA2-PSK password, because NM will happily auto-connect that to your existing profile. If your home network uses WPA3-SAE or 802.1X, that's a much harder attack. If it's a PSK you've handed to every guest for five years, treat the trusted list as a convenience feature, not a security control.

Keep dispatcher scripts fast. The dispatcher runs scripts sequentially, so a script blocked on a hung tailscale call delays every script after it and every subsequent event, and the dispatcher will eventually time out and kill it. Everything in this script is a local socket or D-Bus call. If you're tempted to add a curl to verify your public IP from inside the dispatcher, put it in a separate systemd unit triggered by the script instead.

Silent automation needs a heartbeat. The failure mode of a broken dispatcher is identical to the failure mode of no dispatcher: nothing happens and nobody tells you. The NOT PROTECTED log line is there so a desktop notification hook or a quick journalctl -t wifi-safe -p warning check has something to grep for. If you already push laptop metrics anywhere, a "seconds since last wifi-safe success on an untrusted network" gauge is a decent alert, with the same don't-page-on-flaps discipline I wrote about in Prometheus alerting rules.

Test the permission failure on purpose. Before you trust this on a trip, chmod 0775 the script, reconnect, and confirm that journalctl -u NetworkManager-dispatcher tells you something (or doesn't). Knowing what a broken install looks like on your own machine beats discovering it at the gate.

The bigger shift for me was conceptual. Once the tailnet is the boundary, the Wi-Fi stops being something you evaluate and becomes something you route through. You don't decide whether the airport network is safe. You assume it isn't, every time, and let a fifty-line script make that assumption hold. If you're running Tailscale inside containers rather than on a laptop, the kernel TUN in unprivileged LXC post covers the same interface-binding idea from the server side, and if you want help applying this kind of zero-trust pattern across a fleet, that's the sort of work I take on through GuatuLabs.

Claude Headless Sandboxing: Why /tmp Writes Fail in Non-Interactive Mode

Guatu — Wed, 30 Sep 2026 16:15:47 +0000

A scheduled Claude Code job that writes its report to /tmp/agent_log.md works every time you run it by hand. Under a systemd timer with claude -p, it exits 0, the journal shows no errors, and the file is either missing or empty.

That's the whole symptom. There's no stack trace to search for, which is why this one costs people an afternoon.

I hit this with my second-brain curator. It's a shell script on a timer that points Claude at a pile of session notes, has it pick out what's worth promoting into long-term memory, and writes a curation report. That's the pipeline behind the self-updating wiki setup. The fix itself was small: move the logs out of /tmp and into $HOME/logs. The reason the fix works is the interesting part. It says a lot about how headless agents see the filesystem, and about how interactive mode hides the permission model from you.

What I expected

My mental model was plain Unix. The script runs as my user, /tmp is world-writable with the sticky bit, so any process running as me can create files there. Claude Code is a process running as me. Therefore Claude Code can write to /tmp.

Interactive sessions supported that model. I'd ask Claude to "write the summary to /tmp/curator-test.md" and it would. The file showed up and I moved on.

The broken version looked something like this:

#!/usr/bin/env bash
# curator.sh (the version that silently failed under systemd)
set -euo pipefail

claude -p "Review the notes in ./inbox. Write a curation report \
to /tmp/agent_log.md listing which notes to promote and why." \
  --allowedTools "Read,Glob,Grep,Write"

echo "curator finished"

Run from a terminal, this worked, and the reason is the whole problem.

What actually happened

Two separate mechanisms were stacked on top of each other, and neither of them has anything to do with Unix permissions on /tmp.

1. Claude Code's permission boundary is the working directory, not your UID

Claude Code doesn't scope file access by what your user account can touch. It scopes it by working directories: the directory you launched from, plus anything you explicitly add. File edits inside that boundary follow your permission mode. File edits outside it need approval.

In an interactive session you rarely notice, because when Claude tries to write /tmp/curator-test.md you get a permission prompt, hit "yes," and forget it happened. Maybe you picked "don't ask again" and it went into your local settings, and now you've forgotten that too.

In print mode (-p) nobody is there to answer the prompt. The tool call gets denied, and the model is told the write wasn't allowed. Then the model does what models do: it adapts. Sometimes it prints the report to stdout instead. Sometimes it writes a note saying it couldn't save the file. Sometimes it just summarizes what it would have written. The process finishes normally and exits 0.

So from the script's point of view, nothing failed. An agent that works around a denied permission looks like success to a shell script. That's the gap.

Putting Write in --allowedTools doesn't get you past the boundary either. Allowing the tool isn't the same as allowing the tool to reach any path. The directory scoping still applies.

2. systemd has its own opinions about /tmp

Even after fixing the permission boundary, /tmp under systemd is its own trap. If the service unit (or a drop-in you've forgotten about) sets PrivateTmp=yes, the service gets a private /tmp namespace. Your script writes /tmp/agent_log.md, it lands in a directory like /tmp/systemd-private-<hash>-curator.service-<id>/tmp/, and when you cat /tmp/agent_log.md from your own shell afterward, there's nothing there. The file existed. It just existed somewhere you weren't looking, and it gets cleaned up when the service stops.

Then there's aging. /tmp is subject to systemd-tmpfiles cleanup, which I've already been burned by in a different context (see the Plex EAC3 transcoding post). Logs you want to inspect next week shouldn't live somewhere designed to forget things.

Put those together and /tmp is about the worst place to put the output of an unattended agent. The agent's sandbox doesn't consider it in scope, the service manager may give you a private copy, and the OS will clean it up for you.

The fix

The fix has three parts: give the output a permanent home inside a directory Claude is allowed to write, make the calling shell own the log capture, and make denials visible.

Put output in a directory you explicitly grant

#!/usr/bin/env bash
set -euo pipefail

LOG_DIR="$HOME/logs"
mkdir -p "$LOG_DIR"
STAMP="$(date +%Y%m%d-%H%M%S)"
REPORT="$LOG_DIR/sb-curator-out-$STAMP.md"

cd "$HOME/second-brain"   # working dir = what the agent may edit

claude -p "Review the notes in ./inbox. Write a curation report \
to $REPORT listing which notes to promote and why." \
  --add-dir "$LOG_DIR" \
  --allowedTools "Read,Glob,Grep,Write" \
  --permission-mode acceptEdits

The important pieces:

cd first. The working directory is the permission boundary. In a systemd unit, the default working directory is / unless you set WorkingDirectory=. Launching an agent with / as its project root is a bad idea for reasons that go well beyond logging.
--add-dir extends the boundary to exactly one more directory. That's better than making the boundary bigger.
--permission-mode acceptEdits lets file edits inside the granted directories go through without a prompt. Nothing can answer a prompt in headless mode anyway, so the only real question is whether writes inside the boundary are pre-approved or always denied.

To make the grant persistent instead of passing it on every call, put it in the project's .claude/settings.json:

{
  "permissions": {
    "additionalDirectories": ["~/logs"],
    "allow": [
      "Read",
      "Edit(~/logs/**)"
    ]
  }
}

There's a path-syntax trap here too. In Claude Code permission rules, /logs/** does not mean an absolute path. A single leading slash is resolved relative to the settings file. For a real absolute path you need //tmp/** (double slash), or ~/ for your home directory. I've seen people write Edit(/tmp/**), watch it do nothing, and decide the permission system is broken. It works fine, it just reads the path differently than you'd expect.

Let the shell own the log, not the agent

The second change matters more. Stop asking the agent to write its own run log. The agent writes the artifact (the curation report). The shell captures the transcript:

claude -p "$PROMPT" \
  --add-dir "$LOG_DIR" \
  --allowedTools "Read,Glob,Grep,Write" \
  --permission-mode acceptEdits \
  --output-format stream-json --verbose \
  2>&1 | tee "$LOG_DIR/sb-curator-run-$STAMP.jsonl"

A shell redirect or tee runs as your user with normal Unix semantics. Claude's permission boundary doesn't apply to it at all, because Claude isn't the one writing. Now the transcript always exists, even when the agent's own writes get denied, and that's exactly the case where you need it.

The stream-json flag change

That --verbose flag isn't decoration. On current 2.x releases, asking for streaming JSON in print mode without it fails right away:

# Fails: stream-json in print mode now requires --verbose
claude -p "summarize ./inbox" --output-format stream-json
# Error: When using --print, --output-format=stream-json requires --verbose

# Works
claude -p "summarize ./inbox" --output-format stream-json --verbose

This one does fail loudly, so at least it's honest. It still breaks any older script that set up clean machine-readable logs before the requirement existed. If your pipeline stopped producing JSONL after an upgrade, check this first. The research notes I keep point to around v2.1.76 as where it started showing up for people, so check against whatever version you've pinned.

Make denials visible

This is where headless agents are weak on observability. A denied permission isn't an error exit. You have to go looking for it.

With --output-format json or stream-json, the final result message includes a permission_denials array. If it isn't empty, the agent tried something it wasn't allowed to do. That's worth an alert, because it means either your grants are wrong or the agent is going somewhere you didn't expect. Both deserve a look.

RUN_LOG="$LOG_DIR/sb-curator-run-$STAMP.jsonl"

# Last line of stream-json is the result message
DENIALS=$(tail -n 1 "$RUN_LOG" | jq '.permission_denials | length' || true)
DENIALS=${DENIALS:-0}

if [ "$DENIALS" -gt 0 ]; then
  echo "curator: $DENIALS permission denial(s), see $RUN_LOG" >&2
  exit 2   # make systemd mark the unit failed
fi

# Belt and braces: the artifact must exist and be non-empty
[ -s "$REPORT" ] || { echo "curator: report missing" >&2; exit 3; }

Now the timer unit goes red when the agent quietly works around a denial. Before, it just shrugged and exited 0.

The grep exit code detour

Adding checks like these to the curator exposed a second, unrelated bash bug that's common enough to call out. The script counted flagged items in the report:

# Broken: grep -c prints "0" AND exits 1 when there are no matches
FLAGGED=$(grep -ciE 'promote|stale' "$REPORT" || echo 0)
# On zero matches, FLAGGED becomes "0\n0"
# [ "$FLAGGED" -gt 0 ] -> "integer expression expected"

grep -c with no matches prints 0 and exits with status 1. The || echo 0 fallback fires too, so the variable ends up holding two lines. The fix is to swallow the exit code without adding extra output, then default the empty case (for example, when the file doesn't exist):

FLAGGED=$(grep -ciE 'promote|stale' "$REPORT" || true)
FLAGGED=${FLAGGED:-0}

Under set -euo pipefail you need the || true. Otherwise a zero-match grep kills the script at the assignment, which is its own confusing failure.

The systemd side

Last piece: make the unit explicit instead of relying on defaults.

[Service]
Type=oneshot
User=curator
WorkingDirectory=/home/curator/second-brain
ExecStart=/home/curator/bin/curator.sh
# Output lives in $HOME/logs, so PrivateTmp can stay on
PrivateTmp=yes

Once nothing important lives in /tmp, you can leave PrivateTmp=yes on. It's a good hardening default, and it stops being a trap once you aren't depending on /tmp for state.

Why this matters

This is one specific case of a general problem: interactive mode is a different runtime environment than headless mode, and it looks the same only because a human is quietly approving things. Every "yes" you click in a terminal session is an undocumented dependency. The first time the job runs under cron, systemd, CI, or a Kubernetes CronJob, those dependencies disappear at once.

Here's when you'll hit it:

Promoting a prompt from "works in my terminal" to "runs on a timer." This is the most common path. Prototyping is interactive, production is headless, and nothing tells you about the gap.
Moving scheduled LLM jobs between backends or hosts. When I moved scheduled curation work toward local models, the orchestration moved around too. Every move is another chance for the working directory or temp namespace to change under you.
Any agent that writes to shared scratch space. /tmp feels neutral and safe. For an agent it's outside the project boundary, often namespaced, and actively cleaned up.

Here's how I avoid it now:

Every headless agent run gets an explicit working directory and explicit extra directories. No defaults. If I can't list the paths the agent needs to write, I don't understand the job well enough to schedule it.
The agent writes artifacts; the shell writes logs. A transcript captured by tee survives the agent failing, being denied, or improvising. A log the agent writes itself doesn't.
Denials fail the run. permission_denials being non-empty is a non-zero exit. A headless agent that quietly works around a missing permission is worse than one that crashes, because the crash at least shows up in systemctl --failed.
Check that the artifact exists, not just the exit code. [ -s "$REPORT" ] is one line and catches most of the ways this goes wrong.
Nothing that matters lives in /tmp. It's for scratch data you're happy to lose, and "the only record of what my agent did last night" isn't that.

There's also a memory-design angle. In the six-layer memory setup I run for Claude Code, the filesystem layer is the one that assumes the agent can write where it's told to. Headless mode tests that assumption hard. It's the same lesson as promotion pipelines for agent memory: deciding where an agent may write, and proving the write happened, is the hard engineering. The prompt is the easy part.

The sandbox is doing its job here. Scoping an unattended agent's writes to a known set of directories is exactly what you want in production, and I'd much rather have that default than an agent that can scribble anywhere my UID can reach. The failure mode is that the scoping is invisible until you remove the human, and then the only sign is an empty file. Make the boundaries explicit in the command line, capture everything from outside the agent, and treat "the agent adapted around a denial" as the failure it is.

If you're taking agent pipelines from prototype to unattended production and want a second set of eyes on the operational side, that's the kind of work I help teams with.

Etcd Defragmentation: Reclaiming Space Without Cluster Disruption

Guatu — Mon, 28 Sep 2026 16:15:49 +0000

My etcd database was 508 MB on disk and only 84 MB of it held live data. The other 400-odd megabytes were free pages that etcd had no intention of handing back to the filesystem. Defragmenting brought it back down to 84 MB. That part took seconds. Doing it without the control plane noticing took more care.

If you run your own control plane (kubeadm on bare metal, Proxmox VMs, anything where nobody else looks after etcd for you), this is your job. Managed Kubernetes providers do it in the background. On a homelab or on-prem cluster, nobody does it unless you set it up, and nothing tells you it's needed until you hit a quota alarm or start seeing odd latency.

Why etcd grows and never shrinks

etcd stores data in bbolt, a B+tree in a single memory-mapped file. Every write to a Kubernetes object creates a new revision. Lease renewals, Endpoints updates, Event objects and controller status patches all add revisions. The kube-apiserver compacts old revisions every 5 minutes by default (--etcd-compaction-interval), so history doesn't pile up forever.

The catch is that compaction only marks pages as free inside the bbolt file. The file stays the same size. bbolt reuses those free pages for new writes, so the file doesn't grow without limit, but after a burst of churn (a big rollout, a namespace deletion, a controller stuck in a status-update loop) the file keeps its high-water mark.

That causes three problems:

The quota counts file size, not live data. --quota-backend-bytes defaults to 2 GiB. When the file hits that limit, etcd raises a NOSPACE alarm and goes read-only for writes. The apiserver starts rejecting creates and updates, even if 80% of the file is empty pages.
Snapshots copy the whole file. A snapshot save of a fragmented database writes a fragmented snapshot, so bloat gets multiplied by however many backups you keep.
Bigger mmap, slower startup. A member that has to restart and load a bloated database takes longer to rejoin.

Defragmentation rewrites the bbolt file with only the live pages and releases the rest back to the filesystem. It's the only way to shrink the file short of restoring from a snapshot.

Know when it's actually needed

Don't defrag on a timer just because you can. Defrag when the gap between on-disk size and in-use size is large. etcdctl endpoint status reports both in its JSON output:

export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
ENDPOINTS="https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379"

etcdctl --endpoints="$ENDPOINTS" endpoint status -w json | jq -r '.[] |
  "\(.Endpoint)  total=\(.Status.dbSize / 1048576 | floor)MiB  " +
  "inuse=\(.Status.dbSizeInUse / 1048576 | floor)MiB  " +
  "ratio=\((.Status.dbSizeInUse / .Status.dbSize * 100) | floor)%  " +
  "leader=\(.Status.leader == .Status.header.member_id)"'

Before my defrag, the output looked like this (endpoints anonymized):

https://10.0.0.11:2379  total=508MiB  inuse=84MiB  ratio=16%  leader=true
https://10.0.0.12:2379  total=497MiB  inuse=84MiB  ratio=16%  leader=false
https://10.0.0.13:2379  total=502MiB  inuse=84MiB  ratio=16%  leader=false

The in-use size is almost identical across members because they all hold the same data. The on-disk size differs a bit because each member's bbolt file fragments on its own. A ratio of 16% means five-sixths of the file is dead space.

My rule of thumb: act when the in-use ratio drops below 50% and the total is large enough to matter (more than about 100 MiB). A 40 MiB database at 45% isn't worth a maintenance window. The research brief's version of this check used "DB size vs. total keys", which works as a rough signal, but dbSizeInUse measures the thing you care about directly.

The zero-disruption part: one member at a time, followers first

Defragmentation blocks the member it's running on. While bbolt rewrites the file, that member can't serve reads or writes. On a small database this takes a second or two. On a multi-gigabyte one it can take much longer. The etcd and Kubernetes docs both say the same thing: defrag one member at a time, and never touch more than one at once.

Two things follow from that:

Don't use etcdctl defrag --cluster. It goes through every member in the member list back to back, with no pause and no health check between them. If one member is slow to recover, the next one is already blocked, and in a three-member cluster two members down means no quorum.
Do the leader last. A defrag on a follower barely matters: the leader keeps committing with the other follower, and the defragged follower catches up afterwards. A long defrag on the leader can stall it past the followers' election timeout (1000 ms by default). That triggers an election, and every in-flight apiserver request to etcd waits on it.

Here's the script I run from a control plane node. It finds the leader, defrags each follower, checks health after each one, and handles the leader last:

#!/usr/bin/env bash
set -euo pipefail
source /etc/etcd-maint.env   # exports ETCDCTL_* certs and ENDPOINTS

leader=$(etcdctl --endpoints="$ENDPOINTS" endpoint status -w json \
  | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint')
[[ -n "$leader" ]] || { echo "no leader found, aborting"; exit 1; }

followers=$(tr ',' '\n' <<<"$ENDPOINTS" | grep -vxF "$leader")

for ep in $followers "$leader"; do
  echo "== defrag $ep"
  etcdctl --endpoints="$ep" --command-timeout=120s defrag
  # Every member must be healthy before we touch the next one
  etcdctl --endpoints="$ENDPOINTS" endpoint health
  sleep 30
done

etcdctl --endpoints="$ENDPOINTS" alarm list

Some notes on the choices:

--command-timeout=120s matters. The default is 5 seconds, and on a large database the client gives up while the server is still defragmenting. You get an error, the defrag finishes anyway, and you're left unsure what state things are in.
The health check runs against all endpoints, not just the one you defragged. Under set -e, one unhealthy member stops the script before the next defrag starts. That's the whole safety mechanism, so don't remove it.
The 30-second sleep lets the defragged member catch up on raft entries it missed. It isn't strictly required, but it's cheap.
alarm list at the end catches a leftover NOSPACE alarm (more on that below).

On a kubeadm cluster, if you don't want to install etcdctl on the host, you can kubectl exec into each etcd-<node> static pod and run the same commands. The certificate paths are the same because kubeadm mounts /etc/kubernetes/pki/etcd into the pod.

Handling the leader: move it or accept the election

If the leader stalls, leadership can move somewhere else, and that's what happened in my maintenance window. After I defragged the leader, a different node held leadership and the raft term had gone up. Nothing broke. The apiserver retried and the cluster stayed healthy. It still made me want to check afterwards whether the cluster had settled or was flapping.

There's a cleaner option: move leadership off the node yourself, then defrag it as a follower.

# Pick a follower that's already been defragged as the new leader
new_leader_id=$(etcdctl --endpoints="https://10.0.0.12:2379" endpoint status -w json \
  | jq -r '.[0].Status.header.member_id | tostring')

# move-leader has to be sent to the current leader's endpoint
etcdctl --endpoints="$leader" move-leader "$(printf '%x' "$new_leader_id")"

move-leader takes the member ID in hex, which is the format etcdctl member list prints. The JSON output gives it in decimal, hence the printf. There's also a jq trap here: member IDs are 64-bit, and older jq versions convert large integers to doubles and lose precision. If the hex ID looks wrong, take it straight from etcdctl member list -w table rather than trusting the conversion.

Once leadership has moved, the old leader is a follower, and your defrag loop treats it like any other follower. There's no surprise election.

Whichever route you take, confirm the cluster is stable afterwards:

# Run twice, a minute apart. Same leader and same raft term = settled.
etcdctl --endpoints="$ENDPOINTS" endpoint status -w json | jq -r '.[] |
  "\(.Endpoint) leader=\(.Status.leader == .Status.header.member_id) term=\(.Status.raftTerm)"'

kubectl get --raw='/readyz?verbose' | grep etcd

If the raft term keeps rising between the two runs, leadership is still bouncing. Stop and investigate (disk latency is the usual cause) before you blame the defrag.

The NOSPACE alarm doesn't clear itself

If you're defragging because you already hit the quota, there's one more step. Defragmentation frees the space, but the NOSPACE alarm stays raised until you disarm it:

etcdctl --endpoints="$ENDPOINTS" alarm list
# memberID:1234567890 alarm:NOSPACE

etcdctl --endpoints="$ENDPOINTS" alarm disarm
etcdctl --endpoints="$ENDPOINTS" alarm list   # should print nothing

Until you disarm it, etcd keeps refusing writes even though the file is now a fraction of its old size. This is the step people miss when they defrag during an outage and then wonder why kubectl apply still fails.

The side effect nobody mentions: snapshot sprawl

This is the part the research brief called the "total cost of maintenance", and it's the lesson I'd pass on first. You should take a snapshot before any defrag. Defrag has been reliable in my experience, but "reliable" isn't a backup strategy. On a control plane with a small boot disk (mine are in the 30 GB range), those snapshots add up quickly, especially snapshots of a fragmented database. You fix etcd bloat and get DiskPressure on the control plane node instead, and the kubelet starts evicting things.

The fix is to prune snapshots automatically, on the same schedule that creates them. A systemd timer works well:

# /etc/systemd/system/etcd-snapshot.service
[Unit]
Description=etcd snapshot with retention

[Service]
Type=oneshot
EnvironmentFile=/etc/etcd-maint.env
ExecStart=/bin/sh -c 'etcdctl --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd/etcd-$(date +%%Y%%m%%dT%%H%%M).db'
ExecStartPost=/usr/bin/find /var/backups/etcd -name "etcd-*.db" -mtime +3 -delete

# /etc/systemd/system/etcd-snapshot.timer
[Unit]
Description=Daily etcd snapshot

[Timer]
OnCalendar=*-*-* 03:15:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now etcd-snapshot.timer. Three days of retention is enough locally because the snapshots also go off-box. A snapshot that only lives on the control plane disk doesn't protect you if you lose that disk. In my setup cluster-level backups go to object storage (the Velero + MinIO setup covers that side), and the control plane VMs themselves are backed up at the hypervisor level.

A useful side effect: defragging before your snapshot window makes every snapshot after it smaller, because the snapshot size follows the bbolt file size. In my case that's 84 MB per snapshot instead of 508 MB, which changes how many days of retention a small disk can hold.

Alert on the ratio, not the size

I found my bloat after the fact, while chasing pod restarts that looked unrelated. Nothing had alerted. That's the observability gap: etcd fragmentation doesn't show up as a failing health check, just as a number drifting upward on a metric nobody watches.

etcd exposes both sizes as Prometheus metrics. The etcd-mixin (bundled with kube-prometheus) already ships a rule for this, and if you're writing your own, it looks like this:

groups:
  - name: etcd-maintenance
    rules:
      - alert: EtcdDatabaseHighFragmentation
        expr: |
          (last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m])
            / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5
          and etcd_mvcc_db_total_size_in_use_in_bytes > 104857600
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "etcd {{ $labels.instance }} is >50% free pages; schedule a defrag"

The size guard (100 MiB in use) keeps the alert quiet on small databases, where a bad ratio doesn't matter. The one-hour for rides out short-lived churn during rollouts. Severity is warning, not critical: this is "schedule maintenance this week", not "wake someone up". Pair it with a separate rule on etcd_mvcc_db_total_size_in_bytes approaching etcd_server_quota_backend_bytes, and give that one a higher severity, because it means writes are about to stop. I wrote more about keeping rules like these actionable in Prometheus Alerting Rules That Don't Cry Wolf.

If your control plane is scraped over HTTPS with client certs (kubeadm's default puts metrics on 127.0.0.1:2381 over plain HTTP), check that Prometheus can reach it before you trust the absence of an alert. A missing metric doesn't fire an alert, which is how the gap stays open.

Gotchas and alternatives I considered

Offline defrag with etcdutl. If a member is stopped (for example during a node rebuild), etcdutl defrag --data-dir /var/lib/etcd defragments the files directly without a running server. It's fine for a member that's down anyway. It's the wrong tool for a live cluster, because stopping a member to defrag it offline costs more availability than an online defrag.

Defrag doesn't compact. If compaction isn't running (a custom apiserver flag, or a standalone etcd without --auto-compaction-retention), defrag reclaims almost nothing, because the old revisions are still live data. Check that dbSizeInUse is reasonable first. If in-use is huge, the problem is compaction or a runaway writer, not fragmentation.

Find the churn source. A database that re-fragments within days means something is writing constantly. Event objects with long TTLs, a controller hot-looping on status updates, and oversized ConfigMaps rewritten on every reconcile are the usual suspects. Defrag treats the symptom. etcd_debugging_mvcc_keys_total and apiserver request metrics grouped by resource will point you at the cause.

Disk latency makes everything worse. Defrag writes a whole new copy of the database. On slow or shared storage, that write burst can push fsync latency past what raft tolerates, and then you get elections you didn't ask for. If your control plane VMs share a disk with noisy neighbors, run defrags in quiet hours.

Scheduled automatic defrag. Some people run a nightly defrag on cron. I prefer letting the alert drive it: defrag is cheap but not free, and running it unconditionally hides the churn signal you'd otherwise notice. If you do automate it, use the follower-first script with health gates, never --cluster.

Wrap-up

Defragmentation is safe when you treat it as a rolling operation: snapshot first, one member at a time, followers before the leader (or move leadership off first), health-gate every step, disarm any alarm, and then check that leadership has settled. The command itself is the easy part. The work is in the process around it, especially snapshot retention on small control plane disks, which is how cleaning up one problem turns into the next one.

Use this when the in-use ratio drops below half on a database big enough to matter, when you've hit NOSPACE, or before you take a snapshot you plan to keep for a long time. Set up the fragmentation alert so you hear about it from a warning rather than from a failing kubectl apply. If you're running control planes like this and want a second opinion on the operational side, that's the kind of work I do through GuatuLabs. For more on how the control plane VMs fit into the wider cluster, see Building a Production Homelab.

Blog Pipeline Fence Corruption: How a Naive Sed Rule Swallowed Markdown Code Blocks

Guatu — Mon, 31 Aug 2026 22:15:48 +0000

A draft came out of my blog generation pipeline with the frontmatter intact, the title correct, the word count in range, and roughly 1,400 words of prose rendered inside a single syntax-highlighted code block. Everything after the first bash example was gray monospace on a dark background. The build passed. The exit code was 0.

One sed expression caused it, and it had been sitting in the generation script since the pipeline moved from a local model to a cloud backend. It did exactly what it was told to do, which was the whole problem.

What I expected

The setup is boring and probably familiar. A shell script calls a model in non-interactive mode, captures stdout, and writes the result to src/content/posts/<slug>.md. Astro picks it up from the content collection, Zod validates the frontmatter, and npm run build renders it.

Models like to wrap their output. You ask for "a complete markdown file, output only the markdown," and a meaningful fraction of the time you get back something shaped like this:

```markdown
title: "Whatever"

Post body here.
```

That outer fence is not part of the document. Write it to disk as-is and Astro sees a code block where the frontmatter should be, Zod finds no title, and the build fails loudly. Loud failures are fine. You fix them and move on.

So the pipeline stripped the wrapper before writing. That was the intent, and the expected outcome was a clean file with the outer fence gone and every internal code block untouched.

What actually happened

Here is the rule, as it existed in the generation script:

# The "cleanup" that caused the problem
claude -p "$PROMPT" | sed 's/^```markdown$//; s/^```$//' > "$OUT"

Read the second expression on its own: s/^`$//. It replaces any line consisting of exactly three backticks with an empty line. Every line. Not the last one, not the wrapper's closer specifically, all of them.

Markdown fences are a toggle, not a container. An opening fence with an info string (bash `) turns code mode on. A bare closing fence turns it off. Delete every bare closing fence in a document and you have deleted every "off" switch while keeping every "on" switch.

Corruption follows deterministically from there. The first `bash in the post opens a code block, and that block never closes. Every heading, paragraph, table, and link after it becomes code block content. The renderer is behaving correctly. It was handed a document that says, in effect, "the rest of this file is bash."

Nothing about the file is malformed by CommonMark rules, which is why it looked like corrupted prose rather than a syntax error. An unclosed fence at end-of-document is legal and closes implicitly. The parser doesn't complain, the markdown linter doesn't complain, and Astro renders it happily into a page nobody would read past the first screenful.

That first expression, s/^markdown$//, carried a smaller version of the same flaw. It emptied any line that was exactly markdown, which is a legitimate thing to write in a post about markdown tooling. Delete that opener and you're left with an orphaned closer, which then opens a code block for the remainder of the file. Same failure, inverted.

Two things kept this from being obvious for longer than I'd like.

Local models mostly didn't trigger it. The Ollama-backed path had different post-processing and rarely emitted the markdown wrapper in the first place, which is the kind of behavioral difference you don't discover until you [move a scheduled generation job between backends](https://guatulabs.dev/posts/moving-scheduled-llm-curation-from-cloud-apis-to-local-models/). So the sed` rule was written for a problem that mostly existed on one branch of the pipeline, and it damaged output on both.

Beyond that, nothing in the pipeline had an opinion about document structure. It checked word count. It checked frontmatter fields. It ran an anti-slop word filter. Every one of those checks passes cleanly on a file whose entire body is trapped inside a code block, because none of them parse the document as a document. When the only signal your automation emits is an exit code, you get told "success" for an entire class of failures that never touches the exit code. Structural validity has to be asserted explicitly, the same way you assert that a Kubernetes manifest is valid before it merges rather than after it's applied to a live cluster.

The fix, part one: stop using substitution for a positional problem

The wrapper is a positional artifact. It lives on line 1 and on the final non-empty line, and nowhere else. A substitution rule anchored with ^...$ has no concept of position, so it was the wrong tool from the first commit.

What replaced it matches on a signature rather than a pattern, and touches exactly two lines:

unwrap_markdown_fence() {
  local file="$1"
  local l1 l2 last lastno fences

  l1=$(sed -n '1p' "$file")
  l2=$(sed -n '2p' "$file")
  lastno=$(grep -n '.' "$file" | tail -n1 | cut -d: -f1)
  last=$(sed -n "${lastno}p" "$file")

  # Signature of a whole-document wrapper:
  # line 1 is a fence, line 2 opens frontmatter, last non-empty line closes it.
  case "$l1" in
    '```markdown'|'```md'|'```') ;;
    *) return 0 ;;
  esac
  [[ "$l2" == '---' ]] || return 0
  [[ "$last" == '```' ]] || return 0

  # Even fence count means the wrapper pairs cleanly with itself.
  fences=$(grep -c '^```' "$file")
  (( fences % 2 == 0 )) || return 0

  sed -i "1d;${lastno}d" "$file"
}

Checking line 2 for --- is what makes this safe. A wrapped document always has the frontmatter delimiter immediately after the opener, because that's how every post in the collection starts. A legitimate post that happens to open with a code fence will not have --- sitting on line 2.

Every guard clause returns 0 instead of failing. If the signature doesn't match, the function does nothing and the file passes through untouched. A cleanup step should be inert when it isn't confident, not aggressive. The old rule was the opposite: maximally confident, zero context.

The fix, part two: assert structure, don't count it

Unwrapping correctly is necessary but not sufficient, because the model itself can drop a closing fence. Any repair you write can also have bugs. So the pipeline needed something that reads the finished file and asserts the body is actually prose.

My first instinct was to count fences and check for an even number. That check is close to worthless. A document with a missing closer and a stray opener has an even count and is still broken, and a post that legitimately uses four-backtick fences to demonstrate nested markdown throws the count off entirely. Counting tells you about symbols. You need to know about state.

So check-fences.sh walks the file, tracks whether it's inside a code block, and looks for content that has no business being there:

#!/usr/bin/env bash
set -euo pipefail
file="$1"

awk '
  /^```/ {
    if (!in_code) { in_code=1; open_line=NR } else { in_code=0 }
    next
  }
  # Markdown structures that should never appear inside a fence
  in_code && /^#{1,6} / {
    printf "line %d: heading inside code block opened at line %d\n", NR, open_line
    bad=1
  }
  in_code && /^\|.*\|$/ {
    printf "line %d: table row inside code block opened at line %d\n", NR, open_line
    bad=1
  }
  END {
    if (in_code) {
      printf "unclosed fence opened at line %d (EOF)\n", open_line
      bad=1
    }
    exit bad
  }
' "$file"

Heading detection is what actually catches the sed bug. A swallowed document always contains ## Something inside the runaway block, because posts have sections. Table rows catch the same failure in posts built around comparison tables. The end-of-file check catches the simpler case where the model just forgot a closer.

False positives exist and I decided to live with them. A shell script demonstrating comment syntax can contain a line starting with #, but the awk pattern requires a space after one to six hashes at column zero, and code comments in the posts I generate rarely look like a markdown heading with a capitalized sentence after it. When it does misfire, the failure mode is a rejected draft rather than a published page of gray monospace, and I'll take that trade every time.

The fix, part three: repair before you reject

A validator that only says no creates its own failure mode. The instinct when a generation step fails is to retry, but retrying an expensive model call to fix a mechanical, deterministic defect is the wrong shape of solution. Worse, the regenerated draft is a different draft. You've thrown away good prose to fix a missing three-character line, and the new attempt can trip a different gate entirely.

Corruption from a missing closer has exactly one correct repair, so repair-fences.sh performs it directly and runs before the check:

#!/usr/bin/env bash
set -euo pipefail
file="$1"

awk '
  /^```/ { in_code = !in_code; print; next }
  # A heading inside a code block means the previous fence never closed.
  in_code && /^#{1,6} / { print "```"; in_code=0; print; next }
  { print }
  END { if (in_code) print "```" }
' "$file" > "${file}.tmp" && mv "${file}.tmp" "$file"

Ordering matters here. Repair runs first, the checker runs second, and the checker is the gate. If repair does its job, the check passes and the draft survives. If repair produces something still structurally broken, the check fails and the pipeline stops with a specific line number instead of a shrug. Repair is allowed to be optimistic precisely because it isn't the thing making the final call. That separation of "attempt a fix" from "verify the fix" is the same discipline that makes plan-and-apply workflows trustworthy in infrastructure automation, and it applies just as well to a text pipeline.

One caveat worth stating plainly: a repair step that mutates content is a liability if you can't see what it did. Mine writes a line to the run log whenever it changes the file, including the line number where it inserted a closer. Silent auto-repair is how you end up debugging a bug that a script already "fixed" three stages earlier.

Where the gates fight each other

Content quality gates and structural gates want different things, and stacking them naively produces a pipeline that rejects everything.

My anti-slop checker is a hard gate. Banned words, banned phrases, an em-dash budget, a rule against consecutive paragraphs opening with the same word. It's deterministic and it's strict, which is the point. But a hard gate on prose quality combined with a hard gate on structure means a draft has to clear both in a single generation, and the retry cost is a full regeneration.

The ordering that works for me: run mechanical repairs first (fence balance, wrapper stripping, frontmatter normalization), then run the structural gate, then run the quality gate, and only regenerate on quality failures. Mechanical problems get fixed in place because they have one right answer. Prose problems get sent back to the model with the specific violations included in the retry prompt, which turns a blind retry into a targeted revision. Feeding the checker's own output back into the next attempt is what makes the loop converge instead of thrashing, and it's the same feedback-shaped design that keeps multi-layer agent memory systems from degrading over repeated passes.

Anything a script can repair deterministically should never reach the model twice. That's the rule I landed on, and the fence bug is the clearest example of why: a missing ` cost an entire draft and a model call when a five-line awk program fixes it in milliseconds.

Why this matters beyond one blog

Text transformation with line-anchored regex is a trap any time the text has stateful syntax, and markdown is full of it. Fences toggle. List indentation nests. Frontmatter delimiters and horizontal rules are the same three characters in different positions. HTML comments and YAML block scalars both swallow content until a terminator. A sed expression sees a line. Your document has a grammar, and lines don't know what mode they're in.

Two habits keep this from biting:

Match on position or signature, not on pattern, when the artifact is positional. If what you're stripping is "line 1 and the last line," write a rule that says line 1 and the last line. Anchors are not position.
Assert the shape of the output, not just the presence of the output. Word count, file existence, and frontmatter fields all confirm that something landed on disk. None of them confirm it's a document. A structural check that understands the format is the only thing that closes the gap.

The broader pattern generalizes past blog tooling. Any pipeline that pipes model output through shell transformations and into a build system has this exposure, and the failure is quiet by construction: exit code 0, valid file, wrong document. If you're building automation where a model's output feeds downstream systems and you'd rather not discover the corruption from a reader, that validation layer is worth designing up front. It's a chunk of what I end up building for people in agent and pipeline work, and it's almost always cheaper than the cleanup.

What I'd do differently: I'd have written check-fences.sh before I wrote any transformation step at all. The validator is 20 lines of awk. The sed rule it protects against took a week of published output to notice, and noticing required a human looking at a rendered page. Automation that can't tell you it's broken isn't automation, it's a faster way to be wrong.

Tailscale Kernel TUN in Unprivileged LXC: Direct SSH Without Userspace Networking

Guatu — Sat, 22 Aug 2026 06:15:48 +0000

tailscale up --tun=userspace-networking gets you a green dot in the admin console and almost nothing else. The node appears in your tailnet, tailscale status looks healthy, and then you try to SSH into that container from your laptop and the connection hangs until TCP gives up. Two lines in the LXC config file fix it, and the container stays unprivileged.

That's the whole post, really. But those two lines only make sense once you understand why every guide pushes you toward userspace mode in the first place, and what you're giving up by staying there.

Who should care

Anyone running services in unprivileged LXC containers on Proxmox who wants those containers to be real tailnet members with their own 100.64.0.0/10 address. Not reachable through something else. Reachable directly, over WireGuard, with a kernel network interface that ip addr can see.

If you're already routing everything through a subnet router, you have a working setup and this is an optional upgrade. I covered that pattern in Tailscale Subnet Routers. Treat this as the next rung on the ladder: instead of one node advertising routes on behalf of everyone else, each container carries its own identity, its own ACL surface, and its own direct path to peers.

What userspace networking actually costs you

Every LXC-and-Tailscale guide I've read lands on the same instruction: pass --tun=userspace-networking and move on. It works because it sidesteps the problem entirely. Rather than asking the kernel for a TUN device, tailscaled runs a userspace TCP/IP stack (gVisor's netstack) inside its own process and never opens /dev/net/tun.

Those costs stay invisible until you trip over one.

Outbound traffic needs a proxy. In userspace mode, tailscaled exposes SOCKS5 and HTTP proxies on a local port. Nothing on the system routes to 100.64.0.0/10 automatically, because there is no interface and no route. Every client has to be told about the proxy:

# userspace mode: this is the only way out
export ALL_PROXY=socks5://localhost:1055/
curl http://100.64.0.20:8080/health

Miss that env var in a systemd unit, a cron job, or a nested container, and the traffic silently takes the normal default route instead. No error. It just goes somewhere else, which is the worst failure mode a network can have.

Inbound to a normal daemon doesn't happen. Your sshd binds 0.0.0.0:22 on the container's LAN interface. Packets arriving over the tailnet terminate inside tailscaled's netstack, and there's no path from netstack to a socket the kernel owns unless you build one explicitly with tailscale serve, or you switch to Tailscale SSH where tailscaled itself is the SSH server. Tailscale SSH genuinely works in userspace mode, which is why plenty of people never notice the limitation. But that's tailscaled's SSH implementation, not OpenSSH. If you care about host key pinning, authorized_keys command restrictions, Match blocks, or an sshd_config you've tuned, you now maintain two parallel SSH stories on the same box.

UDP is a mess. Netstack's UDP support has been partial for years. Concrete symptom: mosh does not work. mosh-server starts fine, prints its port and key, and the client sits there forever because the datagrams never arrive. If you SSH over flaky mobile links and lean on mosh to survive roaming, userspace mode is a dead end.

None of this is a bug. It's a documented tradeoff, and for a container that only needs to call out to a couple of HTTP endpoints, it's a perfectly reasonable one. The problem is that most guides present it as the answer rather than the fallback it actually is.

The false starts

First instinct, recommended constantly on forums: make the container privileged. Set unprivileged: 0, restore, done, TUN works. It also hands the container a root that maps to real host root, which throws away the single most valuable property of an unprivileged container. For a machine that's about to sit on a VPN and accept inbound connections from anywhere, that trade is exactly backwards.

Second instinct: --features nesting=1. That flag shows up in nearly every Proxmox thread about containers doing unusual things, and it's genuinely required for Docker-in-LXC and systemd cgroup delegation. It does nothing for TUN. Nesting controls whether the container can see and mount its own cgroup hierarchy and its own procfs/sysfs views. Device access is a completely separate mechanism. Set it, restart, observe zero change, spend twenty minutes wondering what you got wrong.

AppArmor is the third dead end. There's a pile of advice suggesting lxc.apparmor.profile: unconfined to get device access working. That's a sledgehammer for a problem AppArmor isn't causing. The default lxc-container-default-cgns profile does not block /dev/net/tun usage; it blocks a set of mount, ptrace, and /proc write operations, none of which sit in the path here. Dropping confinement to fix a device permission issue is the kind of change that looks like it worked and quietly widens the blast radius. I've seen this pattern bite people in other contexts too, which is the same theme as the runc sysctl trap: the fix that "works" is usually the one that removed a boundary you wanted.

What's actually stopping you is much narrower than any of that.

The actual fix

Three conditions have to hold: the host has the tun module loaded, the container's device cgroup allows that specific char device, and the device node is bind-mounted into the container's filesystem.

1. Load `tun` on the host and persist it

On the Proxmox host:

lsmod | grep -w tun || modprobe tun
ls -l /dev/net/tun
# crw-rw-rw- 1 root root 10, 200 Aug 14 09:12 /dev/net/tun

Persist it so a host reboot doesn't quietly undo everything:

echo tun > /etc/modules-load.d/tun.conf

Do this before the next step. The bind mount below has a source path on the host. If /dev/net/tun doesn't exist when the container starts, the mount fails and the container refuses to start, which is a confusing way to learn that a kernel module wasn't loaded.

Repeat this on every node in the cluster. A container that migrates to a host without tun loaded will fail to start there, and you'll be debugging a mount error at the worst possible moment.

2. Add two lines to the container config

Edit /etc/pve/lxc/<CTID>.conf on the host that currently owns the container:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

That's it. No unprivileged: 0, no AppArmor changes, no nesting. Two details matter in the second line. The target path dev/net/tun has no leading slash, because LXC resolves mount targets relative to the container rootfs. And create=file tells LXC to create the target inode if it doesn't already exist, which it won't on a fresh container.

The first line uses cgroup v2 syntax. Anything running Proxmox 8.x is on the unified hierarchy, so lxc.cgroup2.devices.allow is what you want. Older setups on cgroup v1 used lxc.cgroup.devices.allow with the same argument format; if you're copying a snippet from a 2019 forum post, check which one it uses.

3. Stop and start the container from the host

This part catches people. Rebooting from inside the container does not re-read the config file, because the config is consumed by LXC on container creation, not by the init system inside it. You need a full stop/start cycle driven from the host:

pct stop <CTID> && pct start <CTID>

Until you do that, /dev/net/tun simply won't exist inside the container, and you'll assume the config was wrong.

4. Drop the userspace flag and bring Tailscale up

If tailscaled was previously running in userspace mode, that flag lives in /etc/default/tailscaled:

# /etc/default/tailscaled - before
FLAGS="--tun=userspace-networking"

# after
FLAGS=""

Then restart and authenticate:

systemctl restart tailscaled
tailscale up --hostname=app-container-01 --accept-dns=false

I set --accept-dns=false on servers by habit. MagicDNS rewriting /etc/resolv.conf on a box that already has opinions about DNS is a fight you don't need, especially if you're running your own resolver like AdGuard Home. Turn it on deliberately if you want it.

Verifying it actually worked

Four checks, in order. Each one fails differently, which is what makes them useful.

# 1. The device exists and is world-writable
ls -l /dev/net/tun
# crw-rw-rw- 1 nobody nogroup 10, 200 Aug 14 09:12 /dev/net/tun

# 2. A real kernel interface, not a netstack fiction
ip addr show tailscale0
# 3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 ...
#     inet 100.64.0.31/32 scope global tailscale0

# 3. Routes for the tailnet are installed in the container's table
ip route | grep 100.64
# 100.64.0.0/10 dev tailscale0 scope link

# 4. Peer connectivity is direct, not relayed
tailscale ping 100.64.0.20
# pong from db-node (100.64.0.20) via 192.0.2.14:41641 in 1ms

That fourth line is the one worth reading carefully. via <ip>:<port> means a direct WireGuard tunnel between the two hosts. If it says via DERP(sfo) instead, you're relaying through Tailscale's infrastructure, which still works but adds latency and depends on someone else's servers. tailscale netcheck will tell you whether UDP is reachable and whether you're behind a NAT that's blocking direct paths.

Then test the thing you came for:

ssh user@100.64.0.31       # OpenSSH, your config, your keys
mosh user@100.64.0.31      # UDP works now

Why it works

Three separate mechanisms have to cooperate, and the reason so much bad advice exists is that people conflate them.

Device cgroup allowlist. An unprivileged LXC container starts with a deny-by-default device policy and a short allowlist covering the basics: /dev/null, /dev/zero, /dev/full, /dev/random, /dev/urandom, /dev/tty, and the pty devices. The TUN driver registers as char device major 10, minor 200. It's not on that list, so any open() on it returns EPERM regardless of file permissions. Adding c 10:200 rwm grants read, write, and mknod for exactly one device node and nothing else. That's the surgical part: you're not opening a category, you're naming a device.

The bind mount. Passing the cgroup check gives permission to use the device, but the container's /dev is a fresh tmpfs populated by LXC. Creating the node yourself with mknod inside the container fails, because unprivileged containers don't hold CAP_MKNOD in a namespace where it means anything for device creation. Bind-mounting the host's existing node sidesteps that entirely: the inode already exists, LXC just makes it visible at the right path.

Namespaced capability. Opening the device is only half the operation. Actually creating an interface requires the TUNSETIFF ioctl, which needs CAP_NET_ADMIN. Here's the part that makes the whole approach safe: CAP_NET_ADMIN is evaluated against the user namespace that owns the network namespace. The container's root has full CAP_NET_ADMIN over its own netns, and zero authority over the host's. So tailscaled can create tailscale0, set addresses, and install routes inside the container while remaining completely unable to touch host networking. No host privilege is granted at any point.

About that `nobody:nogroup`

Ownership showing as nobody:nogroup inside the container is expected, not a symptom. The host node is owned by uid 0. An unprivileged container maps its uid 0 to something like host uid 100000, and host uid 0 falls outside the mapped range, so the kernel reports it as the overflow uid (65534) which resolves to nobody. The mode is 0666, so every process in the container can open it anyway. People see nobody:nogroup, assume permissions are broken, and go chase a chown that will never work. Nothing to fix.

Direct paths versus the subnet router model

With a kernel interface, the container installs a route for 100.64.0.0/10 and encrypts packets itself. Traffic to a peer leaves as UDP on tailscaled's port (41641 by default) over the container's normal LAN interface, straight to the peer's endpoint. No hairpinning through a subnet router, no dependency on one container staying healthy, no single node whose reboot takes out remote access to a dozen services.

The ACL story improves too. A subnet router advertises a CIDR, so your policy can only reason about IP ranges. Per-node membership lets you tag containers and write rules against identity:

"acls": [
  {
    "action": "accept",
    "src":    ["tag:admin"],
    "dst":    ["tag:lxc-app:22", "tag:lxc-app:3000"]
  }
]

Tag the container at join time with tailscale up --advertise-tags=tag:lxc-app, using an auth key whose owner is permitted to assign that tag. Now access is a property of the machine, not a property of whatever subnet it happens to sit in. That distinction matters a lot once more than one person needs access, and it's the same principle behind two-tier service accounts for agents: scope by identity, not by network position.

Before you restart anything

Step 3 requires stopping the container, which means killing every process inside it. Worth five minutes of preparation:

Save your terminal state. If you live in tmux inside that container, prefix + Ctrl-s with tmux-resurrect saves the session layout and running programs. Losing a session with twelve panes of half-finished debugging is a self-inflicted wound.
Confirm a recent backup. pct listsnapshot <CTID> or a check in Proxmox Backup Server. The config change is easy to revert, but "the container won't start now" is much less stressful with a restore point.
Know how you'll get back in. pct enter <CTID> from the host works regardless of container networking. If your only access path is SSH over the very network you're changing, fix that first.
Record the change. Two lines in a config file are trivially forgettable six months later when a fresh container mysteriously lacks TUN. Put them in whatever provisioning script or template you use, alongside the rest of your cluster build notes.

Lessons learned

Device access is not privilege. The instinct to reach for unprivileged: 0 comes from treating container security as a binary. It isn't. The device cgroup, capability sets, mount entries, and idmaps are independent knobs, and almost every "just make it privileged" answer online is someone who found one knob and gave up on the rest. Name the exact device you need and grant that.

Flags are not interchangeable. nesting=1 fixes cgroup visibility. AppArmor profiles gate mounts and ptrace. Neither touches /dev. Reading the actual mechanism took less time than trying flags at random, and I'd skip straight to the mechanism next time.

Load the module first, always. Container start failing on a missing bind-mount source is a confusing error to debug backwards, and the /etc/modules-load.d/tun.conf file has to exist on every node the container can migrate to. Cluster-wide, not per-node.

Pick one SSH story. Running both OpenSSH and Tailscale SSH on the same node means two authorization models, two audit trails, and two places to revoke access. With a kernel TUN device you get to use plain OpenSSH over the tailnet, so I turn Tailscale SSH off on these containers rather than leaving both paths live.

The surprise was how boring the fix is. Two config lines, one module, one restart. All the complexity lives in understanding why the defaults deny it, which is generally where the interesting part of infrastructure work hides. If you're designing this kind of access model for a team rather than a homelab, that's a conversation I have often through GuatuLabs, and the boundary questions get more interesting once compliance and multiple operators enter the picture.

One caveat worth stating plainly: this gives the container a kernel TUN device, not the right to advertise subnet routes. If you also want it acting as a subnet router, you need IP forwarding enabled inside the container and --advertise-routes on the Tailscale side, which is a different set of tradeoffs. For the common case (a container that should be its own node, reachable directly, with working UDP and your real sshd), the two lines above are the entire job.

Systemd-tmpfiles vs Plex: When /tmp Aging Breaks EAC3 Transcoding

Guatu — Sat, 22 Aug 2026 04:15:47 +0000

A Plex server with 90 days of uptime, a green service status, and perfectly working H.264 playback can still fail every single EAC3 transcode with a generic "Conversion failed" error. Nothing about Plex changed. No update, no config edit, no disk full. A systemd timer quietly deleted the transcoder's scratch directory out from under it, and Plex has no idea.

This one is worth understanding if you run Plex on any systemd-based Linux, whether that's bare metal, a VM, or an unprivileged LXC on Proxmox. It's especially relevant right now because Debian 13 enabled automatic /tmp cleaning by default, so setups that ran fine for years on Debian 11 or 12 inherit this failure mode on upgrade without touching a single Plex setting.

The failure mode

Plex handles most audio transcoding with its bundled ffmpeg fork. But Dolby Digital Plus (EAC3) is licensed differently, so Plex ships a separate binary called EasyAudioEncoder, usually abbreviated EAE. It doesn't work like a normal encoder that reads stdin and writes stdout. It's a watch-folder daemon: Plex drops audio into a directory, EAE picks it up, converts it, and writes the result back. That watch folder lives under a per-instance temp directory that looks like this:

/tmp/pms-4f3a2b1c-9d8e-7f6a-5b4c-3d2e1f0a9b8c/EasyAudioEncoder/

The important detail: Plex creates that directory tree exactly once, when the plexmediaserver service starts. There's no health check on it, no recreation logic, no periodic touch. As long as the service keeps running, Plex assumes the directory it made on day one is still there.

Meanwhile, systemd-tmpfiles has its own opinion about /tmp. On most modern distros, /usr/lib/tmpfiles.d/tmp.conf contains a rule like this:

# Clear /tmp entries older than 10 days, /var/tmp after 30
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d

And systemd-tmpfiles-clean.timer fires 15 minutes after boot and then once a day, removing anything under /tmp whose access, modification, and change times are all older than the age threshold. If you're a heavy Plex user who transcodes EAC3 daily, the timestamps stay fresh and you never notice. If your household mostly direct-plays and only occasionally hits EAC3 content, the watch folder sits idle, crosses the 10-day line, and gets reaped.

From that point on, the server is a zombie in one specific dimension. Everything that doesn't need EAE works. Anything that does fails at session start.

What it looks like in the logs

The client-side error is useless: "Conversion failed. The transcoder failed to start up." The actual story is in Plex Media Server.log:

ERROR - [Req#a4f/Transcode] EAE watchfolder is not writable:
  /tmp/pms-4f3a2b1c-.../EasyAudioEncoder/Convert to WAV (to 8ch or less) (~4f3a)/
ERROR - [Req#a4f/Transcode] Transcode runner appears to have died
DEBUG - Job exit code 187

Exit code 187 from the transcoder job plus "EAE watchfolder is not writable" is the signature. If you go look, the path either doesn't exist at all or the pms-* parent is gone entirely. You can confirm the cleanup timer is the culprit by checking when it last ran:

systemctl list-timers systemd-tmpfiles-clean.timer
ls -ld /tmp/pms-* 2>/dev/null || echo "gone"

The immediate fix is boring:

systemctl restart plexmediaserver

Restarting recreates the whole /tmp/pms-* tree and EAC3 transcodes work again, instantly. Which is exactly why this problem generates so many confused forum threads that end with "restarted it and it works now, no idea why." The restart fixes the symptom and destroys the evidence, and ten days of idle EAE later it's back.

The permanent fix: a tmpfiles.d exclusion

You don't need to disable /tmp cleaning, and you shouldn't. The tmpfiles.d format has an exclusion type built for exactly this. Drop one file on the host running Plex:

# /etc/tmpfiles.d/plex-eae.conf
# Exempt Plex's per-instance temp tree from age-based cleanup.
# Lowercase 'x' ignores the path AND everything below it.
x /tmp/pms-*

Then verify systemd actually picked it up:

systemd-tmpfiles --cat-config | grep -A1 plex-eae

The character matters here, and it's an easy place to get burned. tmpfiles.d defines two exclusion types that look almost identical:

x /tmp/pms-* excludes the matching paths and their entire contents from cleaning. This is what you want.
X /tmp/pms-* excludes only the directory itself. Its contents still age out, which puts you right back in the same failure with a slightly different shape: the pms-* directory survives but the EasyAudioEncoder subtree inside it gets reaped.

Globs are legal in exclusion rules and evaluated at cleaning time, so the rule keeps working even though the UUID in the directory name changes on every service restart.

To prove the fix without waiting ten days, force a cleanup pass with an aggressive age and watch the directory survive:

# Dry-run style check: run the cleaner with a 1-second age threshold
touch -d "30 days ago" /tmp/pms-*/EasyAudioEncoder 2>/dev/null
systemd-tmpfiles --clean
ls -ld /tmp/pms-*/EasyAudioEncoder && echo "exclusion works"

If the directory is still there after --clean runs against a backdated mtime, the rule is live. This kind of "verify the fix actually engages" step is the same discipline I argued for in the Longhorn read-only mounts post: a service that reports healthy while an underlying filesystem assumption has silently broken is the worst class of failure, because nothing pages you until a user complains.

The alternative fix: move EAE out of /tmp entirely

If you'd rather not maintain an exclusion rule, you can relocate Plex's temp tree to a path systemd-tmpfiles doesn't manage. EAE derives its working directory from the process environment, not from the "Transcoder temporary directory" setting in the Plex UI. That UI setting moves the video transcode segments, but the EAE watch folders follow TMPDIR. So changing the setting in the web UI does not fix this problem, which is a trap plenty of people fall into before finding the real mechanism.

A systemd drop-in handles it cleanly:

systemctl edit plexmediaserver

[Service]
Environment=TMPDIR=/var/lib/plexmediaserver/tmp
ExecStartPre=/usr/bin/mkdir -p /var/lib/plexmediaserver/tmp

After a daemon-reload and restart, the pms-* tree lands under a path with no age rule attached. The tradeoff is that nothing cleans it up anymore, so orphaned trees from unclean shutdowns accumulate until you notice. The x exclusion in /tmp doesn't have that problem, because the directory still gets wiped on reboot like everything else in /tmp. That's the main reason I'd pick the exclusion rule over the relocation: it changes the minimum necessary behavior and leaves the rest of the /tmp lifecycle intact.

There's a third option worth knowing about even if you don't use it: PrivateTmp=true in the unit. Systemd's private tmp directories get mounted under /tmp/systemd-private-*, and stock tmp.conf ships exclusion rules for those paths, so they're immune to aging by design. Plex's packaged unit doesn't enable it, and flipping it on changes how the service sees paths shared with other processes, so I'd treat it as trivia rather than the recommended fix here.

Gotchas and adjacent traps

Debian 13 flipped the default. Debian historically did not age out /tmp contents on a running system, which is why this bite pattern was mostly a Fedora/Arch/RHEL story for years. Trixie moved /tmp to tmpfs and enabled the 10-day /tmp and 30-day /var/tmp cleanup rules by default. If your Plex LXC or VM got upgraded to Debian 13, you inherited this behavior with zero warning. The exact same Plex install that ran for two years without an EAE incident now has a ten-day fuse that resets only on service restart.

LXC makes the uptime trap worse. Containers restart less often than you think. A Proxmox host reboot cycles every VM, but LXCs with long-running services rack up months of uptime effortlessly, and every one of those days is a day the tmpfiles timer inside the container keeps ticking. Long uptime feels like a stability trophy. For any service that stages state in /tmp at startup and never checks on it again, uptime is actually the risk factor. If you're running Plex inside an unprivileged LXC (a reasonable choice, and the general pattern is covered in Building a Production Homelab), put the tmpfiles.d exclusion inside the container, since that's where the timer that does the reaping lives.

Docker installs behave differently. A Plex container image doesn't run systemd inside it, so there's no tmpfiles timer to reap anything, and /tmp lives and dies with the container. This failure mode is specific to package installs on systemd hosts and systemd-based LXCs. If you migrated from Docker to a native package and suddenly hit this after a couple of weeks, that's why.

Don't fix it by disabling the timer. Masking systemd-tmpfiles-clean.timer works, but now nothing ages out /var/tmp either, and every other well-behaved consumer of the aging policy loses it too. Scoped exclusions exist precisely so you don't have to make that trade.

Plex isn't the only service with this shape. Anything that creates sockets, PID files, or working directories under /tmp at startup and holds a path reference for the life of the process is exposed. PostgreSQL's Unix socket default, some JVM apps with java.io.tmpdir scratch state, and various daemons with startup-created lock directories all have variants of this. Distros ship tmpfiles exclusions for the common ones (X11 sockets, systemd's own private dirs). Third-party packages are on their own, and Plex's package doesn't ship one. Auditing long-running services for this dependency class is the kind of infrastructure-drift review I end up doing in consulting engagements more often than any glamorous architecture work, because drift between OS policy and application assumptions is invisible until it isn't.

Takeaway

The application didn't break. The OS didn't malfunction. Two correct behaviors, Plex creating its EAE workspace once at startup and systemd-tmpfiles aging out idle /tmp entries, combined into a failure with a built-in delay timer. That's what makes it nasty: the cause and the symptom are separated by ten days and belong to two different layers of the stack.

The fix is a one-line file, x /tmp/pms-* in /etc/tmpfiles.d/plex-eae.conf, verified with systemd-tmpfiles --cat-config and a forced --clean pass. Reach for this pattern any time a long-running service treats /tmp as if it were persistent, and be suspicious of any "it works after a restart" fix where nobody can explain what the restart actually repaired. On a systemd host, a startup-created /tmp directory plus long uptime is a scheduled outage; the only question is the date.

Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

Guatu — Sat, 22 Aug 2026 02:15:49 +0000

A CronJob exits 0, Pushgateway shows a heartbeat metric that hasn't updated in days, and Prometheus fires nothing. Every dashboard is green while the one thing the heartbeat was supposed to guarantee (that you'd know when the job stops working) has quietly stopped being true.

This is a gotcha about two Kubernetes features that are each fine on their own: the default ndots:5 DNS behavior and namespace-scoped NetworkPolicies. Put them in the same cluster with a wildcard DNS record and a push-based monitoring pattern, and you get a failure mode where the monitoring infrastructure itself is the victim, which means nothing tells you it happened.

What I expected

The heartbeat pattern for batch jobs is simple and well established. Prometheus can't scrape a pod that lived for forty seconds, so instead the job pushes a timestamp to Pushgateway when it finishes, Prometheus scrapes Pushgateway on its normal cycle, and an alert rule fires if the timestamp goes stale. A dead man's switch:

# Last line of the backup job's script
echo "backup_last_success_timestamp $(date +%s)" | curl -s --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local:9091/metrics/job/nightly-backup"

- alert: BackupTooOld
  expr: time() - backup_last_success_timestamp > 86400
  for: 15m
  labels:
    severity: warning

If the backup breaks, the timestamp stops advancing, the expression crosses the threshold, and I get paged. That's the theory. The name in the curl command is even the full service FQDN, so DNS should be unambiguous. It is not.

What actually happened

Three independent failures stack here, and each one masks the next. That stacking is the whole gotcha; any single layer would have been a five-minute fix.

Layer 1: ndots:5 turns your FQDN into a wildcard lookup

A pod with the default dnsPolicy: ClusterFirst gets a resolv.conf that looks like this:

search ai-jobs.svc.cluster.local svc.cluster.local cluster.local lab.example.com
nameserver 10.96.0.10
options ndots:5

Note the last search domain. The kubelet appends the node's own search domains after the cluster ones, so if your nodes carry an internal domain like lab.example.com, every pod inherits it.

Now count the dots in pushgateway.monitoring.svc.cluster.local. Four. The ndots:5 option means any name with fewer than five dots is treated as relative, and the resolver walks the search list before trying the name as-is. So your "fully qualified" name generates these queries, in order:

pushgateway.monitoring.svc.cluster.local.ai-jobs.svc.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.svc.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.cluster.local → NXDOMAIN
pushgateway.monitoring.svc.cluster.local.lab.example.com → an answer

Step 4 is the trap. If your internal DNS has a wildcard record (*.lab.example.com pointing at your ingress controller's LoadBalancer VIP, a common setup for homelab TLS and per-app hostnames), that query matches the wildcard and returns the ingress VIP. The resolver got an answer, so it stops. The actual absolute name, the one that resolves to the real Pushgateway ClusterIP, never gets queried.

Your heartbeat is now aimed at Traefik on port 9091. I've written about this exact mechanism corrupting TLS validation in Wildcard DNS + ndots:5: The TLS Nightmare; the Pushgateway variant is nastier because the failure has no user-facing symptom at all.

The wildcard also changes the character of the failure. Without it, step 4 would return NXDOMAIN, the resolver would fall through to the absolute name, and everything would work (slower, with three wasted queries, but working). The wildcard converts a harmless inefficiency into a wrong answer.

Layer 2: NetworkPolicy makes the wrong answer unreachable, quietly

The monitoring namespace runs default-deny ingress, the pattern from Network Policies with Calico: Default Deny and Namespace Isolation. Pushgateway has an allow rule for Prometheus scrapes and for the namespaces that existed when the policy was written.

Two things go wrong at this layer. First, the connection isn't even headed to Pushgateway anymore; it's headed to a LoadBalancer VIP that sits outside the pod and service CIDRs. If the workload namespace has an egress policy scoped to in-cluster traffic, that packet gets dropped on the way out. Second, even after you fix DNS, a new namespace (say you just added an ai-jobs namespace for LLM batch work) isn't in Pushgateway's ingress allow-list, so its pushes get dropped on arrival instead.

Either way, the drop is silent by design. NetworkPolicy denies don't send RST packets or ICMP errors with most CNI configurations. The curl just hangs until its TCP timeout, which on a default curl with no --max-time can be over two minutes. The job runtime gets longer and nobody notices, because who watches the runtime of a job that's succeeding?

This is the maintenance-burden face of NetworkPolicy that doesn't show up in tutorials: an allow-list written in March is wrong by August, because namespaces get added and every new one silently falls outside the rules.

Layer 3: both the script and the alert fail open

Here's where it becomes invisible. The push was the last line of the script, and it looked like this in the failure case:

curl -s --data-binary @- "http://..." || true

The || true was there so a flaky Pushgateway wouldn't fail an otherwise-good backup. Reasonable instinct, terrible outcome: the curl times out, the job still exits 0, and Kubernetes reports the CronJob as succeeding.

The alert rule fails open too, and this part catches almost everyone. time() - backup_last_success_timestamp > 86400 only evaluates when the series exists. Pushgateway restarted at some point (it stores pushed metrics in memory unless you enable persistence), the series vanished, and the expression started returning an empty result. Empty isn't "greater than 86400." Empty is nothing. No data, no alert.

So: the DNS layer sends the heartbeat to the wrong place, the network layer drops it without an error, the script layer hides the failure from Kubernetes, and the alerting layer can't fire on a metric that doesn't exist. Four layers, zero signals.

The fix

Work from the inside out: make the failure loud first, then fix the path.

1. Make the push failure fail the job. Drop the || true, add -f so HTTP errors count, and bound the timeout so a network drop fails in seconds instead of minutes:

echo "backup_last_success_timestamp $(date +%s)" | \
  curl -fsS --max-time 10 --retry 2 --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local.:9091/metrics/job/nightly-backup"

Notice the trailing dot on the hostname. That marks the name as absolute and bypasses the search list entirely, which makes this one command immune to the ndots problem regardless of pod configuration. It's the cheapest fix in this whole post.

2. Fix ndots at the pod level. The trailing dot fixes one command; dnsConfig fixes every lookup the pod makes:

apiVersion: batch/v1
kind: CronJob
spec:
  jobTemplate:
    spec:
      template:
        spec:
          dnsConfig:
            options:
              - name: ndots
                value: "2"

With ndots:2, any name with two or more dots gets tried as an absolute name first. pushgateway.monitoring.svc.cluster.local resolves on the first query, and even the short form pushgateway.monitoring still works because the search list is consulted after the absolute attempt fails. You lose nothing for cluster-internal names and you stop feeding four-dot FQDNs to your wildcard record.

3. Expand the NetworkPolicy with labels, not names. Enumerating source namespaces by name is what made the policy rot when a new namespace arrived. Select on a label instead:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pushgateway-clients
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pushgateway
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              pushgateway-client: "true"
      ports:
        - port: 9091
          protocol: TCP

Then onboarding a new batch namespace is one label away: kubectl label namespace ai-jobs pushgateway-client=true. The policy stops being a file you have to remember to edit and becomes a contract the namespace opts into. Pair it with a Kyverno rule that requires the label on namespaces containing CronJobs if you want it enforced rather than remembered.

4. Alert on absence, not just staleness. The dead man's switch needs to fire when the series is missing, because "missing" is exactly what the failure mode produces:

- alert: BackupHeartbeatMissing
  expr: |
    (time() - backup_last_success_timestamp > 86400)
    or
    absent(backup_last_success_timestamp)
  for: 15m
  labels:
    severity: critical

The absent() function returns a value of 1 when the series doesn't exist, and the or makes the rule cover both stale and gone. One caveat: absent() collapses labels, so if you push heartbeats for several jobs under one metric name with a job label, you need one absent() clause per job (or a recording-rule pattern). Tedious, but the alternative is the exact blind spot this post is about. The reasoning behind alerting on the negative space is the same one I laid out in Prometheus Alerting Rules That Don't Cry Wolf: an alert that can't fire when the pipeline feeding it dies isn't an alert, it's decoration.

5. Verify the path from inside the namespace. Don't trust the fix until you've watched the resolution happen where the workload runs:

kubectl run -n ai-jobs dbg --rm -it --restart=Never \
  --image=nicolaka/netshoot -- \
  getent hosts pushgateway.monitoring.svc.cluster.local

If that returns your ingress VIP instead of a ClusterIP, the wildcard is still winning. Follow with a curl -v --max-time 5 to port 9091 to confirm the NetworkPolicy actually passes traffic, because DNS being right says nothing about the packet getting through.

Why this matters

You'll hit this if your cluster combines three things that are each individually recommended: a wildcard DNS record for ingress convenience, default-deny NetworkPolicies, and push-based heartbeats for batch workloads. Plenty of production clusters and most serious homelabs check all three boxes. The layers interact in a way none of their docs mention, and the interaction specifically targets the observability path, so the usual answer ("the alert will catch it") doesn't apply. The alert is the thing that's broken.

The deeper lesson is about failure direction. Every layer in this chain failed open: the wildcard answered instead of erroring, the policy dropped instead of rejecting, the script swallowed the exit code, the alert evaluated to empty. Whenever you build a monitoring path, walk it backwards and ask what each hop does when its input disappears. Anything that goes quiet instead of loud needs an absent(), a -f, or a timeout wrapped around it. This applies well beyond Pushgateway; the same audit caught a gap in how I verify vector database backups, which I covered in Backing Up Qdrant Snapshots Correctly.

Cross-layer failures like this one, where DNS, network policy, and alerting each look correct in isolation, are the most expensive class of problem in infrastructure work, and they're a big part of the reliability engineering I consult on. The fix is rarely clever. It's knowing that the layers talk to each other, and testing the seams instead of the components.

What I'd tell you to do today, in order: add the trailing dot to your push URLs, add absent() to every dead man's switch you own, and run getent hosts for your internal service names from inside a workload namespace. The first two take five minutes. The third will either confirm you're fine or save you from finding out the hard way that your heartbeats have been landing in Traefik's 404 handler.

We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

Guatu — Fri, 14 Aug 2026 06:15:48 +0000

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one.

That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time.

What I expected

The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to). Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion. Instead of letting old facts fade by activation weight, you make validity windows explicit.

My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't.

Schema-wise, it was simple enough. In a property graph it looks like this:

// A temporal fact: Node A was in maintenance for a fixed window
MATCH (n:Server {name: 'node-a'})
CREATE (n)-[:HAS_STATE {
  status: 'maintenance',
  valid_from: datetime('2026-07-20T02:00:00Z'),
  valid_to:   datetime('2026-07-20T04:30:00Z')
}]->(:State {kind: 'maintenance'})

Multiple HAS_STATE edges per server, each with its own window. Query the graph, get the state for any point in time. This is the "structured shared memory" pattern I described in Multi-Agent AI Systems, except now the shared memory understands time. The whole thing felt like an upgrade in every dimension. It reads well in a design doc. It demos beautifully. That was part of the problem.

What actually happened

Retrieval is where it fell apart, and the failure is boring in a way that makes it dangerous. Here's roughly the query the agent's tool was generating:

// The "obvious" query - find the status of a server
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

Read that carefully. It orders by valid_from descending and takes the most recent fact. Most of the time that's correct, because the most recently started state is usually the current one. What it never does is filter by the query's reference time. If the agent is reasoning about an incident that happened at 03:00, and a newer "online" state was recorded at 05:00, this query returns "online." The graph knew Node A was in maintenance at 03:00. The retrieval logic threw that knowledge away.

This is the hallucinated-history problem, and it's insidious because the model isn't hallucinating. The fact is real. The timestamp is real. The agent is just being handed a fact from the wrong window and has no way to know it. Worse, the answer is fluent and specific, so every static evaluation gives it a pass. RAGAS-style faithfulness checks look at whether the answer is grounded in the retrieved context. It was. The retrieved context was simply the wrong slice of time.

I want to be precise about where the failure lived, because it wasn't the graph. The graph was correct. The schema was correct. The data was correct. The failure was split across two places: a retrieval query that dropped the temporal filter, and an evaluation suite that had no test capable of noticing. If we'd only had the first problem, we'd have caught it in review. Having both meant the system looked healthy right up until the one test that mattered.

That missing test is short:

import pytest
from agent_memory import query_state  # our TKG retrieval tool

def test_temporal_regression():
    """Agent must report the state valid AT the reference time,
    not the most recently recorded state."""
    # Maintenance window: 02:00-04:30. Online recorded at 05:00.
    result = query_state(server="node-a", at="2026-07-20T03:00:00Z")

    # A later 'online' fact exists, but at 03:00 the truth is 'maintenance'
    assert result["status"] == "maintenance", (
        f"temporal regression: got {result['status']} "
        f"for a timestamp inside the maintenance window"
    )

Run that single assertion across a few dozen historical state transitions and you get the 41% failure rate. Not a subtle degradation. Nearly half of all time-scoped questions returned a state from the wrong window whenever a newer fact existed. Meanwhile the static suite, which only checked "does the agent know the current status," passed everything. Two evals looking at the same system, one green, one red, and only the red one described reality.

The fix

Correcting the retrieval was one clause. You filter edges so the reference time falls inside the validity window before you order or limit anything:

// Filter by the reference time FIRST, then pick the winner
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
WHERE r.valid_from <= datetime($at)
  AND (r.valid_to IS NULL OR r.valid_to > datetime($at))
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

valid_to IS NULL handles the open-ended "current" state, the fact that has started but not yet ended. Everything else is a closed window, and the WHERE clause guarantees you only ever consider edges whose window contains T. The ORDER BY ... LIMIT 1 is still there to break ties if two windows overlap, but now it's picking among facts that are all actually valid at T, not among every fact ever recorded.

One clause. That's the entire retrieval fix. Which tells you the real bug was never in the query, it was in the fact that nobody wrote the eval that would have made the missing clause obvious on day one.

So the second half of the fix was the more important one: the retrieval tool never gets to reason about time on its own. The agent isn't trusted to remember to pass a reference timestamp, and the LLM isn't trusted to filter windows in its head. Instead, the current time is injected as explicit context at the tool boundary, and the tool refuses to answer a state question without it:

def query_state(server: str, at: str | None = None) -> dict:
    if at is None:
        raise ValueError(
            "query_state requires a reference time. "
            "Temporal facts are meaningless without one."
        )
    # ... run the time-filtered Cypher above ...

Making the reference time a required argument sounds trivial. It's the difference between a tool that silently returns plausible garbage and one that fails loudly when it's used wrong. A loud failure is a bug report. A plausible answer from the wrong window is an incident three weeks later that nobody can reproduce.

We also changed how the retrieved fact is handed to the model. Rather than passing "status: maintenance" as a bare string, the prompt gets the window with it: "As of 2026-07-20T03:00:00Z, node-a status is 'maintenance' (valid 02:00-04:30). A later 'online' state exists from 05:00 and is not applicable to this query." Giving the model the window and the reference time in the same breath means that even if the retrieval ever regresses, the model has a fighting chance to notice the mismatch. Defense in depth, applied to a knowledge base.

Why this matters

I keep coming back to the fact that we did not ship this because of the architecture. We almost shipped it because the architecture was correct and only the evaluation was wrong. That inversion is the whole lesson. A TKG is more capable than flat vector memory, and that extra capability comes with an entirely new class of failure that your existing evals were never designed to see. Adding temporal structure to your memory adds temporal bugs, and static tests are blind to all of them by construction.

Here's the trap I see constantly. Teams treat GraphRAG as a strictly-better upgrade over vector RAG, port their old eval suite unchanged, watch it stay green, and conclude the new system is at least as good as the old one. Their green suite is measuring the properties the old system could fail on. It has no assertion about sequence, no assertion about validity windows, no assertion that state at T equals the state that was actually true at T. The new failure mode is invisible not because it's rare but because nothing is looking for it. This is the same gap I described in Cognitive Memory for Agents: the retrieval method changed, so the questions your evals ask have to change too.

If you're building temporal memory for an agent, write the temporal regression test before you write the graph. Seed a handful of known state transitions where a later fact contradicts an earlier one, then assert that a query scoped to the earlier window returns the earlier fact. That test is a dozen lines. It will fail the moment your retrieval forgets to filter by time, which, based on how naturally that ORDER BY valid_from DESC LIMIT 1 query wrote itself, is going to be your very first implementation.

A few things I'd carry into the next attempt. Make the reference time a required parameter on every temporal query, so the tool cannot be called ambiguously. Keep the graph correct and put your paranoia in the retrieval and the eval, because that's where the wrong-window bug actually lives. Expose temporal queries to the agent through a narrow, well-typed interface, whether that's a tool boundary or an MCP server, so the agent can't hand-roll a query that drops the filter. And treat "it passed the static eval" as necessary, never sufficient, the second time itself becomes a dimension of your data. If you're standing up this kind of time-aware memory for a production agent system and want a second set of eyes on the failure modes, that's a chunk of what I do consulting.

We didn't deploy the temporal knowledge graph that quarter. We deployed the eval, fixed the one-clause retrieval bug it exposed, and shipped the graph once the red test went green. The graph was never the risky part. The risky part was almost trusting a system that no test had ever asked the right question.

DEV Community: Guatu

Tailscale Exit Node Logic: Why Auto-Select Fails for Security and How to Fix It

Why you'd have to choose

Option A: Auto-select (auto:any)

What it is

Strengths

Weaknesses

When it shines

Option B: Pinned exit node, enforced by the OS

What it is

Making the pin hold up: a NetworkManager dispatcher

Locking down the local side

Diagnosing "half-loaded pages"

Weaknesses of Option B

Decision framework

My pick and why

Systemd Config Merges: Avoiding Silent Shadowing with Drop-ins on Fedora 43

The Symptom

What I Expected

What Actually Happens

1. Whole-file shadowing by directory priority

2. Drop-in shadowing by filename

3. List settings append, scalars replace

4. Environment precedence across sources

The Fix

Inspect the merge, not the files

Write drop-ins that state their intent

Prefer drop-ins over full copies, and revert the copies you have

One environment file per service

Fixing the config doesn't fix the service

Why This Matters

Firewall Zone Persistence: Why firewalld Runtime Changes Vanish and How to Pin Them

What I Expected

What Actually Happens

Runtime and permanent are two separate configs

NetworkManager owns the zone of the interfaces it manages

Zone bindings are per connection, not per interface

The Fix

1. Set zones on the NetworkManager connection, not in firewalld

2. Make the default zone the untrusted one

3. Bind interfaces NM doesn't manage in firewalld's permanent config

4. If you experimented in runtime, promote it on purpose

The Shadowing Problem You Create Along the Way

Why This Matters

Invisible Text on GNOME: Fixing Qt/KDE Flatpak Theme Conflicts in Sandboxed Environments

What I Expected

What Actually Happened

Diagnosing It Before Changing Anything

The Fix

Option 1: Use the KDE platform theme and give it a color scheme

Option 2: Force a single Qt Quick style

Option 3: Force the GTK platform theme

Quick fix vs. permanent fix

Why This Matters

Trust the Tailnet, Not the Network: Automating Wi-Fi Hardening with NetworkManager Dispatchers

The default state is worse than you think

What I tried first

Per-SSID allowlists

Runtime-only fixes

Tailscale's built-in auto exit node

Trusting the CGNAT range

The actual solution

Step 1: A zone for hostile Wi-Fi

Step 2: A zone for the tailnet, bound to the interface

Step 3: The dispatcher script

Step 4: Install it with the right ownership

Step 5: Verify, and know where to look

Why it works

Lessons learned

Claude Headless Sandboxing: Why /tmp Writes Fail in Non-Interactive Mode

What I expected

What actually happened

1. Claude Code's permission boundary is the working directory, not your UID

2. systemd has its own opinions about /tmp

The fix

Put output in a directory you explicitly grant

Let the shell own the log, not the agent

The stream-json flag change

Make denials visible

The grep exit code detour

Option A: Auto-select (`auto:any`)

1. Load `tun` on the host and persist it

About that `nobody:nogroup`