Systemd Config Merges: Avoiding Silent Shadowing with Drop-ins on Fedora 43

You edit a unit file, run systemctl daemon-reload, restart the service, and nothing changes. Or you add one Environment= line in a drop-in, and a variable you never touched ends up with a different value. In neither case does systemd raise an error. It merges the files it finds using its own rules, then runs the result.

This is a gotcha post about how those merges work. A drop-in can shadow a unit file or another drop-in without any warning, and list-type settings cause the worst of it. My examples come from Fedora 43, but the merge logic has been stable for years, so all of this applies to any modern systemd distro.

The Symptom

Most versions of this bug look like one of these:

• You edit /usr/lib/systemd/system/foo.service (or /etc/systemd/system/foo.service) and the change has no effect.
• You add a drop-in to "just add one thing" to an ExecStartPre= chain or an environment, and something else in the service breaks.
• You change a variable in an env file, restart, and the process still sees the old value.
• You add a drop-in with a new ExecStart= and the unit refuses to start with:

foo.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

That last one is the lucky case, because at least systemd complains. The others fail silently. The service starts, reports active (running), and does something slightly different from what you think you configured.

What I Expected

A unit feels like a file. One file, one config. Drop-ins feel like patches: you put override.conf next to the service, and its lines "win." I worked from that mental model for a long time, and I see it constantly in forum answers and copy-pasted setup scripts.

Under that model, editing the main unit file should change behavior. Adding a line to a drop-in should add exactly that line and nothing else. Changing an env file should change the environment. All three assumptions are wrong in specific, predictable ways.

What Actually Happens

systemd builds every unit from a stack of files spread across several directories. It then applies merge rules that depend on the type of each setting. Four separate mechanisms interact here, and each one can hide a change.

1. Whole-file shadowing by directory priority

Unit files are searched in a fixed order. For system units, these are the directories that matter:

Path	Who owns it	Priority
/etc/systemd/system/	You (admin)	Highest
/run/systemd/system/	Runtime, generators	Middle
/usr/lib/systemd/system/	Packages (RPM)	Lowest

If foo.service exists in both /etc and /usr/lib, the /etc copy wins entirely. systemd doesn't merge the /usr/lib file. It ignores it. That's why systemctl edit --full foo.service is a trap: it copies the vendor unit into /etc, and from then on every package update to the vendor unit is invisible. Months later the RPM ships a fix to ExecStart=, dnf installs it cleanly, and your service keeps running the frozen copy.

It cuts the other way too. You edit the file under /usr/lib directly, but a full copy is sitting in /etc from some earlier experiment. Your edit does nothing, and the next package update will overwrite it anyway.

2. Drop-in shadowing by filename

Drop-ins live in foo.service.d/ directories under each of those same paths. systemd collects every *.conf file from all of them, sorts them by filename (not by directory), and applies them in that order. For scalar settings, later files win.

Here's the catch. If two drop-ins have the same filename in different directories, systemd only uses the higher-priority one. A /etc/systemd/system/foo.service.d/override.conf completely masks a /usr/lib/systemd/system/foo.service.d/override.conf. That vendor drop-in may have set a timeout or a hardening option you never knew about, and now it's gone.

Fedora adds another layer because it ships top-level type drop-ins. A directory named service.d/ applies to every service on the box, and Fedora uses one (10-timeout-abort.conf) to change what happens when a service hits its stop timeout. Prefix drop-ins work too: foo-.service.d/ applies to foo-bar.service, foo-baz.service, and so on. As a result, settings in your unit can come from files whose names never mention it.

3. List settings append, scalars replace

This one causes the most damage. The behavior changes from setting to setting, and nothing in the unit file syntax tells you which kind you're dealing with.

• Scalar settings (User=, TimeoutStartSec=, Restart=, WorkingDirectory=): last assignment wins. Drop-ins behave the way you'd expect.
• List settings (Environment=, EnvironmentFile=, ExecStartPre=, ExecStartPost=, ExecStart= for oneshot, ReadWritePaths=, and many more): each assignment appends to the list. An empty assignment (ExecStartPre= with nothing after it) resets the list to empty.

That gives you two opposite failure modes from the same drop-in.

First, accidental accumulation. You want to replace a pre-start step, so you write a new ExecStartPre= in a drop-in. systemd appends it. Now both the old and the new step run in order, and if the old one fails, the unit never starts.

Second, accidental truncation. Once people learn about the empty-reset trick, some start every drop-in with an empty assignment "to be safe." Every entry that came before gets wiped, including entries from vendor files they never read.

Truncation is the sneakier of the two. Here's the vendor unit:

# /usr/lib/systemd/system/agent-worker.service
[Service]
Type=simple
ExecStartPre=/usr/libexec/agent-worker/check-config
ExecStartPre=/usr/libexec/agent-worker/clean-sockets
ExecStart=/usr/bin/agent-worker --serve

And here's a drop-in written to "add one migration step":

# /etc/systemd/system/agent-worker.service.d/override.conf
[Service]
# Reset "just in case" -- this deletes check-config and clean-sockets
ExecStartPre=
ExecStartPre=/usr/libexec/agent-worker/migrate

After daemon-reload, the merged unit runs only migrate before start. The config check is gone, and so is the socket cleanup. The service starts fine on the first boot, and on the next unclean restart it trips over a stale socket that clean-sockets used to remove. When you go looking, the drop-in looks innocent and the vendor file looks correct. Neither file shows the problem on its own. You only see it in the merged result.

I see the same shape in AI agent setups, which is part of why I'm writing this up. Many agent runners take their allowed-tool or safe-binary list from an environment variable or a list setting in the unit. A drop-in meant to add one tool resets the list, and the agent quietly loses every tool except the new one. If you've read how I structure agent skills, this is the systemd version of a registry losing its entries: nothing crashes, and the agent just reports that it "can't" do things it could do yesterday.

4. Environment precedence across sources

Environment variables add their own ordering. Variables loaded from EnvironmentFile= override variables set with Environment=, no matter which file declared them. Multiple env files are read in the order they're listed, and later files win.

So if the main unit sets Environment=LOG_LEVEL=info, and some drop-in adds EnvironmentFile=/etc/agent-worker/legacy.env containing LOG_LEVEL=debug, your process runs at debug. A drop-in that sets Environment=LOG_LEVEL=warn won't change that, because env files take precedence. There's a timing difference to keep in mind as well. Environment= lines are part of the unit definition and need a daemon-reload. Env files are read from disk each time the service starts, so editing one only needs a restart. Mix up the two and you'll restart a service expecting a change it hasn't loaded yet.

The Fix

None of this is a bug. It's documented behavior, spread across systemd.unit(5), systemd.service(5), and systemd.exec(5). The fix is to stop reasoning about individual files and start checking the merged result.

Inspect the merge, not the files

Start with these commands. Each one answers a different question:

# Every file that contributes to the unit, in merge order
systemctl cat agent-worker.service

# Which file is the "main" unit, and which drop-ins were applied
systemctl show agent-worker.service -p FragmentPath -p DropInPaths

# The final, merged value of a list setting
systemctl show agent-worker.service -p ExecStartPre

# Final environment config (Environment= entries plus the env file list)
systemctl show agent-worker.service -p Environment -p EnvironmentFiles

# System-wide: every overridden, extended, or masked unit
systemd-delta --type=overridden,extended

systemctl cat is where I start, because it prints file paths as comments above each section. A stray full copy in /etc shows up right away. systemctl show -p ExecStartPre is the definitive answer for list settings. It prints one { path=... ; argv[]=... } block per entry, so you can count the entries and confirm which ones survived the merge.

systemd-delta doesn't get used enough. Run it after any distro upgrade. It lists every unit where /etc overrides /usr/lib, and that's exactly the list of services that have stopped receiving vendor fixes.

systemctl show -p Environment only tells you what the unit declares. It doesn't expand env files. To see what the running process actually received, check one named variable and leave the rest alone:

pid=$(systemctl show -p MainPID --value agent-worker.service)
# Check one known, non-secret variable. Don't dump the whole environ,
# it usually contains credentials loaded from env files.
sudo tr '\0' '\n' < /proc/"$pid"/environ | grep '^LOG_LEVEL='

Write drop-ins that state their intent

Two rules cover most of the damage. Never use a reset unless you mean to replace the whole list. When you do mean it, restate the full list in the same file so a reader can see the final value without opening another file.

Appending one step, the correct way:

# /etc/systemd/system/agent-worker.service.d/20-migrate.conf
[Service]
# Appends after the vendor's check-config and clean-sockets
ExecStartPre=/usr/libexec/agent-worker/migrate

Replacing the list on purpose:

# /etc/systemd/system/agent-worker.service.d/20-prestart.conf
[Service]
# Intentional reset: this file owns the complete pre-start chain
ExecStartPre=
ExecStartPre=/usr/libexec/agent-worker/check-config
ExecStartPre=/usr/libexec/agent-worker/clean-sockets
ExecStartPre=/usr/libexec/agent-worker/migrate

For ExecStart= on a non-oneshot service, the reset isn't optional. systemd only allows one entry, so the reset-then-set pattern is the only valid way to override it.

Name your drop-ins. override.conf is the default name systemctl edit uses, which makes collisions with vendor drop-ins of the same name more likely. Use numbered, descriptive names instead, so the sort order is obvious and nothing gets masked by accident. On Fedora 43's systemd you can do that straight from systemctl edit:

sudo systemctl edit --drop-in=20-migrate agent-worker.service

Prefer drop-ins over full copies, and revert the copies you have

If systemd-delta shows a full unit override you didn't mean to keep, systemctl revert removes the /etc copy and every admin drop-in for that unit, so it falls back to the vendor version:

sudo systemctl revert agent-worker.service
sudo systemctl daemon-reload
# Then re-apply only what you actually need, as a named drop-in
sudo systemctl edit --drop-in=20-migrate agent-worker.service

Before reverting, read the /etc copy with systemctl cat and copy any real customizations somewhere. Revert doesn't ask what's in there.

One environment file per service

Environment shadowing goes away when there's one source of truth. My pattern is a single drop-in that resets the env file list and points at one managed file:

# /etc/systemd/system/agent-worker.service.d/10-env.conf
[Service]
# One source of truth; the leading "-" would make it optional, so omit it
EnvironmentFile=
EnvironmentFile=/etc/agent-worker/agent-worker.env

Here the reset is deliberate and earns its place: it guarantees no vendor or legacy env file comes along for the ride. The .env file is generated from version control or a secrets manager, not edited by hand in three different places. This is the same discipline I use for agent credentials in two-tier service accounts. A secret should live in exactly one file, and the unit should name that file explicitly.

User units deserve extra care. A systemd --user service inherits from the user manager's environment, which gets assembled from environment.d files, systemctl --user import-environment calls, and whatever your login session pushed in. That's another layer of shadowing, one you can't see in the unit at all. Check it with systemctl --user show-environment. For anything that matters, set the variables in the unit's env file instead of relying on inheritance. The execution-context surprises in headless Claude sandboxing come from the same root problem: a process gets a different context than the shell you tested in.

Fixing the config doesn't fix the service

This step catches people even after they've fixed the merge. Correcting the drop-in fixes what systemd starts. It doesn't fix what the application remembers. Services that keep state between runs (session files, caches, agent conversation histories) can carry the effects of the broken config into the fixed one.

Agents are the worst case. An agent that ran for a while with a truncated tool list has a history full of failed tool calls and notes like "this tool is unavailable." Restore the tools, restart the service, and the agent can still reload that history and keep avoiding tools that work fine now. The config is correct and the behavior isn't, because the state learned from the broken run.

So my reset procedure has a step after the restart:

#!/usr/bin/env bash
set -euo pipefail
svc=agent-worker.service
state_dir=/var/lib/agent-worker/sessions  # example path

sudo systemctl daemon-reload
# Confirm the merge before trusting it
systemctl show "$svc" -p ExecStartPre -p EnvironmentFiles
sudo systemctl stop "$svc"
# Clear state that was learned under the broken config
sudo find "$state_dir" -mindepth 1 -delete
sudo systemctl start "$svc"
systemctl status "$svc" --no-pager

Stop before you clear state, not after. Otherwise the running process can write a fresh copy of the stale session back to disk during shutdown.

Why This Matters

You'll hit this in a few predictable situations: after a distro upgrade where vendor units changed underneath a full /etc copy, when you inherit a box with years of override.conf files, when a config management tool and a human both write drop-ins for the same unit, and any time you add to a list setting without checking what it already holds. Fedora's frequent release cadence makes the first one more likely than on slower distros, because vendor units change often.

This is the same class of problem as firewalld runtime changes that vanish on reload. The file you edited and the configuration that's actually running are two different things, and the tooling won't point out the gap.

What I'd do differently, and what I now do by default:

• Never use systemctl edit --full on a packaged unit. Use a named drop-in.
• Never start a drop-in with an empty assignment unless the same file restates the complete list.
• Check systemctl show -p <Setting> after every change to a list setting, not just systemctl status.
• Run systemd-delta after every major Fedora upgrade and look at each override.
• Keep one env file per service, and treat any second EnvironmentFile= as a bug.
• Treat "config fixed" and "service fixed" as two separate checks, and clear learned state when a broken config ran long enough to leave it behind.

If you're running agent workloads or long-lived services on hosts where this kind of drift has piled up for years, that's work I do through GuatuLabs consulting. Most of the time, though, systemctl cat and five minutes of reading turn up the problem. The merged config is always available if you ask systemd for it.