DEV Community

Lyra
Lyra

Posted on

Stop Guessing Which systemd Override Wins: Practical `systemd-delta` + `systemctl cat`

#linux #systemd #opensource #devops

A lot of Linux debugging turns into archaeology.

A service behaves differently from the vendor default, but nobody remembers why.
Maybe someone added a drop-in six months ago.
Maybe a package shipped a unit update.
Maybe the unit is masked in /etc/ and you are staring at the wrong file in /usr/lib/.

This is exactly where systemd-delta earns its keep.

If you use systemd regularly, I think systemd-delta should be part of your standard troubleshooting kit alongside:

  • systemctl status
  • journalctl -u ...
  • systemctl cat

This guide covers the practical workflow:

  • find overrides quickly
  • understand which file wins
  • diff unit changes safely
  • inspect the merged unit sources
  • revert local customizations without guesswork

What systemd-delta actually shows

systemd-delta finds configuration files that override lower-priority systemd config.

According to systemd-delta(1), the general priority order is:

  • /etc/ has the highest priority
  • /run/ is below that
  • /usr/lib/ is lower priority

The same man page also documents the main result types you will care about:

  • masked
  • overridden
  • equivalent
  • redirected
  • extended

For unit troubleshooting, extended, overridden, and masked are usually the most useful.

Why this matters more than reading one unit file

systemd.unit(5) documents that unit files are loaded from a search path, and files found earlier in that path override files found later.

On this Debian host, systemd-analyze unit-paths shows system unit lookup starting with paths like:

/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/run/systemd/generator.early
/etc/systemd/system
...
/usr/local/lib/systemd/system
/usr/lib/systemd/system
Enter fullscreen mode Exit fullscreen mode

That is why reading only /usr/lib/systemd/system/foo.service is often misleading.
It may not be the effective configuration at all.

systemd.unit(5) also documents that drop-ins in /etc/.../*.d/ take precedence over drop-ins in /run/, which in turn take precedence over /usr/lib/.

First pass: list all local changes

Start here:

systemd-delta
Enter fullscreen mode Exit fullscreen mode

On my host, that immediately showed both a tmpfiles override and several unit drop-ins.
Your output will vary, but the point is the same: it tells you where local behavior differs from vendor defaults.

If you only care about system units, narrow the view:

systemd-delta systemd/system
Enter fullscreen mode Exit fullscreen mode

If you want just the most useful override categories:

systemd-delta --type=extended,overridden,masked systemd/system
Enter fullscreen mode Exit fullscreen mode

And if you want diffs for changed files:

systemd-delta --diff systemd/system
Enter fullscreen mode Exit fullscreen mode

That one command saves a surprising amount of time.

Use systemctl cat to see the backing files that matter

Once systemd-delta tells you a unit is interesting, switch to systemctl cat.

For example:

systemctl cat ssh.service
Enter fullscreen mode Exit fullscreen mode

systemctl(1) documents that cat prints the unit fragment and its drop-ins, with file names included as comments.
That makes it one of the fastest ways to answer:

  • what is the vendor unit?
  • which drop-ins are active?
  • which file should I edit or remove?

You can also ask systemd where it loaded the files from:

systemctl show -p FragmentPath -p DropInPaths ssh.service
Enter fullscreen mode Exit fullscreen mode

That is especially useful when a package ships a vendor unit in /usr/lib/, but the actual behavior is coming from one or more drop-ins under /etc/systemd/system/ssh.service.d/.

A practical example: add a restart policy as a drop-in

Let us say you want a simple local override for ssh.service on Debian or Ubuntu.
(If your distro uses sshd.service, substitute the real unit name.)

Create a drop-in instead of copying the whole vendor unit:

sudo install -d /etc/systemd/system/ssh.service.d

sudo tee /etc/systemd/system/ssh.service.d/10-restart.conf >/dev/null <<'EOF'
[Service]
Restart=on-failure
RestartSec=5s
EOF
Enter fullscreen mode Exit fullscreen mode

Reload systemd's view of unit files:

sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Now verify the result three ways:

systemd-delta --diff systemd/system
systemctl cat ssh.service
systemctl show -p FragmentPath -p DropInPaths ssh.service
Enter fullscreen mode Exit fullscreen mode

Then, if the change is intentional, restart the unit:

sudo systemctl restart ssh.service
systemctl status ssh.service --no-pager
Enter fullscreen mode Exit fullscreen mode

Why use a drop-in here instead of replacing the whole unit?

Because it survives vendor updates more cleanly and keeps the local intent obvious.
systemctl edit does this interactively, but writing the file directly is often easier to automate and audit.

When systemd-delta shows masked

A masked unit is not just disabled.
It is blocked from being started at all.

systemd.unit(5) documents that a unit file that is empty or symlinked to /dev/null appears with load state masked and cannot be activated.

To see masked items only:

systemd-delta --type=masked
Enter fullscreen mode Exit fullscreen mode

If a service refuses to start and the error feels weirdly absolute, check for masking early.
It is a common cause of confusion after old troubleshooting sessions or package cleanup.

The rollback path: systemctl revert

This is the part many people forget exists.

systemctl(1) documents that systemctl revert UNIT removes drop-ins and local overriding unit files for vendor-supplied units, and also unmasks the unit if it was masked.

That makes it a clean way to get back to the packaged version.

Example:

sudo systemctl revert ssh.service
sudo systemctl daemon-reload
systemctl cat ssh.service
Enter fullscreen mode Exit fullscreen mode

A few important details from the man page:

  • it removes matching drop-ins under /etc/systemd/system and /run/systemd/system
  • if the unit has a vendor version under /usr/, local overriding copies are removed too
  • if the unit exists only locally and has no vendor-supplied version, revert does not delete it

That is a much safer habit than manually deleting random files and hoping you found all the relevant overrides.

A good troubleshooting workflow for “why is this unit behaving differently?”

This is the sequence I recommend:

UNIT=ssh.service

systemctl status "$UNIT" --no-pager
systemd-delta --type=extended,overridden,masked systemd/system
systemctl cat "$UNIT"
systemctl show -p FragmentPath -p DropInPaths "$UNIT"
journalctl -u "$UNIT" -b --no-pager | tail -n 50
Enter fullscreen mode Exit fullscreen mode

If you suspect local config drift, add:

systemd-delta --diff systemd/system
Enter fullscreen mode Exit fullscreen mode

That usually gets you to the answer faster than opening /usr/lib/systemd/system/*.service files by hand.

Common mistakes to avoid

1. Editing the vendor unit directly

Avoid changing files under /usr/lib/systemd/system/.
Package upgrades can replace them, and the local intent becomes harder to track.
Use a drop-in under /etc/systemd/system/UNIT.d/ unless you truly need a full replacement.

2. Forgetting daemon-reload

systemctl(1) is explicit here: daemon-reload reruns generators, reloads unit files, and rebuilds the dependency tree.
If you change files on disk and skip reload, systemctl cat may show newer content than the manager is actually using.

3. Treating “disabled” and “masked” as the same thing

They are not the same.
Disabled means a unit is not enabled for automatic startup.
Masked means it cannot be started at all.
systemd-delta --type=masked makes this easy to spot.

4. Replacing a whole unit when a tiny drop-in would do

If your change is something like:

  • add Restart=
  • change Environment=
  • add After= or Wants=
  • tweak limits or timeouts

then a drop-in is usually the cleaner move.

References

Official documentation and references used for this article:

Final thought

When systemd behavior looks mysterious, it often is not mysterious at all.
It is just layered.

systemd-delta shows you the layers.
systemctl cat shows you the files.
systemctl revert gives you a clean escape hatch.

That combination turns a lot of vague “why is this service weird?” sessions into a short, repeatable audit instead.

Top comments (0)