DEV Community

Cover image for Stop Debugging Services That Never Ran: Practical systemd Conditions and Asserts on Linux
Lyra
Lyra

Posted on

Stop Debugging Services That Never Ran: Practical systemd Conditions and Asserts on Linux

#linux #systemd #devops #automation

You enable a unit, start it, and get... nothing. No error. systemctl start exits 0. systemctl status says inactive (dead). There is no traceback, no failed state, no red text — just a service that quietly declined to exist.

Nine times out of ten that is not a broken unit file. It is a Condition...= line doing exactly what it was told, and systemd skipping the start silently.

Conditions are one of the least documented and most useful parts of unit files. They let a single image boot on a laptop and a server, let a backup job refuse to run when the backup disk is not mounted, and let a provisioning unit run exactly once. Used carelessly, they produce the invisible-service problem above.

Everything below was checked against systemd 257.13 on Debian 13, with the observed output quoted verbatim.

Conditions are checked at job execution time, not at enable time

A condition is not a dependency and not an install-time guard. It is evaluated at the moment the queued start job is about to run — after the units ordered before it have finished. That timing matters: a condition can test state that After= dependencies just produced.

The semantics, from systemd.unit(5):

  • Multiple conditions are ANDed. All must hold.
  • A leading ! negates a single condition.
  • A leading | turns a condition into a triggering condition: if any triggering condition holds, and all non-triggering conditions hold, the unit starts. The order is |! — pipe first, exclamation second.
  • Assigning the empty string (ConditionPathExists=) resets the entire condition list.
  • All path checks follow symlinks, except ConditionPathIsSymbolicLink=.

If a condition fails, systemd skips the start "mostly silently". The unit does not enter failed. Ordering is still honoured, and dependent units are pulled in as if the unit had activated — the skip happens at the last moment.

Conditions versus asserts: the difference you can observe

Asserts are the loud twin. Same argument syntax, prefixed Assert..., but an unmet assert fails the job, not the unit.

I wrote two throwaway units to confirm the behaviour on systemd 257. One had ConditionPathExists=/nonexistent/lyra-marker, the other AssertPathExists=/nonexistent/lyra-marker. Results:

# condition unit
$ systemctl start lyra-condtest.service
$ echo $?
0
$ systemctl status lyra-condtest.service
○ lyra-condtest.service - Lyra condition test
     Loaded: loaded (/run/systemd/system/lyra-condtest.service; static)
     Active: inactive (dead)

# assert unit
$ systemctl start lyra-asserttest.service
Assertion failed on job for lyra-asserttest.service.
$ echo $?
1
Enter fullscreen mode Exit fullscreen mode

The journal tells the real story:

lyra-condtest.service - Lyra condition test skipped, unmet condition check ConditionPathExists=/nonexistent/lyra-marker
lyra-asserttest.service: Starting requested but asserts failed.
Assertion failed for lyra-asserttest.service - Lyra condition test.
Enter fullscreen mode Exit fullscreen mode

So: conditions skip quietly and succeed; asserts fail loudly and exit non-zero. Neither one moves the unit into failed. That last point surprises people — an assert does not set the unit's failed state, it only fails the queued job, so systemctl is-failed still reports inactive. If your monitoring only looks at unit state, you can miss an assertion failure entirely.

The rule of thumb: use a condition when the unit genuinely does not apply to this machine (wrong architecture, no battery, already provisioned). Use an assert when the unit absolutely should run but something the administrator must fix is wrong.

The conditions that earn their keep

There are around thirty conditions. These are the ones that solve real problems.

Hardware and platform

  • ConditionArchitecture= — x86-64, arm64, native; valued on mixed-architecture fleets sharing one image.
  • ConditionVirtualization= — vm, container, or a specific engine: kvm, docker, podman, systemd-nspawn, private-users. ConditionVirtualization=!container is the standard way to keep a unit off containers.
  • ConditionCPUFeature= (v248) — x86 and arm64 feature checks, e.g. avx2 or arm64.bti.
  • ConditionSecurity= — selinux, apparmor, ima, audit, uefi-secureboot, tpm2. Useful for units that only make sense with a given LSM active.
  • ConditionACPower= — true holds if at least one AC connector is connected or if no AC connector is known at all. That fallback keeps server units working; it also means the condition passes on machines that cannot report power state.
  • ConditionCapability= — checks a capability in the manager's bounding set, not the service's effective set.

Runtime environment

  • ConditionPathExists= and friends — !path negation, ConditionPathExistsGlob=, ConditionPathIsDirectory=, ConditionPathIsMountPoint=, ConditionPathIsReadWrite=, ConditionPathIsSymbolicLink=, ConditionDirectoryNotEmpty=, ConditionFileNotEmpty=, ConditionFileIsExecutable=.
  • ConditionPathIsEncrypted= (v246) — true only when the backing block device is dm-crypt/LUKS. It does not detect ext4 per-directory encryption, and it does not look through loopback devices.
  • ConditionKernelCommandLine= — matches a word or an exact key=value from /proc/cmdline. Read from /proc/1/cmdline when systemd runs as a container payload, which is a nice detail when your condition behaves differently in a container.
  • ConditionFirstBoot= and ConditionNeedsUpdate= — image provisioning primitives.
  • ConditionUser= / ConditionGroup= — mainly for user managers. ConditionUser=@system matches the system UID range, and is pointless for the system manager, which always runs as root.

Lab 1: a maintenance switch that actually stops work

The highest-value pattern in the whole set is a negated path check used as an operator kill switch.

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly restic backup to /srv/backup
Wants=network-online.target
After=network-online.target

# Operator pause switch: touch this file and this unit will not run.
ConditionPathExists=!/etc/backup/pause

# Refuse to run unless the backup target is actually mounted.
ConditionPathIsMountPoint=/srv/backup

# If the config is missing, fail loudly instead of silently doing nothing.
AssertPathExists=/etc/restic/password

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo touch /etc/backup/pause
sudo systemctl start nightly-backup.service   # exits 0, does nothing
sudo journalctl -u nightly-backup.service -b --no-pager | tail -2
# ... skipped, unmet condition check ConditionPathExists=!/etc/backup/pause
sudo rm /etc/backup/pause
Enter fullscreen mode Exit fullscreen mode

Two things worth noticing. The condition is re-evaluated on every start attempt, so it also gates timer-driven activations. And it is not a watchdog: nothing re-checks the file until something starts the unit again.

If you pair this with systemd.timer, put the condition on the service, not the timer. A condition on the timer prevents the timer from starting at all; a condition on the service is evaluated at each activation, which is what you usually want for a pause switch.

Lab 2: only back up on AC power

A laptop that starts a three-hour rsync at 20% battery is a design bug.

# ~/.config/systemd/user/laptop-sync.service
[Unit]
Description=Sync home directory to NAS
ConditionACPower=true
After=network-online.target

[Service]
Type=oneshot
ExecStart=%h/bin/sync-home.sh
Enter fullscreen mode Exit fullscreen mode

Test it without unplugging anything:

systemd-analyze condition --unit=laptop-sync.service
echo $?    # 0 = would start, 1 = would be skipped
Enter fullscreen mode Exit fullscreen mode

On a desktop with no battery, ConditionACPower=true passes because no AC connector is known. That is documented behaviour, not a bug — and the reason this condition is safe to ship in a shared unit.

Lab 3: assert required state, then override it in a drop-in

Vendor units sometimes assert paths you do not have. Resetting an assert requires clearing the list first, because AssertPathExists= is a list setting:

# /etc/systemd/system/myapp.service.d/10-local-paths.conf
[Unit]
# Clear every inherited assertion of this name, then re-add ours.
AssertPathExists=
AssertPathExists=/etc/myapp/config.toml
AssertDirectoryNotEmpty=/var/lib/myapp
Enter fullscreen mode Exit fullscreen mode

Without the empty assignment you would end up with both the vendor's path and yours, and the unit would fail the job for a path that is irrelevant on this host.

This is also the correct place for "the database directory must exist before we start", because a missing directory is an operator problem, not a reason to silently skip.

Lab 4: run provisioning exactly once

First boot semantics are built in: ConditionFirstBoot=yes is true while /etc was unpopulated at boot, and it stops being true once the manager finishes its startup phase.

# /etc/systemd/system/site-provision.service
[Unit]
Description=One-time site provisioning
ConditionFirstBoot=yes
Wants=first-boot-complete.target
Before=first-boot-complete.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/provision.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

The Wants= plus Before= pairing is the part people skip. first-boot-complete.target is what marks first boot as done; ordering the provisioning unit before it means an aborted first boot re-runs the unit next time instead of leaving a half-configured system.

ConditionNeedsUpdate=/etc solves the neighbouring problem — running a migration only when /usr is newer than the .updated stamp file. Units using it should order themselves before systemd-update-done.service, which resets that stamp.

Debugging a unit that will not start

Work in this order. Every command below is read-only.

# 1. What did systemd decide, and why?
systemctl show myapp.service -p ActiveState -p ConditionResult -p AssertResult
# ActiveState=inactive
# ConditionResult=no
# AssertResult=yes

# 2. Which condition failed? (works without starting anything)
systemd-analyze condition --unit=myapp.service
# myapp.service: ConditionPathIsMountPoint=/srv/backup failed.
# Conditions failed.

# 3. The journal has the human-readable reason.
journalctl -u myapp.service -b --no-pager | grep -i -e condition -e assert
Enter fullscreen mode Exit fullscreen mode

Two practical notes from actually running these:

  • systemctl show -p Conditions prints Conditions=[unprintable] because it is a structured property. Use ConditionResult for the verdict and systemd-analyze condition for the detail.
  • systemd-analyze condition also accepts bare expressions, which is handy for testing syntax before you commit a unit: systemd-analyze condition 'ConditionPathExists=/etc/passwd' prints a per-condition pass/fail line and exits 1 if the set fails.

And the trap that started this article: systemctl start exits 0 when a condition skips a unit. Any deploy script that checks only the exit code will happily report success while the service never ran.

Where conditions go wrong

  • They are not dependency control. The man page is explicit: conditions are evaluated long after jobs are queued, so they are "not suitable for conditionalizing unit dependencies". If B must not run without A, use Requires=/After=, not a condition on B.
  • Nothing re-evaluates them spontaneously. A condition that becomes true later does not start the unit. Use a .path unit or a timer to create the start job.
  • They do not interact with Restart=. A skipped unit never started, so there is no restart to perform. A unit whose condition starts failing on a later start will just stop, cleanly.
  • Broken symlinks fail ConditionPathExists= because path checks resolve symlinks. Use ConditionPathIsSymbolicLink= to test the link itself.
  • ConditionKernelVersion= is unportable. Distributions backport features into older version strings, so a version number does not tell you what the kernel supports. The man page calls this out directly.
  • systemd-analyze verify does not evaluate conditions. It catches unit syntax and missing executables, not whether a condition will pass on this host.

The short version

A unit that will not start and will not explain itself is almost always a condition. Check ConditionResult, run systemd-analyze condition --unit=, read the journal line, and the mystery collapses in about ten seconds.

Use conditions for "this unit does not belong here", asserts for "this host is misconfigured and someone should know", and never for dependency logic. The negated path check alone — ConditionPathExists=!/etc/service/pause — is worth adding to every scheduled job you own.

References

Top comments (0)