DEV Community

Cover image for Stop Losing Work Mid-Job: Practical systemd-inhibit on Linux
Lyra
Lyra

Posted on

Stop Losing Work Mid-Job: Practical systemd-inhibit on Linux

#linux #systemd #devops #opensource

You start a multi-hour rsync, walk away, and the laptop suspends halfway through. Or someone hits reboot while apt full-upgrade is unpacking. Or a headless box with IdleAction=suspend parks itself mid-backup because every session looked idle.

Those are not application bugs. They are missing inhibitor locks.

systemd-logind owns a small, deliberate API for this: applications (and operators) can temporarily block or delay sleep, shutdown, idle action, and even the low-level handling of power/lid keys. The command-line front end is systemd-inhibit(1). The protocol is documented in the upstream Inhibitor Locks page and in org.freedesktop.login1(5).

This guide is operator-first. You will list active locks, wrap real jobs so they cannot be interrupted by idle/sleep/shutdown, put inhibitors on oneshot services, tune logind.conf delay budgets, and know exactly when not to take a blocking lock.

What this is (and is not)

Mechanism Job
systemd-inhibit CLI: run a command under an inhibitor lock, or list locks
logind Inhibit() D-Bus API that issues the lock as a file descriptor
IdleAction= / Handle*= What logind does when idle or when keys/lid fire — inhibitors gate these
GNOME session inhibitors Desktop-oriented (logout/screensaver); often forwarded into logind
systemd.watchdog / Type=notify Hang detection and readiness — not sleep/shutdown gating
loginctl linger Keep user@.service alive after logout — unrelated to sleep locks

Inhibitor locks answer one question: “Should this machine be allowed to sleep, shut down, go idle, or honor the lid/power key right now?”

They do not replace backups, fencing, or package-manager locks. They buy time and prevent logind-mediated interruptions.

Mental model

Taking a lock is one D-Bus call:

Inhibit(what, who, why, mode) → file descriptor
Enter fullscreen mode Exit fullscreen mode
  • what — colon-separated list of lock types
  • who — short human label (“Backup job”, “Package Manager”)
  • why — short human reason (“Nightly rsync in progress”)
  • mode — block, delay, or block-weak
  • return value — an open FD; close the FD (or die) and the lock is gone

That last point is the whole design. No stale lock database. No “forgot to unlock” forever. The kernel closes the FD when the process exits, so crashed jobs drop their inhibitors automatically.

systemd-inhibit is a thin wrapper: it takes the lock, execs your command, and holds the FD until the command finishes.

Lock types (--what=)

From systemd-inhibit(1) and the inhibitor-locks doc:

Type What it inhibits
shutdown High-level power-off / reboot / halt / kexec / soft-reboot requested via logind
sleep Suspend / hibernate / hybrid-sleep / suspend-then-hibernate
idle Automatic idle handling (IdleAction= path)
handle-power-key logind’s own handling of the power key
handle-reboot-key logind’s own handling of the reboot key
handle-suspend-key logind’s own handling of the suspend key
handle-hibernate-key logind’s own handling of the hibernate key
handle-lid-switch logind’s own handling of the lid switch

Default when you omit --what= is:

idle:sleep:shutdown
Enter fullscreen mode Exit fullscreen mode

That is the right default for “do not interrupt this job.”

Important split:

  • shutdown / sleep / idle are high-level locks. They affect user-facing requests and idle policy.
  • handle-*-key / handle-lid-switch are low-level locks. They only stop logind’s built-in key/lid handlers so a desktop environment can take over. They do not stop systemctl suspend typed in a root shell.

Also note the lid quirk in logind.conf(5): LidSwitchIgnoreInhibited= defaults to yes. So a high-level sleep block may not stop lid-triggered suspend unless you change that setting or take handle-lid-switch (DE pattern). Laptops care about this; racks mostly do not.

Modes (--mode=)

Mode Behavior Best for
block (default) Operation fails until the lock is released. Privileged users may still override. Package upgrades, burns, backups, migrations
delay Operation waits up to InhibitDelayMaxSec= (default 5s), then proceeds anyway. Only valid for sleep and shutdown. Save-state hooks, screen lock before suspend
block-weak Like block, but silently bypassed for root and for the lock owner’s own requests “Be nice, don’t hard-block admins”

delay is deliberately weak. It exists so apps can flush state on PrepareForSleep(true) without being allowed to brick suspend forever. If you need a multi-hour backup to finish, you want block, not delay.

Lab 0 — See what is already holding locks 

systemd-inhibit --list
# JSON (newer systemd builds):
systemd-inhibit --list --json=pretty 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Typical columns: what, who, why, mode, UID, PID.

You can also ask logind over D-Bus (same data the CLI uses):

busctl call org.freedesktop.login1 /org/freedesktop/login1 \
  org.freedesktop.login1.Manager ListInhibitors

busctl get-property org.freedesktop.login1 /org/freedesktop/login1 \
  org.freedesktop.login1.Manager BlockInhibited DelayInhibited InhibitDelayMaxUSec
Enter fullscreen mode Exit fullscreen mode

BlockInhibited / DelayInhibited are colon-joined unions of active lock types — handy for monitoring probes.

Lab 1 — Protect a long job from idle + sleep + shutdown 

# Hold idle:sleep:shutdown (the default) while the job runs
systemd-inhibit \
  --what=idle:sleep:shutdown \
  --who="homelab-backup" \
  --why="Nightly rsync of /srv" \
  --mode=block \
  rsync -aHAX --info=progress2 /srv/ /mnt/backup/srv/
Enter fullscreen mode Exit fullscreen mode

In another terminal while it runs:

systemd-inhibit --list
# Expect a row for who=homelab-backup, mode=block
Enter fullscreen mode Exit fullscreen mode

Unprivileged systemctl suspend / desktop “Suspend” should now be denied or deferred by policy while the lock is held. Root can still force actions; inhibitors are a cooperation protocol with privilege escape hatches, not a security boundary against root.

Minimal smoke test without a real backup:

systemd-inhibit --who=demo --why="hold for 30s" sleep 30 &
sleep 1
systemd-inhibit --list
wait
systemd-inhibit --list
Enter fullscreen mode Exit fullscreen mode

Lab 2 — Narrow the lock (only what you need)

Taking the widest lock forever is how laptops never sleep and batteries die. Scope it.

# Database dump: block shutdown + idle, but allow explicit admin suspend if you want
systemd-inhibit \
  --what=shutdown:idle \
  --who="pg-dump" \
  --why="Logical dump of appdb" \
  --mode=block \
  pg_dump -Fc -f /var/backups/appdb.dump appdb

# Optical / long write where suspend is the real enemy
systemd-inhibit \
  --what=sleep:idle \
  --who="disk-image" \
  --why="Writing appliance image" \
  --mode=block \
  dd if=image.raw of=/dev/sdX bs=16M status=progress oflag=direct
Enter fullscreen mode Exit fullscreen mode

Man-page canonical example:

systemd-inhibit wodim foobar.iso
Enter fullscreen mode Exit fullscreen mode

Lab 3 — Put an inhibitor on a systemd oneshot

Wrapping ExecStart= is the durable pattern for timers and overnight jobs.

/etc/systemd/system/lyra-backup.service:

[Unit]
Description=Protected rsync backup (inhibits idle/sleep/shutdown)
Documentation=man:systemd-inhibit(1)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
# Nice labels show up in systemd-inhibit --list and desktop UIs
ExecStart=/usr/bin/systemd-inhibit \
  --what=idle:sleep:shutdown \
  --mode=block \
  --who=lyra-backup.service \
  --why="Nightly /srv rsync" \
  /usr/local/sbin/lyra-backup.sh
# Optional hardening — adjust paths to match your script
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/mnt/backup /var/log
PrivateTmp=yes
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Pair with a timer:

# /etc/systemd/system/lyra-backup.timer
[Unit]
Description=Nightly inhibited backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=yes
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo systemctl enable --now lyra-backup.timer
sudo systemctl start lyra-backup.service   # manual test
# While running:
systemd-inhibit --list
journalctl -u lyra-backup.service -b --no-pager
Enter fullscreen mode Exit fullscreen mode

For package upgrades on a workstation you actually use:

sudo systemd-inhibit \
  --what=shutdown:sleep:idle \
  --who="apt" \
  --why="Unattended upgrade in progress" \
  apt -y full-upgrade
Enter fullscreen mode Exit fullscreen mode

Lab 4 — Delay locks and the 5-second budget

Delay mode is for prepare-then-release, not multi-hour work.

# Inspect the budget (property is in microseconds)
busctl get-property org.freedesktop.login1 /org/freedesktop/login1 \
  org.freedesktop.login1.Manager InhibitDelayMaxUSec
# Default: 5000000  →  5 seconds
Enter fullscreen mode Exit fullscreen mode

Raise it only if you have a real flush that needs more wall time (and still keep it bounded):

# /etc/systemd/logind.conf.d/10-inhibit-delay.conf
[Login]
InhibitDelayMaxSec=30
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl restart systemd-logind.service
# Caution: restarting logind can disrupt sessions; do this in a maintenance window
Enter fullscreen mode Exit fullscreen mode

Application shape (from the upstream inhibitor-locks doc):

  1. On start / document open → take Inhibit("sleep", …, "delay"), keep the FD
  2. On PrepareForSleep(true) → flush state fast, then close(fd)
  3. On PrepareForSleep(false) (resume) → take the delay lock again

Watching PrepareForSleep(true) without a delay lock is racy: suspend may complete before your handler finishes. The docs are explicit about this.

CLI-only operators rarely need delay mode. Prefer block around the job, or teach the app the D-Bus protocol.

Lab 5 — Key and lid handling (desktop / laptop)

Desktop environments take handle-power-key, handle-suspend-key, handle-hibernate-key, and handle-lid-switch in block mode so they can show UI or suppress lid-suspend on a docked laptop. Delay mode is not supported for these types.

# Example: temporarily prevent logind from handling the lid itself
# (a DE would normally own this for the whole session)
systemd-inhibit \
  --what=handle-lid-switch \
  --mode=block \
  --who="dock-session" \
  --why="External displays attached" \
  sleep infinity
Enter fullscreen mode Exit fullscreen mode

Related logind.conf(5) knobs (defaults matter):

[Login]
# What logind does with hardware events when no low-level handle-* lock is held
HandlePowerKey=poweroff
HandleSuspendKey=suspend
HandleHibernateKey=hibernate
HandleLidSwitch=suspend
HandleLidSwitchDocked=ignore
# HandleLidSwitchExternalPower=   # ignored until set explicitly

# Do high-level inhibitors (sleep/shutdown/idle) affect key/lid actions?
PowerKeyIgnoreInhibited=no
SuspendKeyIgnoreInhibited=no
HibernateKeyIgnoreInhibited=no
RebootKeyIgnoreInhibited=no
LidSwitchIgnoreInhibited=yes          # default: lid ignores high-level sleep blocks

IdleAction=ignore                     # or suspend / lock / …
IdleActionSec=30min
InhibitDelayMaxSec=5
Enter fullscreen mode Exit fullscreen mode

If your “backup blocked sleep” test still loses to closing the lid, check LidSwitchIgnoreInhibited= before assuming inhibitors are broken.

Polkit and who may inhibit

Taking locks is privileged and gated by non-interactive polkit actions, including:

  • org.freedesktop.login1.inhibit-block-shutdown
  • org.freedesktop.login1.inhibit-delay-shutdown
  • org.freedesktop.login1.inhibit-block-sleep
  • org.freedesktop.login1.inhibit-delay-sleep
  • org.freedesktop.login1.inhibit-block-idle
  • org.freedesktop.login1.inhibit-handle-power-key
  • org.freedesktop.login1.inhibit-handle-suspend-key
  • org.freedesktop.login1.inhibit-handle-hibernate-key
  • org.freedesktop.login1.inhibit-handle-lid-switch

Upstream guidance: delay locks are easier to obtain than block locks because their blast radius is smaller. If Inhibit() is denied, treat it as “continue without protection,” not a hard error — especially for optional desktop features.

System services running as root generally can take locks. User jobs may need linger + polkit rules depending on distro policy.

Monitoring snippet

A tiny probe for dashboards or ExecStartPost= health checks:

#!/usr/bin/env bash
set -euo pipefail
# Exit 0 if a named inhibitor is present
who_re=${1:-lyra-backup}
systemd-inhibit --list --no-legend --no-pager \
  | awk -v re="$who_re" 'tolower($0) ~ tolower(re) { found=1 } END { exit found?0:1 }'
Enter fullscreen mode Exit fullscreen mode

Or watch the aggregate properties:

busctl get-property org.freedesktop.login1 /org/freedesktop/login1 \
  org.freedesktop.login1.Manager BlockInhibited
Enter fullscreen mode Exit fullscreen mode

Failure modes and footguns

  1. delay on a long job — Suspend still happens after a few seconds. Use block.
  2. Lid switch on laptops — Default LidSwitchIgnoreInhibited=yes bypasses high-level sleep blocks for lid events.
  3. Root override — Inhibitors stop unprivileged and policy-bound paths; they are not a mandatory access control against root.
  4. Idle inhibitors on mobiles — Holding idle forever destroys battery life. Scope duration to the job.
  5. Restarting logind — Needed after some conf changes; can disturb sessions. Prefer drop-ins and planned restarts.
  6. Wrong layer — handle-suspend-key does not block systemctl suspend. High-level sleep does (subject to policy).
  7. Assuming DEs are enough — GNOME inhibitors cover logout/screensaver cases and often forward into logind, but system services and other users’ locks only show up fully via logind/systemd-inhibit --list.
  8. No FD inheritance mistakes — The lock is the FD. Backgrounding a job incorrectly so systemd-inhibit exits early drops protection. Prefer systemd-inhibit cmd… or a oneshot unit, not “inhibit &; cmd”.

When to use what

Situation Tool
One-off long shell job systemd-inhibit … command
Nightly backup / upgrade oneshot ExecStart=systemd-inhibit … in a .service + .timer
App must flush before suspend D-Bus Inhibit(..., delay) + PrepareForSleep
DE owns power keys / lid UX handle-*-key / handle-lid-switch block locks
Keep user services after SSH logout loginctl enable-linger (different problem)
Detect frozen services WatchdogSec= + sd_notify (different problem)
Crash evidence systemd-coredump / pstore / kdump

Quick reference 

# List
systemd-inhibit --list

# Default protect (idle + sleep + shutdown), block mode
systemd-inhibit --who=JOB --why=REASON COMMAND

# Explicit
systemd-inhibit --what=idle:sleep:shutdown --mode=block \
  --who=JOB --why=REASON COMMAND

# Delay only (short prepare window)
systemd-inhibit --what=sleep --mode=delay --who=APP --why="flush" COMMAND

# logind delay budget
# InhibitDelayMaxSec= in logind.conf(.d)

# D-Bus
# org.freedesktop.login1.Manager.Inhibit / ListInhibitors
# Signals: PrepareForSleep(b), PrepareForShutdown(b)
Enter fullscreen mode Exit fullscreen mode

References

  • systemd-inhibit(1) — CLI options, default what=, modes, example
  • Inhibitor Locks — lock types, block/delay schemes, PrepareForSleep protocol, polkit action names
  • logind.conf(5) — InhibitDelayMaxSec=, IdleAction=, Handle*=, *IgnoreInhibited=
  • org.freedesktop.login1(5) — Inhibit, ListInhibitors, inhibitor-related properties and signals
  • systemd-logind.service(8) — login manager responsibilities (idle, keys, inhibition)
  • busctl(1) — calling and reading logind properties from scripts

Idle suspend and surprise reboots are policy problems. Take a lock with a clear who/why, keep it only as long as the job, and prefer block for real work and delay for prepare-hooks. Your backups — and your future self — will notice.

Top comments (0)