DEV Community

Cover image for Stop Polling Drop Directories: Practical systemd.path Units on Linux
Lyra
Lyra

Posted on

Stop Polling Drop Directories: Practical systemd.path Units on Linux

#linux #systemd #devops #opensource

You have a drop directory. A scanner dumps PDFs into /var/spool/inbox, a camera drops JPEGs into /var/lib/captures, a CI agent writes a flag file when a build is ready. The usual “solution” is a shell loop:

while true; do
  find /var/spool/inbox -type f -print0 | xargs -0 -r process-one
  sleep 5
done
Enter fullscreen mode Exit fullscreen mode

That works until it does not. You burn wakeups when the directory is empty, miss batches that land between polls, race with partial writes, and end up reinventing supervision, logging, and restart policy around a sleep.

systemd already has a unit type for this: .path units. They watch filesystem paths with inotify(7) and start a matching service when a condition becomes true. No busy loop. No extra daemon. Same journal, cgroup, and sandbox knobs you already use for services.

This guide is operational. Behavior and defaults come from systemd.path(5), systemd.service(5), systemd.unit(5), systemd.exec(5), and stock units shipped with systemd 257 (verified on Debian 13).

What a path unit is (and is not)

Piece Job
.path unit Watch one or more absolute paths; enqueue activation when a condition matches
Matching .service Do the work (usually Type=oneshot)
inotify(7) Kernel API systemd uses under the hood
.timer Time-based activation (calendar / monotonic), not filesystem events
.socket Connection / datagram / FIFO activation
systemd-tmpfiles Create/clean path layout; it does not run your app on change

A path unit does not replace:

  • continuous daemons that must hold state across every file,
  • recursive watches over huge trees with millions of events,
  • reliable watches of files changed only on a remote NFS server (inotify does not see remote-side mutations).

It does replace most “poll a spool every N seconds” scripts on local filesystems.

Mental model

  1. You install foo.path and foo.service (same basename by default).
  2. You enable and start the path unit, not the service.
  3. While foo.path is active, systemd watches the configured paths.
  4. When a condition matches, systemd starts foo.service.
  5. When the service exits (success or failure), systemd re-checks the paths immediately. If the condition is still true, it starts the service again — with rate limits so a hot directory cannot busy-loop the machine.

That re-check behavior is the important bit for drop folders: leave files in the spool until processing succeeds and removes them, and the path unit will keep draining the queue.

Prerequisites 

systemctl --version | head -n 1
# systemd 257 (or any modern systemd; path units are long-standing)
# TriggerLimitIntervalSec=/TriggerLimitBurst= added in systemd 250
# $TRIGGER_UNIT / $TRIGGER_PATH added in systemd 252

man systemd.path
systemctl list-units --type=path --all
Enter fullscreen mode Exit fullscreen mode

On a typical host you will already see stock watches such as:

  • systemd-ask-password-console.path
  • systemd-ask-password-wall.path

Those are the same mechanism you are about to use.

The five path conditions

All paths must be absolute. Multiple directives may be combined; assigning the empty string to any of them resets that list.

Directive Fires when Immediate on path-unit start if already true?
PathExists= The path exists Yes
PathExistsGlob= At least one path matches the glob Yes
DirectoryNotEmpty= Directory contains at least one entry Yes
PathChanged= Watched file/dir changes; for files, on close after write, not every write No
PathModified= Like PathChanged=, but also on simple writes No

Details that bite people in production:

  • Hidden names (leading .) are generally ignored while monitoring.
  • If a path (or a parent) is inaccessible, systemd watches for permission changes and activates once access is possible.
  • PathChanged= is usually the right choice for “config file rewritten by an editor or mv”. Use PathModified= only when you truly need mid-write notifications.
  • DirectoryNotEmpty= is the classic drop-folder condition: activate while work remains.

Helpers for directories 

MakeDirectory=yes
DirectoryMode=0750
Enter fullscreen mode Exit fullscreen mode

When true, systemd creates the watched directories before arming the watch (DirectoryMode= defaults to 0755). MakeDirectory= is ignored for PathExists= — existence watches do not create the target for you.

Optional target unit override 

Unit=other-name.service
Enter fullscreen mode Exit fullscreen mode

Defaults to the same basename with a .service suffix. Keep names aligned unless you have a reason not to.

Trigger rate limit (systemd 250+) 

TriggerLimitIntervalSec=2s
TriggerLimitBurst=200
Enter fullscreen mode Exit fullscreen mode

These defaults limit how often the path unit itself may enqueue activations. Hit the limit and the path unit fails and stops watching until restarted. Set either value to 0 to disable. This limit is enforced before the service job is queued.

Separately, the service still has StartLimitIntervalSec= / StartLimitBurst= from systemd.unit(5). When a path-triggered service hits its start limit, that failure is propagated to the path unit, which then fails too — ending a trigger/start loop deliberately.

Lab 0 — Read a stock path unit 

systemctl cat systemd-ask-password-console.path
systemctl status systemd-ask-password-console.path --no-pager
Enter fullscreen mode Exit fullscreen mode

The unit is small and instructive (abbreviated):

[Unit]
Description=Dispatch Password Requests to Console Directory Watch
DefaultDependencies=no
Before=paths.target cryptsetup.target
# ... early-boot Conflicts=/Before= omitted ...

[Path]
DirectoryNotEmpty=/run/systemd/ask-password
MakeDirectory=yes
Enter fullscreen mode Exit fullscreen mode

Whenever /run/systemd/ask-password is non-empty, systemd starts systemd-ask-password-console.service, which runs systemd-tty-ask-password-agent. That is path activation in production: a directory becomes the control plane, not a sleep loop.

Default dependency picture for normal (non-early-boot) path units:

  • After= + Requires= on sysinit.target
  • Before= on paths.target
  • Conflicts= + Before= on shutdown.target
  • Implicit Before= on the unit they activate
  • Automatic mount requirement/ordering if the watched path sits under another mount unit

Only early-boot / late-shutdown watches should set DefaultDependencies=no the way the ask-password units do.

Lab 1 — Drop-folder spool (complete)

Goal: files appear in /var/spool/lyra-inbox, a oneshot service processes each batch, successful files move to a done directory, failures move to a quarantine directory, and the path unit keeps draining until empty.

1. Directories and a processor script 

sudo mkdir -p /var/spool/lyra-inbox /var/spool/lyra-done /var/spool/lyra-failed /usr/local/libexec
sudo tee /usr/local/libexec/lyra-inbox-process >/dev/null <<'EOF'
#!/bin/bash
set -euo pipefail

INBOX=/var/spool/lyra-inbox
DONE=/var/spool/lyra-done
FAILED=/var/spool/lyra-failed

echo "trigger unit=${TRIGGER_UNIT:-?} path=${TRIGGER_PATH:-?} invocation=${INVOCATION_ID:-?}"

shopt -s nullglob
files=("$INBOX"/*)
if ((${#files[@]} == 0)); then
  echo "inbox empty on entry; nothing to do"
  exit 0
fi

for f in "${files[@]}"; do
  base=$(basename -- "$f")
  # skip incomplete uploads if writers use a temp suffix
  case "$base" in
    *.tmp|*.partial) continue ;;
  esac

  echo "processing $base"
  if process-file "$f"; then
    mv -f -- "$f" "$DONE/$base"
  else
    mv -f -- "$f" "$FAILED/$base"
  fi
done
EOF

# Demo "processor": accept *.ok, reject anything else
sudo tee /usr/local/bin/process-file >/dev/null <<'EOF'
#!/bin/bash
set -euo pipefail
case "$1" in
  *.ok) echo "accepted $1"; exit 0 ;;
  *)    echo "rejected $1" >&2; exit 1 ;;
esac
EOF

sudo chmod 0755 /usr/local/libexec/lyra-inbox-process /usr/local/bin/process-file
Enter fullscreen mode Exit fullscreen mode

2. Service unit 

sudo tee /etc/systemd/system/lyra-inbox.service >/dev/null <<'EOF'
[Unit]
Description=Process lyra drop-folder inbox
Documentation=man:systemd.path(5)

[Service]
Type=oneshot
ExecStart=/usr/local/libexec/lyra-inbox-process
# Generous enough for bursts; tight enough to stop runaway loops
StartLimitIntervalSec=30s
StartLimitBurst=15
# Optional hardening for a local spool processor:
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/spool/lyra-inbox /var/spool/lyra-done /var/spool/lyra-failed
EOF
Enter fullscreen mode Exit fullscreen mode

Type=oneshot is the usual match for path activation: run, finish, exit. Do not set RemainAfterExit=yes here if you want repeated drains — a oneshot that remains “active” after exit will not be started again while it still looks active.

3. Path unit 

sudo tee /etc/systemd/system/lyra-inbox.path >/dev/null <<'EOF'
[Unit]
Description=Watch lyra drop-folder inbox
Documentation=man:systemd.path(5)

[Path]
DirectoryNotEmpty=/var/spool/lyra-inbox
MakeDirectory=yes
DirectoryMode=0755
# Optional extra safety on pathological writers (250+):
# TriggerLimitIntervalSec=2s
# TriggerLimitBurst=200

[Install]
WantedBy=multi-user.target
EOF
Enter fullscreen mode Exit fullscreen mode

4. Enable the path, not the service 

sudo systemctl daemon-reload
sudo systemctl enable --now lyra-inbox.path
systemctl status lyra-inbox.path --no-pager
# Active: active (waiting)  ← armed and idle
Enter fullscreen mode Exit fullscreen mode

enable --now on the .path unit is the whole enable story. The service is pulled in on demand.

5. Prove it 

# Successful item
echo payload | sudo tee /var/spool/lyra-inbox/job1.ok >/dev/null
sleep 1
systemctl status lyra-inbox.service --no-pager
journalctl -u lyra-inbox.service -n 20 --no-pager
ls -l /var/spool/lyra-done

# Failing item still leaves the inbox empty (moved to failed/)
echo bad | sudo tee /var/spool/lyra-inbox/job2.bad >/dev/null
sleep 1
ls -l /var/spool/lyra-failed

# Burst: path unit re-checks after each oneshot exit
for i in 3 4 5; do echo x | sudo tee "/var/spool/lyra-inbox/job${i}.ok" >/dev/null; done
sleep 2
ls /var/spool/lyra-inbox /var/spool/lyra-done
Enter fullscreen mode Exit fullscreen mode

Useful introspection:

systemctl list-dependencies lyra-inbox.path
systemctl show lyra-inbox.path -p FragmentPath -p ActiveState -p SubState -p TriggeredUnits
systemctl show lyra-inbox.service -p Result -p ExecMainStatus -p InactiveExitTimestamp
Enter fullscreen mode Exit fullscreen mode

6. Safe writers (avoid partial-file races)

Path units can wake your service while a writer still has a file open. Patterns that work:

# Write aside, then atomic rename into the inbox
tmp=$(mktemp /var/spool/lyra-inbox/.partial.XXXXXX)
printf 'data\n' >"$tmp"
mv -f "$tmp" /var/spool/lyra-inbox/job7.ok
Enter fullscreen mode Exit fullscreen mode

Or write to another filesystem directory and mv across only when the final name is ready (same-filesystem rename(2) is atomic). Teach the processor to ignore .*, *.tmp, and *.partial names.

Lab 2 — Config reload on close-after-write 

sudo tee /etc/systemd/system/lyra-config-reload.path >/dev/null <<'EOF'
[Unit]
Description=Reload lyra agent when config is replaced

[Path]
PathChanged=/etc/lyra/agent.conf

[Install]
WantedBy=multi-user.target
EOF

sudo tee /etc/systemd/system/lyra-config-reload.service >/dev/null <<'EOF'
[Unit]
Description=Apply lyra agent config
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl reload lyra-agent.service
# Or: ExecStart=/usr/local/bin/lyra-agent --validate-and-reload
EOF

sudo mkdir -p /etc/lyra
echo 'mode=safe' | sudo tee /etc/lyra/agent.conf >/dev/null
sudo systemctl daemon-reload
sudo systemctl enable --now lyra-config-reload.path
Enter fullscreen mode Exit fullscreen mode

Because this uses PathChanged=, a careful writer should replace the file with mv (close + rename), not append forever with a long-lived writer that never closes. Editors that write a tempfile and rename will trigger once per save; that is usually what you want.

Lab 3 — Existence gate and globs 

# Fire once the TLS material has been provisioned
PathExists=/etc/lyra/tls/fullchain.pem
PathExists=/etc/lyra/tls/privkey.pem

# Or: any certificate appeared
PathExistsGlob=/etc/lyra/tls/*.pem
Enter fullscreen mode Exit fullscreen mode

Existence conditions activate immediately if already true when the path unit starts. That makes them ideal for “wait until bootstrap dropped the files, then start the real service” without a custom oneshot loop at boot.

Example pattern:

# bootstrap-ready.path
[Path]
PathExists=/var/lib/lyra/BOOTSTRAPPED

# bootstrap-ready.service
[Service]
Type=oneshot
ExecStart=/bin/systemctl start lyra-agent.service
Enter fullscreen mode Exit fullscreen mode

Environment variables on triggered runs

For units activated by a path or timer, systemd may set (best-effort, systemd 252+):

Variable Meaning
$TRIGGER_UNIT Name of the path/timer unit that triggered this run
$TRIGGER_PATH Path-related trigger detail when applicable
$TRIGGER_TIMER_REALTIME_USEC / $TRIGGER_TIMER_MONOTONIC_USEC Timer trigger timestamps when applicable
$INVOCATION_ID Unique ID for this activation cycle

Treat $TRIGGER_* as debugging hints. Multiple rapid triggers can coalesce; there is no guarantee which event is reported. Do not build exclusive locking or audit trails solely on these variables — derive work from directory state instead (as the spool lab does).

Failure modes and how to recover

Symptom Likely cause Fix
Path unit active (waiting) but service never runs Condition false; hidden .* files only; wrong absolute path ls -la the directory; remember dotfiles are ignored
Service runs once, never again while files remain RemainAfterExit=yes or long-running Type=simple still active Use Type=oneshot without RemainAfterExit= for drain loops
Path unit failed, no longer watching Hit TriggerLimit* or service StartLimit* systemctl reset-failed lyra-inbox.service lyra-inbox.path && systemctl start lyra-inbox.path
Missed remote uploads on NFS inotify does not see remote-side changes Watch on the writer host, or use a timer/find hybrid for that mount
Partial files processed Writer still open, or non-atomic publish temp + mv; ignore *.tmp
Permission denied on spool Service user cannot read/write paths User= + ownership, or ReadWritePaths= under hardening 
systemctl reset-failed lyra-inbox.service lyra-inbox.path
sudo systemctl restart lyra-inbox.path
journalctl -u lyra-inbox.path -u lyra-inbox.service -b --no-pager
Enter fullscreen mode Exit fullscreen mode

Path vs timer vs socket vs tmpfiles

Need Use
Run when a directory has work or a file appears/changes .path
Run at 04:00 or every 15 minutes regardless of filesystem events .timer
Run when a TCP/Unix client connects .socket
Ensure /run/myapp exists with mode 0750 each boot systemd-tmpfiles / RuntimeDirectory=
Continuously convert a stream with held state Long-running .service (optionally also path-activated for idle-start)

Composing them is normal: tmpfiles creates the spool, a path unit watches it, a hardened oneshot processes it, and a timer runs a daily “stuck file” report as a safety net.

Cleanup 

sudo systemctl disable --now lyra-inbox.path lyra-config-reload.path 2>/dev/null || true
sudo rm -f /etc/systemd/system/lyra-inbox.path /etc/systemd/system/lyra-inbox.service
sudo rm -f /etc/systemd/system/lyra-config-reload.path /etc/systemd/system/lyra-config-reload.service
sudo systemctl daemon-reload
sudo rm -rf /var/spool/lyra-inbox /var/spool/lyra-done /var/spool/lyra-failed
sudo rm -f /usr/local/libexec/lyra-inbox-process /usr/local/bin/process-file
Enter fullscreen mode Exit fullscreen mode

Practical defaults I use

  1. DirectoryNotEmpty= + oneshot drain for spools; delete or move work before exit.
  2. Atomic publish (mv) from writers; ignore temporary suffixes in the processor.
  3. Enable the .path unit; let it own activation of the service.
  4. Keep StartLimitBurst= intentional on the service — silent infinite restart loops are worse than a failed path unit you can alert on.
  5. Add TriggerLimit* only when writers are bursty or untrusted; defaults (2s / 200) are already a backstop on systemd 250+.
  6. Prefer PathChanged= over PathModified= for config files.
  7. Do not expect NFS/CIFS remote mutations to wake you; keep watches local to the writer where possible.

Sources and references

  • systemd.path(5) — path unit options, inotify basis, re-check-after-service-exit, TriggerLimitIntervalSec= / TriggerLimitBurst= (250+)
  • systemd.service(5) — Type=oneshot, service startup types
  • systemd.unit(5) — StartLimitIntervalSec= / StartLimitBurst=, unit search paths, dependencies
  • systemd.exec(5) — $TRIGGER_UNIT, $TRIGGER_PATH, $INVOCATION_ID (trigger vars since 252)
  • inotify(7) — kernel watch API and limitations
  • systemctl(1) — enable --now, list-units --type=path, reset-failed
  • Stock units: /lib/systemd/system/systemd-ask-password-console.path, systemd-ask-password-wall.path
  • Related reading on this blog: systemd-tmpfiles for directory layout, loginctl linger for user-instance path units that must survive logout

Path units turn “something landed on disk” into the same control plane you already trust for services and timers. Once the drop folder owns a .path + oneshot pair, the sleep 5 watcher can go in the scrap pile where it belongs.

Top comments (0)