Stop Paying Full Reboot Downtime: Practical systemd soft-reboot on Linux

You finished a userspace package update. Libraries changed. Daemons need a clean boot graph. The box is “up,” but half the stack is still running the old generation.

The reflex is reboot. That means firmware, bootloader, kernel bring-up, initrd, and every cold-start cost that comes with them—even when the kernel itself did not change.

systemd has a narrower tool for that case: soft-reboot. It tears down and restarts userspace while the kernel keeps running. Downtime shrinks from “full machine cycle” to “service manager reexec + new boot transaction.”

This is a practical operator guide: when soft-reboot is the right hammer, how /run/nextroot/ root switches work, what can survive the cut, verification, and the hard boundaries where you still need a real reboot or kexec.

What soft-reboot actually is

From systemd-soft-reboot.service(8) and systemctl(1):

• Trigger with systemctl soft-reboot (do not start systemd-soft-reboot.service by hand).
• That isolates toward soft-reboot.target, which requires systemd-soft-reboot.service.
• Near the end, leftover processes get SIGTERM, then SIGKILL (no wait for polite exit on the final kill wave).
• If /run/nextroot/ exists (directory, mount, or symlink to either), the root filesystem is switched to it.
• PID 1 reexecutes the service manager from the (possibly new) root and enqueues a new boot transaction, similar to a normal reboot’s userspace phase.

It deliberately does not go through:

• the second phase of regular shutdown (systemd-shutdown)
• return to the initrd context
• hardware reboot
• firmware initialization
• bootloader initialization
• kernel initialization
• initrd initialization

systemctl soft-reboot was added in systemd 254. On this host’s generation it ships as part of modern systemd (Debian 13 ships 257.x; upstream man pages track 261).

# Presence / version
systemctl --version | head -1
systemctl cat soft-reboot.target
systemctl cat systemd-soft-reboot.service
man 8 systemd-soft-reboot.service

On a stock install you should see something like:

• soft-reboot.target: Requires=systemd-soft-reboot.service, AllowIsolate=yes, JobTimeoutSec=30min, JobTimeoutAction=soft-reboot-force
• systemd-soft-reboot.service: DefaultDependencies=no, ordered after shutdown.target / umount.target / final.target, SuccessAction=soft-reboot-force

That last line is the fail-closed escape hatch: if the soft path cannot complete cleanly within policy, systemd escalates to a forced soft-reboot action rather than hanging forever in a half-shut state.

Mental model: three reboot depths

Depth	Command	Kernel	Firmware / bootloader	Typical use
Userspace only	systemctl soft-reboot	stays	skipped	userspace refresh, A/B root flip via /run/nextroot/
New kernel, skip firmware	systemctl kexec (after load)	replaced	mostly skipped	kernel update without full POST
Full cycle	systemctl reboot	replaced	full path	firmware/kernel/initrd/device topology changes

Soft-reboot is not “reboot but faster magic.” It is a different contract: kernel state is continuous; userspace state is not (unless you deliberately pin survivors).

Prerequisites

• systemd ≥ 254 with soft-reboot.target present.
• Root (or polkit rights equivalent to other power commands).
• A reason that is userspace-shaped: package refresh, config generation that needs a full dependency graph, image-based root handoff.
• Acceptance that sysctl /sys kernel knobs are not reset and the running kernel is unchanged.

ArchWiki’s systemd page adds two operator notes that match the man page:

• unlocked dm-crypt devices can remain attached across soft-reboot (no full teardown through the usual shutdown path)
• do not soft-reboot as your only action after updates that changed kernel + initramfs—those need kexec or a full reboot

Lab 0: inspect before you pull the lever

On a spare VM or lab host, record a baseline. Soft-reboot will disconnect your SSH session the same way a reboot does for interactive work—plan console or out-of-band access.

# Boot identity
uname -r
cat /proc/sys/kernel/random/boot_id
cat /proc/sys/kernel/osrelease

# Userspace manager
systemctl show -p Version -p UserspaceTimestamp -p FirmwareTimestamp -p LoaderTimestamp -p KernelTimestamp

# What soft-reboot will isolate toward
systemctl list-dependencies soft-reboot.target
systemctl show soft-reboot.target -p Requires -p After -p JobTimeoutUSec -p JobTimeoutAction

Useful expectations:

• boot_id: changes on full reboot / kexec paths that start a new boot; treat soft-reboot as a userspace cycle—verify with timestamps and service start times rather than assuming every “boot counter” UI agrees.
• FirmwareTimestamp / LoaderTimestamp / KernelTimestamp: should stay continuous across soft-reboot; UserspaceTimestamp advances when PID 1 starts a new userspace boot transaction.
• Kernel release from uname -r must be unchanged after soft-reboot (if it changed, you did not soft-reboot).

# After soft-reboot, compare:
systemctl show -p UserspaceTimestamp -p KernelTimestamp -p FirmwareTimestamp
journalctl -b -o short-iso | head
systemctl --failed

The default operation: plain soft-reboot

For “restart all of userspace on the same root”:

# Optional wall / schedule knobs work like other power commands:
# systemctl soft-reboot --when=2026-09-07 23:30:00
# systemctl soft-reboot --when=cancel

sudo systemctl soft-reboot

From systemctl(1):

• equivalent to starting soft-reboot.target with --job-mode=replace-irreversibly --no-block
• asynchronous: the command returns after enqueue
• honors --force and --when= similarly to halt

Never do:

# Wrong: do not start the implementation unit directly
# sudo systemctl start systemd-soft-reboot.service

The man page is explicit: trigger via systemctl soft-reboot (or the target isolation path it implements), not by hand-starting the helper service.

What still happens / what does not

Still happens (userspace shutdown graph toward soft-reboot):

• normal units stop via the usual stop jobs as the transaction isolates
• mounts that are part of the shutdown graph can unmount
• journal and unit state for the new userspace cycle start fresh from PID 1 reexec

Does not happen:

• systemd-shutdown second phase
• executables under /usr/lib/systemd/system-shutdown/ (skipped because systemd-shutdown is not run)
• firmware / bootloader / kernel / initrd
• automatic reset of /proc/sys and /sys policy

If you rely on shutdown hooks in /usr/lib/systemd/system-shutdown/ for disk flush gymnastics or LED scripts, soft-reboot will not run them. Put critical work in normal service ExecStop= / ExecStopPost= instead.

Lab 1: switch root with /run/nextroot/

This is the power feature for image-based and A/B setups: prepare a complete root filesystem hierarchy at /run/nextroot/, then soft-reboot into it without losing kernel-held state (routes, some device setup, non-reset sysctls, unlocked LUKS mappings as applicable).

/run/nextroot/ may be:

• a plain directory on the existing /run tmpfs
• a mount point (bind mount, loop, image)
• a symlink to either

systemd will turn a non-mount /run/nextroot/ into a mount point automatically on soft-reboot when needed.

Minimal directory-based next root (teaching lab)

This is intentionally small and not a full distro root. Use it only to prove the switch mechanism on a disposable VM.

# WARNING: disposable lab only. A real next root needs a bootable userspace:
# /usr, /etc, /var skeleton, device nodes or devtmpfs policy, and a systemd binary.

sudo mkdir -p /run/nextroot
# Example: bind an already-prepared root tree (snapshot, unpacked image, ostree checkout)
# sudo mount --bind /var/lib/images/root-b /run/nextroot

# Or, for image-based hosts, mount the inactive slot:
# sudo mount /dev/disk/by-partlabel/root-b /run/nextroot

find /run/nextroot -maxdepth 2 -type d | head
test -x /run/nextroot/usr/lib/systemd/systemd \
  -o -x /run/nextroot/lib/systemd/systemd \
  && echo "systemd binary visible in next root"

Then:

sudo systemctl soft-reboot

After reconnecting:

findmnt -no SOURCE,FSTYPE,OPTIONS /
# Confirm you landed on the intended slot/device
lsblk -o NAME,LABEL,PARTLABEL,MOUNTPOINTS

Important interaction with plain reboot

From both systemctl reboot and systemctl soft-reboot docs:

• if /run/nextroot/ is set up, systemctl reboot performs a soft-reboot instead, unless you set:

export SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1
sudo systemctl reboot

That is easy to miss in automation. Image updaters that stage /run/nextroot/ and then call reboot are often depending on this auto-soft-reboot behavior. Operators who stage a next root for testing and then expect a full reboot can be surprised—set SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 when you truly need firmware/kernel bring-up.

Resource pass-through (use sparingly)

Soft-reboot can carry selected runtime resources into the next userspace cycle. The man page lists the official mechanisms—and warns to use them sparingly, because mixing old and new generations is how you get “half updated” systems.

1) /run stays mounted

/run is shared across the soft-reboot boundary. It is the natural place for:

• next-root staging (/run/nextroot/)
• one-shot flags for the next boot transaction
• ephemeral coordinator state that should not hit disk

Do not treat /run as durable storage; treat it as kernel-lifetime memory backed state that happens to survive userspace recycle.

2) File descriptor store

Services can stash file descriptors with the manager (FileDescriptorStoreMax= / fdstore protocol). Across soft-reboot:

• FDs in the store of units that remain active until the end can be handed to the same unit after the next cycle
• alternatively, set FileDescriptorStorePreserve=:

Value	Behavior (from systemd.service(5))
no	release store when the service stops
restart (default)	keep while unit is not inactive/failed, or a job is queued, or restart is expected
yes	keep until the unit is removed from memory—useful to pin until the manager exits

FileDescriptorStorePreserve= was added in version 254 (same generation as soft-reboot).

# Inspect a unit's fdstore (when in use)
systemd-analyze fdstore some.service
sudo systemctl clean --what=fdstore some.service

Units that must keep the store without staying “up” the entire shutdown usually combine preserve mode with careful dependency design—or stay alive with the survivor pattern below.

3) .socket units that never stop

Socket FDs remain open and connectible if the .socket unit is not stopped during the transition. Achieved with DefaultDependencies=no and by avoiding conflicts that pull the socket into the stop set. Clients can keep connecting while userspace recycles—powerful for load balancers and local brokers, easy to get wrong if activation policy disagrees after the new boot transaction.

4) Surviving service processes

A process may keep running across soft-reboot only if all of the following hold (paraphrased from the man page example):

# /etc/systemd/system/my-survivor.service
[Unit]
Description=My Surviving Service
SurviveFinalKillSignal=yes
IgnoreOnIsolate=yes
DefaultDependencies=no
After=basic.target
Conflicts=reboot.target kexec.target poweroff.target halt.target rescue.target emergency.target
Before=shutdown.target rescue.target emergency.target

[Service]
Type=oneshot
ExecStart=/usr/bin/sleep infinity
# Real services: your long-running binary; oneshot+sleep is the man-page sketch only
RemainAfterExit=no

For templated units ([e-mail]), also ship a slice that survives, because instances default into a template slice that would otherwise be stopped:

# /etc/systemd/system/system-foo.slice
[Unit]
SurviveFinalKillSignal=yes
IgnoreOnIsolate=yes
DefaultDependencies=no

Official guidance is blunt: prefer not to survive. Survivors:

• pin old filesystem mounts and libraries in memory
• skip code updates for that process
• require D-Bus reconnect logic if they talk to the system bus (the broker does restart; connections drop)

If you need isolation from the host OS tree, the man page points at Portable Services—and warns not to BindPaths= host OS files, or the old tree stays pinned for the life of the unit.

5) Mounts and complex storage

Mounts can remain mounted if configured with DefaultDependencies=no and without Conflicts=umount.target. That is how complex storage can stay attached while userspace turns over—aligned with the “unlocked LUKS stays” operational reality.

Homelab pattern: userspace refresh after apt without a full reboot

A common safe pattern on servers where the kernel did not change:

# 1) Update packages
sudo apt-get update
sudo apt-get upgrade

# 2) If a new kernel/initrd landed, do NOT soft-reboot as the only step
if [ -n "$(find /boot -name 'vmlinuz-*' -newer /proc/1/root 2>/dev/null | head -1)" ]; then
  echo "Kernel payloads changed under /boot — schedule kexec or full reboot"
fi

# 3) Otherwise recycle userspace cleanly
# sudo systemctl soft-reboot

Pair with:

• needrestart or package hooks to see which daemons still need attention
• a scheduled full reboot window for kernel CVEs (livepatch if you use it)
• monitoring that distinguishes “userspace boot age” from “kernel boot age”

Verification checklist

# Before
KTS=$(systemctl show -p KernelTimestamp --value)
UTS=$(systemctl show -p UserspaceTimestamp --value)
echo "kernel=$KTS userspace=$UTS"
uname -r | tee /run/soft-reboot-pre-uname

# Perform soft-reboot, then after login:

# Kernel identity unchanged
diff -u <(cat /run/soft-reboot-pre-uname) <(uname -r)

# Userspace transaction should look like a new boot
systemctl show -p UserspaceTimestamp -p KernelTimestamp -p FirmwareTimestamp
systemctl is-system-running
systemctl --failed

# Soft-reboot path evidence in the journal (wording varies by version)
journalctl -b -u systemd-soft-reboot.service -u soft-reboot.target --no-pager | tail -50
journalctl -b | head -40

Success signals:

• uname -r unchanged
• firmware/loader/kernel timestamps continuous (where exposed)
• userspace services show fresh start times
• failed units list is empty (or only known pre-existing failures)
• if you used /run/nextroot/, findmnt / points at the new backing store

Failure / rollback signals:

• stuck isolating toward soft-reboot.target past the job timeout (watch for soft-reboot-force)
• services pinned to old trees via accidental survivors / bind mounts
• expectation of reset sysctls that never reset—re-apply /etc/sysctl.d/ explicitly if needed:

sudo sysctl --system

Boundaries (what not to expect)

Need	Use instead
New kernel / modules / initramfs	systemctl kexec or full systemctl reboot
Firmware / UEFI variable / boot-entry change	full reboot (--boot-loader-entry=, --firmware-setup, etc.)
Reset all sysctls and kernel object state	full reboot (or explicit sysctl + module reload discipline)
Run /usr/lib/systemd/system-shutdown/ helpers	full shutdown path; or move logic into unit stop jobs
Crash dump / kdump path testing	real panic/kdump flow (separate stack)
Cold device re-probe that depends on boot firmware	full reboot
“Survive everything, zero disruption”	not soft-reboot; use rolling services, FD store carefully, or cluster failover

Related tools that are not substitutes:

• systemctl daemon-reexec: reexec PID 1 only, without a full userspace boot transaction
• systemctl isolate rescue.target: maintenance mode, not an update cycle
• kexec: replaces the kernel; still skips much of firmware, but it is not soft-reboot
• nspawn / containers: recycle a machine without touching the host kernel—different layer

Suggested roll-out

1. Confirm systemd ≥ 254 and read systemctl cat soft-reboot.target.
2. Try plain systemctl soft-reboot on a disposable VM; compare kernel vs userspace timestamps.
3. Document whether your fleet stages /run/nextroot/ and whether reboot auto-promotes to soft-reboot.
4. Keep survivors and FD-store pass-through off by default; add them per service with an explicit threat/availability model.
5. After package updates: soft-reboot for userspace-only changes; kexec/full reboot when /boot kernel artifacts change.
6. Re-apply sysctl policy if you depend on clean defaults after the cycle.
7. Monitor systemctl --failed and boot-age metrics that understand two clocks (kernel uptime vs userspace boot).

Quick reference

# Userspace-only reboot (same root)
sudo systemctl soft-reboot

# Stage a new root, then soft-reboot into it
sudo mount /dev/disk/by-partlabel/root-b /run/nextroot
sudo systemctl soft-reboot

# Force a *full* reboot even if /run/nextroot exists
sudo SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 systemctl reboot

# Inspect implementation units
systemctl cat soft-reboot.target systemd-soft-reboot.service

# FD store preserve (unit drop-in sketch)
# [Service]
# FileDescriptorStoreMax=128
# FileDescriptorStorePreserve=yes

References

• systemd-soft-reboot.service(8) — Debian man page / man7
• systemctl(1) — soft-reboot, auto-soft-reboot on reboot when /run/nextroot/ exists, SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT
• systemd.special(7) — soft-reboot.target
• systemd.service(5) — FileDescriptorStoreMax=, FileDescriptorStorePreserve= (254+)
• ArchWiki — Systemd § Soft reboot
• Portable Services overview — systemd.io/PORTABLE_SERVICES (survivor isolation option cited by the soft-reboot man page)

---

Full reboots are for kernels, firmware, and cold topology. When only userspace needs a clean graph, soft-reboot is the proportionate tool: same kernel, new service manager transaction, optional root flip through /run/nextroot/, and a short list of deliberate pass-through knobs you should almost always leave alone until you have a specific reason not to.