Keep the Kernel Running: systemd soft-reboot

Rebooting is the bluntest instrument we own. A service gets stuck, a library is updated, we say "something happened but I couldn't find out what" — and we always press the same button. That button charges the same fee regardless of how big the problem was: every process dies, the kernel stops, the hardware resets, the firmware performs its own little ceremony, the boot loader shows up, the kernel comes back. On a desktop this is called "time for a coffee"; in a server room where you stare at the screen for minutes while a disk controller introduces itself, it is called an outage.

The annoying part is this: in most of the cases where we pay that price, we have no quarrel with the hardware or the kernel at all. Our quarrel is with userspace — with services, libraries, half-finished state. The soft-reboot that systemd introduced in version 254 draws exactly that line: it rebuilds userspace from scratch and never touches the kernel.

My argument in this article is that describing soft-reboot as a "fast reboot" is both wrong and dangerous. It is a different operation with different guarantees. Anyone who uses it without knowing what it gives you — and, more importantly, what it does not — will one day discover at the worst possible moment that their most trusted tool only did half the job.

What we call a "reboot" is really seven separate jobs

While the upstream documentation lists what soft-reboot does not do, it also spells out how many stages a normal reboot goes through. A userspace reboot passes through none of the following: the second phase of a regular shutdown (systemd-shutdown), the third phase (the return to the initrd context), the hardware reboot operation, firmware initialization, boot loader initialization, kernel initialization, and initrd initialization.

Looking at that list, it becomes clear how carelessly we have been using the word "reboot" for years. We call seven jobs by a single name and pay for all of them together.

So what does systemctl soft-reboot actually do? It sends SIGTERM to any processes left running — without waiting for them to exit — and follows up with SIGKILL. If the /run/nextroot/ directory exists (it may be a regular directory, a directory mount point, or a symlink to either), it switches the file system root to it. Then it re-executes the service manager off the — now possibly new — root, which enqueues a new boot transaction just like a normal boot.

PID 1 survives, but it renews itself. The kernel has no idea any of this happened.

The survivors: /run, sockets and file descriptors

What makes soft-reboot interesting is not its speed but what comes out of the transition alive. The documentation lists these one by one under "resource pass-through".

The most valuable of them is the file descriptor store. Descriptors belonging to services that stay active until the very end are passed to the next boot and placed back into the same unit's store. For this to work, the unit must declare DefaultDependencies=no (and must not tie itself to shutdown with something like Conflicts=shutdown.target); alternatively, FileDescriptorStorePreserve= keeps the store pinned even while the unit is down.

The distinction looks thin on paper, but the consequence isn't; journald is the best illustration. On my own server running Ubuntu 24.04, journald's settings look like this:

$ systemctl show systemd-journald.service \
    -p FileDescriptorStoreMax -p FileDescriptorStorePreserve -p DefaultDependencies
FileDescriptorStoreMax=4224
FileDescriptorStorePreserve=restart
DefaultDependencies=no

$ systemd-analyze fdstore systemd-journald.service
FDNAME TYPE DEVNO     INODE RDEVNO PATH               FLAGS
stored sock 0:8   405178459 -      socket:[405178459] rw
stored sock 0:8   375205204 -      socket:[375205204] rw
stored sock 0:8   371286301 -      socket:[371286301] rw

What sits in that store are the sockets applications write their logs to. When journald is restarted it takes those sockets back, so services don't stumble with "I can't write logs". But FileDescriptorStorePreserve=restart is the default value, and it makes a narrow promise: the store is only kept across a restart of the service, and released when the service is stopped altogether.

And journald is stopped precisely during a soft-reboot transition. The upstream unit file does this deliberately, with the reasoning right there in a comment:

# To avoid journald SIGKILLed during soft-reboot and corrupting journals.
Before=soft-reboot.target
Conflicts=soft-reboot.target

So on the 255 running on my server, journald releases its store on the way into the transition. Upstream treated this as a bug and fixed it in 256 by adding FileDescriptorStorePreserve=yes to the journald unit; the commit message describes the symptom precisely: a unit using the default StandardOutput=journal lost its stdout/stderr sockets when journald stopped, because the descriptors on journald's side were not preserved.

The reason I care about this detail is that soft-reboot is not magic that works on its own. Which resources come out of the transition alive depends on lines written into individual units — and upstream itself left its own logging service out in the first release. Don't enter a transition without asking the same two questions about your own critical service: does this unit stop during the transition, and is its store preserved?

The list of survivors doesn't end there. Descriptors associated with .socket units remain open and connectible if the units aren't stopped during the transition — so an incoming request isn't refused, it waits in the queue. The /run/ file system stays mounted and populated; it is the officially recommended place to pass state between two userspace reboot cycles. File system mounts can also stay mounted if they are configured to remain until the very end of shutdown.

Even processes themselves can survive, but at a cost. The recipe in the documentation goes like this: the unit uses SurviveFinalKillSignal=yes to be skipped in that final SIGTERM/SIGKILL spree, IgnoreOnIsolate=yes so it isn't stopped on isolate, DefaultDependencies=no to step out of the normal shutdown chain, After=basic.target for correct ordering at boot, plus Conflicts= and Before= lines so it does stop on real shutdowns (reboot, poweroff, rescue, emergency).

[Unit]
Description=My Surviving Service
SurviveFinalKillSignal=yes
IgnoreOnIsolate=yes
DefaultDependencies=no
After=basic.target
Conflicts=reboot.target kexec.target poweroff.target halt.target rescue.target emergency.target
Before=shutdown.target rescue.target emergency.target

Templated units add a slice detail: since a [e-mail] instance runs in a slice named system-foo.slice by default, the same three lines must be added to that slice too. And applications that publish a service over D-Bus need to re-establish the connection, because the broker is stopped and started again.

What strikes me here is that upstream refuses to market its own feature. The documentation suggests using this pass-through sparingly, and specifically recommends avoiding letting processes survive. The reasoning is solid: as long as a process lives, code updates are necessarily incomplete, and since a process pins the file system beneath it, two versions of the OS may be held in memory at once. In other words, the enthusiasm of "let me keep everything alive" shortens the outage while making the system more fragile.

What it doesn't solve: the kernel, sysctl, hardware

In the documentation's own words there are two big gaps. First: because the kernel is not reset and keeps running, the OS update remains incomplete. Second: kernel settings under /proc/sys/ (i.e. sysctl) and /sys/ are not reset.

Don't take the second one lightly. Anyone who runs sysctl -w by hand one evening and thinks "I'll reboot in the morning, it'll clear itself" will be surprised to find the same setting still in place after a soft-reboot. The fix is what the docs also point at: keep sufficiently comprehensive /etc/sysctl.d/ files — derive the system's state from files rather than from memory.

The first gap points straight at security. soft-reboot is not an answer to a kernel CVE. There you need either a real reboot or live patching; we compared those two in detail in kernel live patching and the maintenance model in enterprise Linux. The documentation names the same two mitigations: kernel live-patching and proper sysctl files.

One more small detail that can bite: because systemd-shutdown is not executed, the executables under /usr/lib/systemd/system-shutdown/ are not executed either. If your team leaves shutdown cleanup there, soft-reboot quietly skips it.

The list left behind after apt upgrade

The most everyday benefit of this feature isn't in exotic image-based systems but in that familiar scene with an ordinary package manager: the upgrade finishes, the libraries on disk are new, but running processes still carry the old code in memory. On Debian and Ubuntu, needrestart is what reminds you; version 3.6 is installed on my own server and its description says the job out loud — check which daemons need to be restarted after library upgrades.

Looking at that list and typing systemctl restart one by one usually works. But the moment you miss an ordering dependency, or a process that isn't on the list is still holding the old library, you're left with a half-finished upgrade — and that half-finished state usually shows up weeks later as an unrelated-looking bug. soft-reboot is a broad broom here: it starts everything again with the new code, without paying for the firmware round.

With one condition: this broom never visits the kernel. On Ubuntu, if the /var/run/reboot-required file is there, that file is asking for a real reboot, not a soft-reboot. Don't conflate them; the first is a problem of services, the second a problem of the kernel.

Containers: the kill spree doesn't discriminate

On a host running Docker, the first question that came to my mind was whether --live-restore would save me.

It won't, because we're talking about two different things. Docker's live restore keeps containers running if the daemon crashes or is restarted deliberately. Its documented limits are clear too: it is only supported across patch releases, reconnection can break if the daemon configuration changes, Swarm services are out of scope, and if the daemon stays down long enough the default 64K logging buffer can fill up and block containers from logging.

But with soft-reboot, the daemon restarting isn't even the issue. Every remaining process gets SIGTERM and then SIGKILL; the only exceptions are units that have explicitly declared themselves exempt with the recipe above. Container processes are inside that spree. So on a container host, soft-reboot means containers restart.

Worse, live restore doesn't improve things here — it makes them harsher. With it off, containers go through the daemon's own stop flow when docker.service is stopped, so the application sees the grace period docker stop grants. With it on, containers stay alive after the daemon goes down, so the spree at the end of the transition catches them as leftover processes and they take a SIGKILL without a chance at an orderly shutdown. For a container writing data, that difference matters.

The quiet trap: /run/nextroot

There is a behavior in the release notes that doesn't draw much attention but will surprise you in operations: since systemd 255, when a reboot operation is invoked, systemctl automatically performs a soft-reboot if a new root file system has been set up under /run/nextroot/.

The phrase "set up" stays vague in the man page; the actual condition lives in the source, and it has two gates. The logind side requires /run/nextroot to be a valid OS tree — in practice, an os-release must be found inside it, so an empty directory doesn't trigger the switch. From systemd 256 onwards a second gate was added on the systemctl side: the directory must also be a mount point. The comment in the code spells out why — if the new root were stored directly on the /run tmpfs, /run/nextroot could never go away, and every reboot would soft-reboot forever.

The operational translation: in a team running an image-based update flow, what actually happens when someone types reboot depends on the machine's state at that moment. An operator who believes they applied a kernel patch may carry on with a machine whose kernel is unchanged. The ordering makes that risk worse: in logind's decision the soft-reboot branch comes before the kexec branch. Even if you prepared a new kernel with kexec --load, a reboot with nextroot set up gives you a soft reboot, and that kernel is never loaded. This detail matters most on immutable or image-based setups; in my article about adding tools to a server without leaving a trace I also touched on A/B image flows with systemd-sysupdate — in that world, preparing a new root tree and switching to it is nothing unusual.

There is an escape hatch: if the SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 environment variable is set, systemctl skips the automatic switch and performs the real reboot you asked for. The same logic exists for kexec, and SYSTEMCTL_SKIP_AUTO_KEXEC=1 turns that one off. A small but painful detail: you have to pass the variable through sudo, because the sudoers default resets the environment.

sudo SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 systemctl reboot

All of this translation also goes through the logind path. systemctl reboot --force bypasses logind, so neither the automatic soft reboot nor the kexec substitution kicks in — but --force also skips the orderly shutdown of units, so treat it as an emergency exit rather than a solution.

And run the right check; ls -ld /run/nextroot produces false positives, because an existing empty directory doesn't trigger the switch:

findmnt /run/nextroot
ls /run/nextroot/usr/lib/os-release /run/nextroot/etc/os-release

Make comparing uname -r after maintenance a habit too. If you're writing automation, state your intent explicitly: systemctl soft-reboot when you want userspace, and systemctl reboot with the escape variable when you want the kernel renewed.

systemd already took care of measuring it

I'm not going to invent a number for "how much faster?" — and there's no need, because systemd records it itself. As of version 256 the moment shutdown began is passed to the next service manager, which logs the overall "grey-out" time of the operation: from the start of shutdown until the system is fully up again. The same release added a counter of how many soft reboot cycles the system has gone through, exposed over D-Bus. In 257 that counter is also handed to generator processes as the $SYSTEMD_SOFT_REBOOTS_COUNT environment variable.

I verified on my own machine that the counter depends on the version; on a server carrying systemd 255.4 the query comes back empty:

$ systemctl --version | head -1
systemd 255 (255.4-1ubuntu8.17)

$ systemctl show -p SoftRebootsCount
$

So if you plan to build telemetry around "how many soft reboots have I done", check the systemd versions across your fleet first. As I write this the stable line is at 261.2 and the first release candidate for 262 was tagged on 1 September 2026 — but the version running on your servers is most likely whatever your distribution ships.

How to decide

The question to ask about your own setup isn't "should I use soft-reboot?" but "which layer is this maintenance actually about?" The answer picks the tool.

But the first question across a fleet is simpler: do I even have this command? soft-reboot arrived with systemd 254; before that it doesn't exist at all. Debian 12 bookworm carries 252, so the command isn't there; Debian 13 trixie carries 257, and Ubuntu 24.04 carries 255 — that's the version on my own server. Collect systemctl --version across every machine before you write the plan; on a mixed fleet that single line decides whether the runbook is applicable at all.

Work that stays in userspace — library and service updates, applying a configuration change everywhere, the distribution's "restart these 14 services" list, opening a wedged userspace with a clean slate, switching to a new root tree on an image-based system — is soft-reboot's natural territory. Whenever you suspect the kernel, a driver or module, sysctl, or the hardware, a real reboot is the only honest answer.

Don't forget the step in between either: systemctl kexec. This command shuts down and reboots the system via kexec; it loads a kexec kernel if one isn't loaded yet, or fails if it can't (--force falls back to a normal reboot). That automatic loading has an easily missed prerequisite: because the enumeration follows the Boot Loader Specification, the system must be using UEFI and the boot loader entries must be configured appropriately — bootctl list shows you what it sees. On a classic BIOS/GRUB install, this command won't load a kernel for you unless you prepared one by hand with kexec --load. Loading it by hand is also what you want when you need a custom initrd or extra kernel command line options. So you have three steps available: soft-reboot never touches the kernel, kexec renews the kernel but skips the firmware and boot loader round, and a full reboot resets everything. If you suspect the hardware, nothing but the third will do the job. If only the kernel needs patching and the machine must never go down, live patching is the fourth option on the table.

Run the first attempt on a test machine born from the same image rather than in production, and watch what the log stream, socket connections and database clients do on the way back. Two preparations are mandatory: out-of-band access (your SSH session dies during the transition, and if a unit gets stuck on the way back you're left without a console) and making sure the services that must come up at boot are actually enabled — the system doesn't remember what you started by hand, it builds a normal boot transaction.

A short checklist for the first run:

• Is out-of-band console access (IPMI/BMC/provider console) open and verified?
• Have the services that must come up been confirmed with systemctl is-enabled?
• Does /run/nextroot exist — i.e. what would actually happen if you typed reboot?
• Is there a pending kernel-patch marker (/var/run/reboot-required) sitting there?
• Have you written down the three post-transition checks (systemctl --failed, health endpoints, version verification)?

One more note: the man page says systemd-soft-reboot.service and related units should never be executed directly. The correct entry point is systemctl soft-reboot.

Get to know the failure modes in advance too; they all follow from the nature of the transition. A service that depends on an old file system pinned by another surviving process may meet an unexpected version at startup. A client speaking over D-Bus may have lost its connection — because the broker restarted — and may not reconnect on its own. Lock files you left in /run/ stay where they are even though their owner is dead, because /run/ isn't cleared during the transition: a gift if you want to pass state, a trap if you delegate lock cleanup to boot.

As for a rollback plan: there isn't one. If userspace doesn't come back after a soft-reboot, there is no button that says "return to the previous state"; your only path is a real reboot from the console. On a mutable root, that reboot brings you back to the upgraded packages, not to the old version. So your actual rollback mechanism isn't soft-reboot at all — it's whatever update method sits beneath it: A/B images, snapshots, or package pinning.

There is no single thing called a restart

What soft-reboot really taught me isn't a command but a distinction. For years we treated "restart" as one operation and bought the most expensive version of it every time. Yet a system's life isn't measured by a single clock: the kernel has its clock, userspace has another, and individual services run on faster ones still. Knowing which clock you need to reset in order to fix a problem is half the work.

The real gain here, in my view, isn't shaving a few minutes off an outage. The real gain is replacing the "turn it all off and on again" reflex with the question "which layer broke?" Once you start asking that, the next step arrives on its own — and sometimes the answer is to not touch the machine at all.

Official Sources

• systemd-soft-reboot.service(8) — upstream source of the man page
• systemd.service(5) — file descriptor store settings
• systemd NEWS — changelog entries for v254, v255, v256 and v257
• Linux kernel: livepatch documentation
• Ubuntu Livepatch — applying kernel patches without rebooting
• Docker: live restore and its limits