systemd-sysext: Adding Tools Without a Trace

It's the middle of the night, you're chasing a strange network behaviour on a server, and strace isn't installed. The reflex is obvious: apt install strace. A minute later the tool is in your hands, the problem is solved, everyone goes back to bed. Three months later nobody can describe that server accurately any more — because it carries a handful of packages that are written down nowhere. I call this "midnight drift", and it is exactly what hollows out every inventory I have ever kept.

systemd's sysext mechanism is a sensible alternative to that reflex. Instead of installing your toolbox onto the system, you lay it temporarily over the system; when you're done you pack it up and the machine returns to its original state — provided you actually pack it up, which turns out to be subtler than it sounds. That sounds too good, so let me say it up front: there is a price, and most write-ups skip it. Every output below came from actually running the commands in a throwaway Debian 13 container on my own machine (systemd 257.13, arm64).

The Mechanism: Layering, Not Copying

systemd-sysext activates and deactivates "system extension images". What it does is combine the /usr/ and /opt/ trees inside the extension with the host's own hierarchies via overlayfs, and then mount the result over those directories. The documentation calls this "merging"; the reverse is "unmerging", where taking the mount down reveals the untouched original tree underneath.

The critical detail here is one that's right there in the name yet constantly forgotten in practice: no file is copied. The package database is never touched, dpkg -l output stays the same. An extension is a mount, and a mount does not survive a reboot.

Don't conclude from that "a reboot cleans it up", though — I did, which is why I'm writing it down. If systemd-sysext.service is enabled, the extension is merged again at boot for as long as the image sits in /var/lib/extensions/. What a reboot removes is the mount; the actual cleanup is removing the image.

Extensions are searched for in three directories: /etc/extensions/, /run/extensions/ and /var/lib/extensions/. The first two are not suitable for carrying large images — they exist for holding symlinks. The primary install location is /var/lib/extensions/. Directories found there are treated as directory-based extensions, while files with a .raw suffix are treated as disk-image-based extensions.

I covered the philosophical relative of this approach, immutable infrastructure discipline, in an earlier post. sysext is the official answer to that discipline's most annoying practical question: "fine, but how do I debug inside a read-only image?" Yet as the documentation states plainly, it makes no difference whether the host /usr is managed as an immutable disk image or is a traditional, writable, package-manager-controlled tree; the mechanism works on an ordinary Debian too. Every experiment below is on an ordinary Debian.

The Smallest Working Example

To produce an extension you need no image-building tooling, no build system, nothing. Three things are enough: a directory, a usr/ tree inside it, and an identity file.

mkdir -p /var/lib/extensions/demo/usr/bin \
         /var/lib/extensions/demo/usr/lib/extension-release.d

printf '#!/bin/sh\necho "hello, I come from the extension"\n' \
  > /var/lib/extensions/demo/usr/bin/hello
chmod +x /var/lib/extensions/demo/usr/bin/hello

cat > /var/lib/extensions/demo/usr/lib/extension-release.d/extension-release.demo <<'EOF'
ID=debian
VERSION_ID=13
EOF

The rest is two commands:

# systemd-sysext list
NAME TYPE      PATH                     TIME
demo directory /var/lib/extensions/demo Fri 2026-09-04 05:36:31 UTC

# systemd-sysext merge
Using extensions 'demo'.
Merged extensions into '/usr'.

# hello
hello, I come from the extension

# systemd-sysext status
HIERARCHY EXTENSIONS SINCE
/opt      none       -
/usr      demo       Fri 2026-09-04 05:36:31 UTC

After unmerge, the hello command is gone. The file is still there, of course, waiting patiently under /var/lib/extensions/demo/; it simply isn't visible through /usr any more. Taking a mount down is a far cheaper operation than undoing an installation — because no installation ever happened.

Version Locking: extension-release and the _any Shortcut

That identity file is not decorative. The extension's /usr/lib/extension-release.d/extension-release.NAME file is compared against the host's os-release, and the name must match exactly. ID= has to match; if SYSEXT_LEVEL= is defined it is checked, otherwise VERSION_ID= has to match. If ARCHITECTURE= is defined it must match the kernel's architecture.

I wanted to see what a mismatch actually looks like, so I tried two scenarios. When I deleted the identity file entirely, and when I wrote VERSION_ID=12 and tried to merge onto Debian 13, the answer was the same both times:

No suitable extensions found (1 ignored due to incompatible image(s)).

That message strikes me as rather tight-lipped. It says "there was an image, it didn't fit", but not which field failed — missing identity file, wrong version, wrong architecture, you can't tell them apart. When your extension goes invisible, look here first; it took me a second attempt to remember that.

The escape hatch is ID=_any. Write that and the version check is switched off entirely:

# printf 'ID=_any\n' > .../extension-release.demo
# systemd-sysext merge
Using extensions 'demo'.
Merged extensions into '/usr'.

Reassuring, isn't it? A little too much so. That lock is not arbitrary bureaucracy, it stands against a real hazard: if the binaries in your extension link dynamically against host libraries, the extension breaks silently the moment the host's libc is upgraded underneath you. This is why the documentation requires that binaries shipped in an extension may only link against host libraries if the extension is bound to the host OS version, and must otherwise be statically linked. Writing _any silences the check, not the risk. Reasonable for a single statically built binary of your own; a trap you set for yourself with a dynamically linked toolset.

Where the Bill Arrives: /usr Is Read-Only While Merged

Now we reach the real reason for this post. Extensions are strictly read-only by default. Even on a writable host file system, /usr/ and /opt/ become read-only while extensions are merged — unless mutability is explicitly enabled.

That sentence sits in the documentation, and when I read it I thought "sure, makes sense" and moved on. Then, to see what it actually means, I tried installing an ordinary package while merged:

# systemd-sysext merge && apt-get install -y sl
dpkg: error processing archive /var/cache/apt/archives/libncurses6_...deb (--unpack):
 unable to create '/usr/lib/aarch64-linux-gnu/libform.so.6.5.dpkg-new'
 (while processing './usr/lib/aarch64-linux-gnu/libform.so.6.5'): Read-only file system
dpkg: error while cleaning up:
 unable to remove newly-extracted version of '...libform.so.6.5': Read-only file system
E: Sub-process /usr/bin/dpkg returned an error code (1)

Merging an extension breaks package installation on that machine. With a half-finished dpkg transaction. Now imagine hitting this in production, months later, because an extension you had forgotten about is merged automatically at boot: a routine security patch blows up at midnight, and the error message never mentions the extension — it just says "Read-only file system". Black boxes spend their worst nights without telling anyone.

The rule I take from this is clear: use sysext for debugging and temporary intervention. If you intend to make the merged state a permanent operating condition, accept that package installation no longer works on that machine, and write it down.

Mutability: Getting Write Access Back

The write ban isn't absolute. The --mutable= switch (and, since v259, Mutable= in the configuration file) offers six modes — but there's a naming trap here. The documentation's prose describes them as disabled and enabled, which are not the values the command line accepts. The accepted ones are no (the default), yes, auto, import, ephemeral and ephemeral-import. Type the prose name verbatim and you get:

# systemd-sysext merge --mutable=disabled
Failed to parse argument to --mutable=: disabled

In practice the most useful mode is ephemeral: it permits writes, but they go to empty temporary directories, so they evaporate on unmerge — exactly what you want when you need to poke around under /usr without leaving a mark.

The modes that actually route writes under /var/lib/extensions.mutable/ are auto and yes: writes to /usr land in .../usr/, /opt in .../opt/, /etc in .../etc/. Here's the elegant part — if those subdirectories are symlinks, writes follow the link to its target. So creating /var/lib/extensions.mutable/usr/ → /usr/ routes writes back to the original tree.

But note this carefully: the symlink alone does nothing. The default mode is no, and the documentation is explicit — immutable mode is forced even if write routing directories exist. With the symlink in place and default settings, the answer I got was the same as if I had never created it:

# ln -sfn /usr /var/lib/extensions.mutable/usr
# systemd-sysext merge
# touch /usr/bin/test
touch: cannot touch '/usr/bin/test': Read-only file system

So the permanent fix to that apt accident isn't "the symlink" — it's the symlink plus --mutable=auto. And for an extension that merges automatically at boot, even that isn't enough: v257's systemd-sysext.service unit calls systemd-sysext refresh, with no --mutable in it. To make this stick on Debian 13 you need a drop-in overriding ExecStart with --mutable=auto; from v259 onwards you can do the same with Mutable=auto in /etc/systemd/sysext.conf.

confext: The Same Trick for /etc

systemd-confext applies exactly the same principle to /etc. The image carries an /etc tree and an /etc/extension-release.d/extension-release.NAME identity file, and the search directories are /run/confexts/, /var/lib/confexts/, /usr/lib/confexts/ and /usr/local/lib/confexts/.

# systemd-confext merge
Using extensions 'settings'.
Merged extensions into '/etc'.
# cat /etc/app.conf
setting=from-extension
# systemd-confext status
HIERARCHY EXTENSIONS SINCE
/etc      settings   Fri 2026-09-04 05:37:02 UTC

The same price applies here — you cannot write to /etc while merged. There's also a small surprise: the merged /etc hierarchy is mounted nosuid and (unless you disable it with --noexec=false) noexec, so a hook script shipped via confext won't execute. The real gain the documentation emphasises is different: when a confext is removed, all the old configuration files disappear at once. Anyone who has tried to clean up hand-edited /etc files knows what that is worth. When you want to change a configuration parameter and roll it back without deploying new code, confext is genuinely a good tool.

What It Is Not

Knowing what extensions don't do matters as much as knowing what they do.

Not a package manager. There is no dependency scheme. An extension must carry every file it needs itself, except those already shipped in the host image. The documentation says so explicitly: it "should not be misunderstood as a generic software packaging framework".

Cannot reach outside /usr and /opt. The /etc and /var directories inside the extension image are not merged. When I put /etc/test.conf inside the extension, the file simply wasn't there after merging — silently ignored, with no warning, which I find the more insidious part. That does not mean "sysext can't ship configuration": vendor configuration under /usr/lib (systemd units, drop-ins, tmpfiles.d, sysusers.d) travels fine. What doesn't travel is /etc, the administrator's territory.

Provides no isolation. Files added via sysext appear as if they were part of the host image; they draw no security boundary. If you want to isolate services, what you're after is Portable Services — those ship their own libraries and genuinely decouple from the host. I covered a similar distinction on the container side in the Podman Quadlet post.

One more warning: during a refresh (unmerge + merge), there is a brief moment when neither the old nor the new overlayfs is mounted. Every resource the extension provides disappears in that instant — even for an extension that exists continuously. If a running service's binary comes from an extension, plan for this deliberately; and if the extension ships a systemd unit, its identity file needs EXTENSION_RELOAD_MANAGER=1, otherwise the service manager never sees the new unit file.

So what does an unmerge do to a process currently running from the extension? The answer matters to anyone following the "pack it up when you're done" recipe:

# /usr/bin/sleeper &          # binary comes from the extension
# systemd-sysext unmerge
Unmerged '/usr'.
# is the running process alive?     → yes
# can a new one be started?
bash: /usr/bin/sleeper: No such file or directory

Unmerge doesn't fail with "busy" and doesn't kill anything. Running processes keep seeing the old tree; new processes can't. Operationally reassuring, but sneaky for inventory purposes: systemd-sysext status can look perfectly clean while a process from the extension is still running on the machine.

Durability and Deployment: Boot, .raw and sysupdate

Merging by hand is fine for the experimentation phase. For a permanent setup there are three things to know.

First, automatic merging at boot happens via systemd-sysext.service and systemd-confext.service, and when those are enabled every installed extension is activated automatically — there is no concept of enabling or disabling them individually. The services are guaranteed to finish before basic.target, so extension files are ready by the time ordinary services start. To switch everything off at once you can put systemd.sysext=0 on the kernel command line (the parameter takes a boolean — don't leave it bare).

Disabling a single extension works in a way that looks odd at first: you place an empty directory with the same name into /etc/extensions/, shadowing the one in the lower-priority directory. It works — but the message it produces is misleading:

# mkdir -p /etc/extensions/tools     # mask tools.raw
# systemd-sysext merge
No suitable extensions found (1 ignored due to incompatible image(s)).

(The count is one because tools.raw was the only extension on the machine at that point.)

So masking is reported not as "disabled" but as an "incompatible image" — a deliberate mask and a broken identity file get the same sentence. Since there's nowhere to leave a comment next to it, keep your masks documented in your configuration management, so the you of six months from now doesn't read one as a fault.

Second, using a .raw disk image instead of a directory: an erofs, squashfs or ext4 file system, optionally GPT-labelled and Verity-protected. This is the right format for production deployment; directories are for development and emergency intervention. Producing the image isn't laborious — packing the same tree with mksquashfs is enough:

# mksquashfs /tmp/build /var/lib/extensions/tools.raw -noappend
# systemd-sysext list
NAME  TYPE PATH                          TIME
tools raw  /var/lib/extensions/tools.raw Fri 2026-09-04 05:42:37 UTC

# systemd-sysext merge
Using extensions 'tools.raw'.
Merged extensions into '/usr'.

Note that the extension's name now comes from the file name: the identity file inside tools.raw must be extension-release.tools. If the names don't match you get that tight-lipped "incompatible image" answer again.

There's one more wall you'll hit if you try to package a real distribution tree as-is: an extension may not ship /usr/lib/os-release. The reasoning is sound — merged into /usr, that file would override the host's OS identity. But the way it gets rejected is the high-water mark of the silence I keep complaining about in this post:

# cp /etc/os-release /var/lib/extensions/demo/usr/lib/os-release
# systemd-sysext merge
Failed to read metadata for image demo: No medium found

"No medium found." You are not looking for a floppy drive; you packaged the wrong file. When an image is rejected inexplicably, make this your second stop after the identity file.

Third, getting images onto machines. systemd-sysupdate does this with A/B semantics and .transfer files. Watch the version differences here: the suffix for transfer definitions changed from .conf to .transfer in v257 (the old one is still supported), and in v261, the current stable release, the timer units are still systemd-sysupdate.service / .timer. Those units are renamed to systemd-sysupdate-update.service / .timer in v262 — compatibility symlinks remain, but a good share of the examples on the internet are about to go stale.

There is also a path being abandoned: the experimental systemd-sysupdated D-Bus API will be removed in v263, with clients expected to talk to systemd-sysupdate directly over Varlink instead; updatectl will be reworked along those lines. If you are building automation on sysupdated today, don't.

To make the version question concrete: Debian 13 trixie ships systemd 257.13. Everything above works there, but v259's sysext.conf configuration file and v262's new unit names do not exist. Writing a runbook against upstream documentation without checking the version in your own repository is how you manufacture "why don't I have this" hours later on.

Security: No Signature Check by Default

There's a detail that must not be missed when discussing this mechanism in production. .raw images may optionally carry Verity authentication information — but the default image policy is this:

root=verity+signed+encrypted+unprotected+absent:usr=verity+signed+encrypted+unprotected+absent

The word unprotected in that list means unsigned, unverified images are accepted too. The unsigned image I built with mksquashfs above mounted into /usr without a word of protest for exactly this reason.

In practice this means: anyone who can write files under /var/lib/extensions/ can introduce binaries into that machine's /usr hierarchy, and do so without ever touching the package database. The directory is root-owned, so this is not a privilege escalation bug — but for an attacker who is already root, it is an extremely quiet persistence method. On the audit side, dpkg -l and file integrity checks won't show those binaries; systemd-sysext status will. Adding that command to your inventory and detection scripts is a five-minute job.

If you want to tighten things, --image-policy= lets you narrow the policy and drop the unprotected option; if your setup can produce signed images, that is the correct move. Don't overestimate its reach, though: image policy applies to disk images. Tightening it will not stop someone who drops a plain directory into /var/lib/extensions/ — a directory has no image to sign in the first place. The policy enforces your signing discipline; it does not close the directory path. systemd itself already uses a stricter default (root=signed+absent:usr=signed+absent) for the /.extra/sysext/ path in the initrd — so expecting signatures in a production flow isn't an enterprise luxury, it's the line the project itself prefers.

A Decision Framework

The questions I ask myself, in order:

• Do I need this tool permanently? If yes, put it in the base image, not in an extension.
• Will packages be installed on this machine? If yes, either merge the extension only for the duration of the intervention, or set up the /var/lib/extensions.mutable/usr/ → /usr/ symlink and merge with --mutable=auto (a unit drop-in is required for this to hold at boot).
• Are the binaries dynamically linked? If yes, don't use ID=_any; lock the extension to the host version.
• Am I shipping configuration or a binary? Configuration is confext's job, binaries are sysext's.
• Do I want isolation? Then neither — use Portable Services or a container.
• Is a service running from the extension? Account for the gap during refresh.
• Are the images signed? If not, add systemd-sysext status to your inventory and detection scripts.

Closing

The reason I like sysext isn't that it makes installing tools easy — apt install is already easy. The reason I like it is that it makes the easy thing reversible: choosing a change's lifetime deliberately, while you are making it. That is what keeps your inventory accurate a year later.

The real lesson, though, was a different one. I read the sentence "/usr becomes read-only during merge" and thought I had understood it. I only saw what it meant when dpkg smeared a half-finished installation across my face. A throwaway container cost me five minutes; learning the same lesson in production would have cost a great deal more.

Official Sources

The man pages below document the development version (the v262 tree); the commands in this post were tested against v257, the version Debian 13 ships. Compare against the version in your own repository.

• systemd-sysext(8) — Activates System Extension Images
• systemd-sysupdate(8) — Automatically Update OS or Other Resources
• sysupdate.d(5) — Transfer Definition Files for Automatic Updates
• systemd NEWS — release-by-release changelog
• Debian trixie systemd package version
• Linux kernel: overlayfs documentation