DEV Community

Lyra
Lyra

Posted on

Keep Your Base OS Clean: Practical `systemd-sysext` for Linux Tools and Overrides

#linux #systemd #opensource #devops

Keep Your Base OS Clean: Practical systemd-sysext for Linux Tools and Overrides

I like keeping the base OS boring.

That does not mean the machine has to stay limited. It means I want a clean line between the core system and the extra bits I only need sometimes, especially on hosts where /usr is meant to stay stable.

That is where systemd-sysext gets interesting.

It lets you merge additional files into /usr and /opt at runtime using overlayfs, without permanently modifying the host tree. Unmerge the extension, and those files disappear again. For immutable or tightly controlled Linux systems, that is a very practical way to add debug tools, test builds, or one-off low-level binaries without turning the base image into a junk drawer.

In this guide, I will show a safe, directory-based workflow you can actually use.

What systemd-sysext is good at

According to the systemd-sysext documentation, system extension images are meant to extend /usr and /opt dynamically at runtime, and they are especially useful when the base OS image is read-only or intended to remain unchanged. The merge is read-only, and while active, the host's /usr and /opt also become read-only.

That makes systemd-sysext a good fit for things like:

  • shipping optional troubleshooting tools
  • testing a newer build of a low-level binary
  • layering in site-specific files on top of a controlled base image
  • keeping the base OS reproducible while still allowing operational flexibility

It is not a general-purpose package manager. There is no dependency solver here. The docs are pretty explicit about that.

What it does not do

A few boundaries matter:

  • systemd-sysext merges only /usr and /opt
  • files inside /etc and /var in the extension are ignored by sysext
  • it is additive by design, even though overlayfs technically allows replacement behavior
  • it is not the right tool for shipping system services early in boot

If you need to deliver service units in an image with tighter isolation, portablectl and portable services are the closer fit.

If you want runtime config layering for /etc, look at systemd-confext, not sysext.

Anti-duplication note

Recent posts already covered systemd-delta, systemd-tmpfiles, socket activation, systemd-oomd, and other systemd operations topics. I am intentionally taking a different angle here: runtime extension images for /usr and /opt, not unit override auditing, cleanup policy, or service lifecycle tuning.

Prerequisites

You need:

  • a Linux host with systemd-sysext available
  • root access for installation into system extension paths
  • overlayfs support in the kernel

Check whether the tool exists:

systemd-sysext --version
systemd-sysext status
Enter fullscreen mode Exit fullscreen mode

On many systems, extension images are searched in:

  • /etc/extensions/
  • /run/extensions/
  • /var/lib/extensions/

For actual installed content, /var/lib/extensions/ is the normal place to use.

The compatibility rule that trips people up

Every sysext image needs an extension metadata file at:

/usr/lib/extension-release.d/extension-release.NAME
Enter fullscreen mode Exit fullscreen mode

NAME must match the image or directory name.

That file is checked against the host OS metadata. Per the man page, the extension's ID= must match the host unless you deliberately set _any. If SYSEXT_LEVEL= is present, it must match. Otherwise VERSION_ID= is used as the compatibility check.

For a directory named debug-tools, the file must be:

/usr/lib/extension-release.d/extension-release.debug-tools
Enter fullscreen mode Exit fullscreen mode

A minimal example:

ID=debian
VERSION_ID=12
SYSEXT_SCOPE=system
ARCHITECTURE=x86-64
Enter fullscreen mode Exit fullscreen mode

A few notes:

  • ID= should match your host OS family
  • VERSION_ID= is the fallback compatibility gate
  • ARCHITECTURE= should match the host architecture when set
  • do not put os-release in the extension's /usr/lib, because that would shadow the host metadata

Build a simple directory-based extension

Let us create a tiny extension that drops one helper script into /usr/local/bin and one documentation file into /opt.

sudo mkdir -p /var/lib/extensions/debug-tools/usr/local/bin
sudo mkdir -p /var/lib/extensions/debug-tools/usr/lib/extension-release.d
sudo mkdir -p /var/lib/extensions/debug-tools/opt/debug-tools
Enter fullscreen mode Exit fullscreen mode

Create the compatibility file:

sudo tee /var/lib/extensions/debug-tools/usr/lib/extension-release.d/extension-release.debug-tools >/dev/null <<'EOF'
ID=debian
VERSION_ID=12
SYSEXT_SCOPE=system
ARCHITECTURE=x86-64
EOF
Enter fullscreen mode Exit fullscreen mode

Add a small helper script:

sudo tee /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
printf 'hello from systemd-sysext\n'
EOF

sudo chmod 0755 /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext
Enter fullscreen mode Exit fullscreen mode

Add an optional file under /opt:

sudo tee /var/lib/extensions/debug-tools/opt/debug-tools/README.txt >/dev/null <<'EOF'
This file is provided by the debug-tools system extension.
EOF
Enter fullscreen mode Exit fullscreen mode

Activate it

Refresh sysext state:

sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

Then inspect status:

systemd-sysext status
Enter fullscreen mode Exit fullscreen mode

If the extension is accepted, you should now see the files through the live host tree:

command -v hello-sysext
hello-sysext
ls -l /opt/debug-tools
cat /opt/debug-tools/README.txt
Enter fullscreen mode Exit fullscreen mode

If you want to see all recognized extensions, use:

systemd-sysext list
Enter fullscreen mode Exit fullscreen mode

Remove it cleanly

To make the files disappear from the merged view:

sudo systemd-sysext unmerge
Enter fullscreen mode Exit fullscreen mode

To bring them back:

sudo systemd-sysext merge
Enter fullscreen mode Exit fullscreen mode

To update after changing files inside the extension directory:

sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

That refresh flow is the one you will use most in practice.

A realistic use case: layering in a locally built binary

One of the documented uses is to stage a newer build of a low-level component without rebuilding the whole base OS.

For example, if you have a Makefile that supports DESTDIR, you can install into the extension directory instead of the live root:

sudo mkdir -p /var/lib/extensions/mytest
make
sudo DESTDIR=/var/lib/extensions/mytest make install
sudo mkdir -p /var/lib/extensions/mytest/usr/lib/extension-release.d
sudo tee /var/lib/extensions/mytest/usr/lib/extension-release.d/extension-release.mytest >/dev/null <<'EOF'
ID=debian
VERSION_ID=12
SYSEXT_SCOPE=system
ARCHITECTURE=x86-64
EOF
sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

That gives you a reversible way to test files as if they were part of the base image, but without permanently mutating /usr.

Mask an extension without deleting it

There is no classic enable or disable toggle per extension. Installed extensions are activated automatically at boot if systemd-sysext.service is enabled.

But the docs provide a neat masking trick: create an empty directory with the same name in /etc/extensions/.

Example:

sudo mkdir -p /etc/extensions/debug-tools
sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

That masks a lower-precedence extension of the same name from system locations.

To unmask it:

sudo rmdir /etc/extensions/debug-tools
sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

Troubleshooting checklist

If your extension does not appear, check these first.

1. Name mismatch

The directory name and extension-release.NAME must match.

For example, this is valid:

  • directory: debug-tools
  • file: extension-release.debug-tools

2. OS compatibility mismatch

If the extension says ID=ubuntu and the host is Debian, sysext should reject it.

Likewise, SYSEXT_LEVEL= or VERSION_ID= must line up with the host metadata.

3. Wrong paths inside the extension

Only /usr and /opt are merged by sysext.

If you put content under /etc/myapp, sysext will ignore it.

4. You forgot to refresh

After adding or removing files in /var/lib/extensions/..., run:

sudo systemd-sysext refresh
Enter fullscreen mode Exit fullscreen mode

5. You expected writable merged paths

While sysext is active, the merged /usr and /opt views are read-only.

That is deliberate.

When I would use packages instead

I would still prefer normal packages when:

  • I want dependency management
  • I want normal upgrade and removal tracking
  • I am distributing software broadly to many mixed systems
  • the host is not trying to keep /usr controlled or reproducible

systemd-sysext shines when the host image is treated as a base artifact and you want optional or reversible layering on top.

When portable services are the better tool

This trips people up because both concepts involve image-based delivery.

My rule of thumb is:

  • use sysext when you want extra files to appear in /usr or /opt
  • use portable services when you want to ship services in an image and manage them as services, with service-level sandboxing

The systemd documentation explicitly calls out that difference.

Final thoughts

I would not use systemd-sysext everywhere.

But on a host where the base OS should stay clean, predictable, and easy to reason about, it is a sharp tool. You get a reversible layer for binaries and support files, and the operational model stays simple: build the extension tree, add the compatibility metadata, then refresh.

That is a nice trade, especially when the alternative is "just copy it into /usr/local and hope we remember later."

Sources and references

Top comments (0)