Stop Shipping Full Containers for Host Extensions: Practical systemd Portable Services with portablectl

You already know the two common answers for shipping extra software onto a Linux host:

1. Full OS containers (systemd-nspawn, LXC) — their own PID 1, their own network stack, their own lifecycle.
2. App containers (Podman/Docker) — OCI images, separate runtime, often a separate mental model from systemctl.

There is a third option that sits in a useful gap: systemd portable services.

A portable service is just an OS tree (directory, btrfs subvolume, or .raw disk image) that carries:

• the service binary and its libraries
• one or more systemd unit files
• a normal os-release file

You attach that image with portablectl. systemd copies the matching units onto the host, pins them to RootDirectory= / RootImage=, and applies a security profile. From then on the payload is a normal host unit: systemctl start, journald, cgroups, timers, sockets — same tools you already use.

This post is a practical operator guide: build a minimal image, attach it, pick a profile, upgrade with reattach, keep state on the host, and tear it down cleanly.

What portable services are (and are not)

From the Portable Services design doc:

• They do not invent a new image format. Directory trees and GPT/raw images already work.
• They do not run your app as PID 1. It is a normal service process under host systemd.
• They do not fully isolate like Docker by default. The point is host integration with optional lockdown, not a separate world.
• They do bundle dependencies and apply stricter default sandboxing via profiles.

Approach	Runs as	Root FS	Typical control plane
Host package / unit	host service	host root	systemctl
Portable service	host service	image root (RootImage= / RootDirectory=)	portablectl + systemctl
systemd-nspawn -b	container PID 1	image root	machinectl / nspawn
Podman/Docker	container runtime	OCI layers	podman / docker

Use portable services when the software should feel like a host extension (agent, exporter, appliance sidecar, “super-privileged container” style workload) while still shipping its own userspace tree.

Prerequisites

• systemd 239+ with portable support (portablectl, systemd-portabled.service). On many modern distros this ships with systemd; Arch’s man page documents systemd 261-era behavior.
• Root (or polkit) for attach/detach.
• Ability to build a Linux userspace tree (debootstrap, dnf --installroot=, or mkosi).

Check the CLI:

command -v portablectl
portablectl --version
systemctl status systemd-portabled.service --no-pager

If portablectl is missing, install/enable the portable bits for your distro (package naming varies; on some systems the binary historically lived under /usr/lib/systemd/ before landing on $PATH).

Image search paths

portablectl list looks in:

• /var/lib/portables/ (preferred store for large images)
• /etc/portables/
• /run/portables/
• /usr/local/lib/portables/
• /usr/lib/portables/

Recommendation from portablectl(1): put real images under /var/lib/portables/ and only put symlinks in /etc/portables/ or /run/portables/.

Unit name prefix rules (this trips people up)

When you attach foobar_47.11.raw, the default unit prefix is foobar (filename without .raw, truncated at the first _).

Only units whose names match that prefix followed by ., -, or @ are copied. Examples that match:

• foobar.service
• foobar-agent.service
• foobar@.service
• foobar.socket / foobar.timer / foobar.path / foobar.target

Units that do not match the prefix are ignored. You can override prefixes on the command line after the image name.

Optional hardening in the image’s os-release:

PORTABLE_PREFIXES=foobar
PORTABLE_SCOPE=system

PORTABLE_PREFIXES= documents and constrains allowed prefixes (especially useful with authenticated/verity images). PORTABLE_SCOPE= can be system, user, or any (default implies system).

Lab: build a minimal directory image

You do not need a full distro. The design doc’s minimal tree is enough for a static binary. Here is a small directory image you can attach without building a GPT disk.

sudo mkdir -p /var/lib/portables
IMG=/var/lib/portables/minagent_1.0.0
sudo rm -rf "$IMG"
sudo mkdir -p \
  "$IMG/usr/bin" \
  "$IMG/usr/lib/systemd/system" \
  "$IMG/usr/lib" \
  "$IMG/etc" \
  "$IMG/proc" "$IMG/sys" "$IMG/dev" "$IMG/run" "$IMG/tmp" "$IMG/var/tmp"

# Placeholder "daemon": write a heartbeat then sleep forever.
# Prefer a real static binary in production; this is a lab stand-in.
sudo tee "$IMG/usr/bin/minagentd" >/dev/null <<'EOF'
#!/bin/sh
echo "minagentd starting on $(hostname) at $(date -Is)"
while true; do
  echo "minagentd heartbeat $(date -Is)"
  sleep 30
done
EOF
sudo chmod 0755 "$IMG/usr/bin/minagentd"

sudo tee "$IMG/usr/lib/systemd/system/minagent.service" >/dev/null <<'EOF'
[Unit]
Description=Minimal portable agent example
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/minagentd
# Writable state lives on the host, not inside a read-only image.
StateDirectory=minagent
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo tee "$IMG/usr/lib/os-release" >/dev/null <<'EOF'
NAME="minagent portable"
ID=minagent
VERSION_ID=1.0.0
PORTABLE_PREFIXES=minagent
PORTABLE_SCOPE=system
EOF

# Required mount points / bind targets (empty is fine; host over-mounts them).
sudo : > "$IMG/etc/resolv.conf"
sudo : > "$IMG/etc/machine-id"

Important reality check for /bin/sh labs

A pure static tree is ideal. A #!/bin/sh script needs a shell and dynamic loader inside the image (or a static busybox). For a quick lab on the same distro family, clone a thin root with debootstrap --variant=minbase or copy the interpreter + libs. Production images are usually built with mkosi / installroot so the tree is self-contained.

A debootstrap-shaped lab skeleton:

# Debian/Ubuntu-shaped example — adjust suite/mirror for your host.
sudo debootstrap --variant=minbase bookworm /var/lib/portables/minagent_1.0.0 http://deb.debian.org/debian
# Then install your unit + binary paths as above, and set os-release PORTABLE_* fields.

Or follow the official walkthrough repo that builds a C daemon with mkosi: systemd/portable-walkthrough.

Inspect before you attach

sudo portablectl inspect /var/lib/portables/minagent_1.0.0
sudo portablectl inspect --cat /var/lib/portables/minagent_1.0.0

You should see os-release metadata and the matching unit list (minagent.service).

Attach, enable, start

# Persistent attach (units under /etc/systemd/system.attached/)
sudo portablectl attach --profile=default --enable --now /var/lib/portables/minagent_1.0.0

# Or temporary until reboot:
# sudo portablectl attach --runtime --profile=default --now /var/lib/portables/minagent_1.0.0

What attach does (portablectl(1) / design doc):

1. Copies matching .service / .socket / .target / .timer / .path units into /etc/systemd/system.attached/ (or /run/... with --runtime).
2. Writes a drop-in (20-portable.conf) with RootDirectory= or RootImage=, plus Environment=PORTABLE=... and LogExtraFields=PORTABLE=....
3. Links a profile drop-in (10-profile.conf).
4. Symlinks the image into the portable search path if needed.
5. Reloads the manager (unless --no-reload).

Verify:

sudo portablectl is-attached minagent_1.0.0
sudo portablectl list
systemctl status minagent.service --no-pager
systemctl cat minagent.service
ls -la /etc/systemd/system.attached/minagent.service.d/

Expected attachment states include: detached, attached, attached-runtime, enabled, enabled-runtime, running, running-runtime.

Profiles: the real product feature

Profiles are host-local drop-ins under:

• /usr/lib/systemd/portable/profile/<name>/service.conf
• /etc/systemd/portable/profile/<name>/ for custom profiles

Shipped profiles:

Profile	Intent
default	Medium lockdown; journal + D-Bus + IP networking allowed
nonetwork	Like default, but PrivateNetwork=yes and IPAddressDeny=any
strict	Tightest stock profile: no caps, AF_UNIX only, no net, low TasksMax
trusted	Minimal restrictions; effectively full host trust

Select at attach time:

sudo portablectl attach --profile=strict /var/lib/portables/minagent_1.0.0

Excerpt from the stock default profile (systemd v256 tree):

[Service]
MountAPIVFS=yes
BindReadOnlyPaths=/dev/log /run/systemd/journal/socket /run/systemd/journal/stdout
BindReadOnlyPaths=/etc/machine-id
BindReadOnlyPaths=-/etc/resolv.conf
BindReadOnlyPaths=/run/dbus/system_bus_socket
DynamicUser=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictNamespaces=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native
# plus a reduced CapabilityBoundingSet= ...

Excerpt from strict:

[Service]
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX
PrivateNetwork=yes
IPAddressDeny=any
NoNewPrivileges=yes
TasksMax=4
# ... same Protect*/Private* baseline as default ...

Operator rule: the image vendor ships code; the host admin chooses the profile. That split is intentional.

Need host files or sockets? Use normal unit directives in the image unit (or a host drop-in via systemctl edit):

[Service]
BindReadOnlyPaths=/run/myservice.sock
BindPaths=/var/lib/minagent-extra

Writable data with immutable images

All profiles except trusted lean on ProtectSystem=strict. Keep the image read-only and put mutable data on the host with StateDirectory= / CacheDirectory= / LogsDirectory= / RuntimeDirectory= in the unit.

# After the unit has run once under DynamicUser/StateDirectory:
ls -la /var/lib/minagent
journalctl -u minagent.service -n 50 --no-pager

Optional:

sudo portablectl read-only minagent_1.0.0 yes

Upgrade with less downtime: reattach

Ship a new version as minagent_1.0.1 (underscore versioning). reattach detaches the old attachment and attaches the new image; with --now it restarts updated units instead of a blunt stop+start cycle, which helps preserve runtime state such as the file descriptor store when applicable.

# Build/copy new tree or .raw named minagent_1.0.1
sudo portablectl reattach --profile=default --now /var/lib/portables/minagent_1.0.1
sudo portablectl is-attached minagent_1.0.1

Partial matching: only the part before the first _ must match for upgrade pairing.

Keep the image path stable while attached. Moving the file breaks RootImage= / RootDirectory= in the generated drop-in.

Extensions (sysext/confext layering)

Since v249, portablectl attach --extension PATH ... can stack OverlayFS layers: a shared base runtime image plus thin app extensions. Extension images need matching extension-release metadata (ID= + SYSEXT_LEVEL=/VERSION_ID= or confext equivalents). Same extensions, same order, are required on detach.

# Conceptual pattern from the design doc:
sudo portablectl attach \
  --extension foobar_0.7.23.raw \
  debian-runtime_11.1.raw \
  foobar

Use this when many agents share one distro runtime and you only want to ship the app layer repeatedly.

Logging fields you can query

Portable attach injects structured journal fields such as:

• PORTABLE=
• PORTABLE_NAME_AND_VERSION= (from IMAGE_ID/ID + IMAGE_VERSION/VERSION_ID/BUILD_ID)
• with extensions: PORTABLE_ROOT=, PORTABLE_EXTENSION=, PORTABLE_*_NAME_AND_VERSION=

journalctl FIELD=PORTABLE=minagent_1.0.0 -n 20 --no-pager
# or filter by unit as usual
journalctl -u minagent.service -f

Instantiation

No special portable API: ship foobar@.service and instantiate after attach.

sudo portablectl attach foobar_0.7.23.raw
sudo systemctl enable --now [e-mail]
sudo systemctl enable --now [e-mail]

Detach and clean up

# Stop + disable + detach
sudo portablectl detach --enable --now minagent_1.0.0

# systemd v256+: also remove host state/logs/cache/runtime dirs for the service
sudo portablectl detach --clean --now minagent_1.0.0

# Remove only a search-path symlink/image entry (does not follow and delete the target if it is a symlink)
sudo portablectl remove minagent_1.0.0

Confirm:

sudo portablectl is-attached minagent_1.0.0   # detached
systemctl status minagent.service --no-pager  # not-found / inactive as expected

Copy mode and image policy notes

• --copy=copy|symlink|auto|mixed controls whether attached units/profiles/images are copied or symlinked. Raw disk images force copy when symlink is impossible. mixed (v256+) symlinks profile drop-ins but copies units/images.
• On attach, systemd can generate an image policy that pins the content discovered at attach time so a later swap cannot silently drop protections such as dm-verity without reinstall. See systemd.image-policy(7).

Operational checklist

1. Build a self-contained tree or .raw with matching unit prefix + os-release.
2. portablectl inspect before attach.
3. Choose profile deliberately (default / nonetwork / strict / custom / trusted).
4. Prefer StateDirectory= over writable images.
5. Store images under /var/lib/portables/ and version with _.
6. Upgrade via reattach --now.
7. Tear down with detach --now and --clean when you want host state gone.
8. Treat portable units like any other units for cgroup limits: systemctl set-property minagent.service MemoryMax=512M CPUQuota=50%.

Boundaries (so this does not blur into other tools)

• Not a replacement for Podman Quadlet app containers when you need OCI registries, rootless stacks, and Kubernetes-shaped workflows.
• Not a replacement for systemd-nspawn -b when you need a full guest OS with its own init and machine lifecycle.
• Not a substitute for whole-disk integrity (dm-verity) or per-file authenticity (fs-verity); those compose with portable images when you ship .raw + verity.
• Not soft-reboot / image A/B host update logic; portable services extend a live host, they do not replace host OS update strategy.

Sources and references

• Portable Services Introduction (systemd.io)
• portablectl(1) — Arch man pages
• systemd-portabled.service(8)
• Walkthrough for Portable Services — Lennart Poettering
• systemd/portable-walkthrough example repo
• Stock profiles in systemd source: default/service.conf, strict/service.conf, nonetwork/service.conf
• Related: mkosi, Discoverable Partitions Specification

Wrap-up

Portable services are the “missing middle” for Linux operators who want bundled dependencies and profile-based sandboxing without leaving the systemd control plane. Build an image, portablectl attach --profile=... --enable --now, manage it with systemctl and journald, upgrade with reattach, and detach without scattering random unit files by hand.

If your next component is “basically a host service, but I refuse to pollute the host rootfs with its library tree,” try a portable image before you reach for a full guest OS container.