DEV Community: Lyra

Stop Putting Secrets in Environment Variables: Practical systemd-creds on Linux

Lyra — Wed, 30 Sep 2026 05:02:13 +0000

Passwords in Environment= lines, API tokens in world-readable unit drop-ins, and EnvironmentFile=/etc/myapp.env that every process on the host can cat once it gets a shell — that pattern still shows up in homelab and production units alike.

systemd has a better primitive: credentials. They are limited-size binary or text blobs that the service manager loads at activation time, decrypts if needed, places in a private directory, and exposes to the service via $CREDENTIALS_DIRECTORY. The operator tool is systemd-creds.

This guide is operational. Commands and behavior come from systemd-creds(1), the Credentials section of systemd.exec(5), and the upstream System and Service Credentials document. Examples target systemd 250+ (encryption tooling); user-scoped encrypt/decrypt needs 256+. The host used for command verification here runs systemd 257.

Why not just use an env var?

Approach	Problem
`Environment=SECRET=…`	Unit files are world-readable on disk and via D-Bus; children inherit the env
`EnvironmentFile=`	Same visibility problem; easy to leave mode `0644`
Plain file under `/etc`	Readable by any process that can open it; not scoped to the unit
SOPS/age in Git	Great for repo secrets; still need a runtime hand-off into the service
Clevis/Tang / LUKS	Disk unlock, not per-service secret injection

Credentials fix the runtime side:

Acquired at activation, released on deactivation, immutable while the service runs
Files under $CREDENTIALS_DIRECTORY, mode-restricted to the service UID (and root)
Optionally AES-256-GCM encrypted/authenticated at rest (TPM2 and/or host secret)
Preferable to env for secrets; binary-safe; ~1 MB accumulated size limit per unit
Work cleanly with RootDirectory= / RootImage= / portable services (host path not required inside the image)

Prerequisites

command -v systemd-creds
systemd-creds --version
# encrypt/decrypt/list/cat/setup: systemd 250+
# --user / --uid= encrypt: 256+

# Optional: see whether TPM2 is usable for credential protection
systemd-creds has-tpm2 || true

You need root (or equivalent) for host-key setup and for system-unit examples. Transient labs use systemd-run.

The five unit settings (and when to use each)

From systemd.exec(5):

Setting	Source	Sensitive?
`LoadCredential=ID[:PATH]`	File, AF_UNIX socket, or relative search path	Path should be protected; data stays plaintext on disk
`LoadCredentialEncrypted=ID[:PATH]`	Encrypted credential file/socket	Yes — decrypt at activation
`SetCredential=ID:VALUE`	Literal in the unit file	No for secrets (public keys, IDs only)
`SetCredentialEncrypted=ID:VALUE`	Encrypted blob literal in the unit	Yes — safe to paste ciphertext into drop-ins
`ImportCredential=GLOB`	Propagate system credentials (or search stores) by name/glob	Depends on how the system got them

Search paths for relative sources (also used by ImportCredential=):

Plain: /etc/credstore/, /run/credstore/, /usr/lib/credstore/ (plus credentials already passed into the system)
Encrypted: /etc/credstore.encrypted/, /run/credstore.encrypted/, /usr/lib/credstore.encrypted/
Per-user manager: $XDG_CONFIG_HOME/credstore/, $XDG_RUNTIME_DIR/credstore/, $HOME/.local/lib/credstore/ (and .encrypted counterparts)

Inside the service:

$CREDENTIALS_DIRECTORY/<ID>   # primary interface
%d/<ID>                       # unit-file specifier for Environment=
${CREDENTIALS_DIRECTORY}/<ID> # ExecStart= interpolation
/run/credentials/<unit>       # system services only — prefer $CREDENTIALS_DIRECTORY

Lab 1 — Plain LoadCredential with a transient unit

Prove the plumbing before encryption:

# As root
echo -n 'hello-from-disk' > /root/demo-plain.txt
chmod 600 /root/demo-plain.txt

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  systemd-creds cat demo
# → hello-from-disk

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  systemd-creds list
# Shows name, size, and security state (secure/weak/insecure)

systemd-creds list and cat read $CREDENTIALS_DIRECTORY in the current execution context — that is why the lab runs them inside systemd-run with LoadCredential= attached.

Reference the path from a shell-style ExecStart without putting the secret in the environment:

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  bash -c 'wc -c < "$CREDENTIALS_DIRECTORY/demo"; sha256sum "$CREDENTIALS_DIRECTORY/demo"'

Lab 2 — Encrypt with the host key

Default encrypt picks a key mode automatically: TPM2 when present and not in a container, host secret when /var/lib/systemd/ is persistent media, and both when both apply (host+tpm2). On VMs or hosts without a usable TPM, force the host key:

# Ensures /var/lib/systemd/credential.secret exists (also done implicitly on encrypt)
systemd-creds setup

# Encrypt stdin → file. Output basename becomes the embedded credential name.
echo -n 's3cr3t-db-password' | systemd-creds encrypt --with-key=host - /etc/credstore.encrypted/db-password.cred
chmod 600 /etc/credstore.encrypted/db-password.cred
# Directory may need creating first:
# mkdir -p /etc/credstore.encrypted && chmod 700 /etc/credstore.encrypted

# Round-trip check (name must match the embedded name = filename by default)
systemd-creds decrypt /etc/credstore.encrypted/db-password.cred
# → s3cr3t-db-password

# Wipe any leftover plaintext you used during creation
shred -u /root/demo-plain.txt 2>/dev/null || true

Key modes from systemd-creds(1) (--with-key= / -H / -T):

Mode	Binds decryption to
`host` (`-H`)	`/var/lib/systemd/credential.secret` (root-only)
`tpm2` (`-T`)	Local TPM2 only
`host+tpm2`	Both (typical default when both available)
`auto`	Default selection logic
`auto-initrd`	TPM2 if present, else null-style fallback — for initrd/generator use where `/var` is not mounted
`null`	Fixed empty key — no confidentiality/authenticity; restricted acceptance on Secure Boot + TPM systems

Encryption algorithm: AES-256-GCM, keyed from a SHA-256 derivation of the selected secret(s). Ciphertext is Base64.

Initrd / early-boot note: if a credential must decrypt before /var is available, do not bind it to the host secret. Use --with-key=tpm2 or auto-initrd.

Lab 3 — LoadCredentialEncrypted in a real unit drop-in

# Example consumer: a oneshot that only prints that the secret length is correct
cat >/etc/systemd/system/cred-demo.service <<'EOF'
[Unit]
Description=Credential demo service

[Service]
Type=oneshot
# Relative path → searched under credstore.encrypted (and friends)
LoadCredentialEncrypted=db-password
ExecStart=/usr/bin/bash -c 'test -n "$CREDENTIALS_DIRECTORY"; install -m 0400 "$CREDENTIALS_DIRECTORY/db-password" /run/cred-demo-check; wc -c /run/cred-demo-check; rm -f /run/cred-demo-check'
# Optional hardening that also implies PrivateMounts= on many distros:
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
EOF

systemctl daemon-reload
systemctl start cred-demo.service
systemctl status cred-demo.service --no-pager
journalctl -u cred-demo.service -n 20 --no-pager

Explicit path form (same effect):

LoadCredentialEncrypted=db-password:/etc/credstore.encrypted/db-password.cred

Failed decrypt/authenticate for LoadCredentialEncrypted= / SetCredentialEncrypted= fails the unit. Failed decrypt for something only pulled via ImportCredential= is skipped with a warning (credential absent).

Lab 4 — Paste ciphertext into the unit with SetCredentialEncrypted

When you want the secret in the drop-in (no separate .cred file), generate a pasteable line:

systemd-ask-password -n | systemd-creds encrypt --with-key=host --name=mysql-password -p - -

-p / --pretty prints a SetCredentialEncrypted=mysql-password: \ block. Pipe it into a drop-in:

mkdir -p /etc/systemd/system/myapp.service.d
systemd-ask-password -n | (
  echo '[Service]'
  systemd-creds encrypt --with-key=host --name=mysql-password -p - -
) >/etc/systemd/system/myapp.service.d/50-password.conf
chmod 600 /etc/systemd/system/myapp.service.d/50-password.conf
systemctl daemon-reload
systemctl restart myapp.service

The unit file remains world-readable in the usual case; the ciphertext is not useful without the host secret and/or TPM. That is the point of SetCredentialEncrypted= versus plaintext SetCredential=.

Name embedding: the credential name is stored inside the encrypted blob. Renaming mysql-password.cred to other.cred makes decrypt fail unless you override with --name=. Empty --name= disables the check (rarely what you want).

Lab 5 — Wire apps that do not know $CREDENTIALS_DIRECTORY yet

Many daemons want a path on the CLI or a single env var pointing at a file — not an env var containing the secret itself.

[Service]
User=myapp
LoadCredentialEncrypted=api-token
# Path-only env (the secret stays in the file; children do not inherit the bytes as env)
Environment=API_TOKEN_FILE=%d/api-token
ExecStart=/usr/local/bin/myapp --token-file ${CREDENTIALS_DIRECTORY}/api-token

%d expands to the credentials directory in unit settings; ${CREDENTIALS_DIRECTORY} works in Exec*= command lines. Prefer these over copying secrets into Environment=SECRET=….

For software that insists on reading /run/credentials/myapp.service/api-token, that path works for system units, but $CREDENTIALS_DIRECTORY is the portable interface (also correct for systemd --user).

Lab 6 — Propagate system credentials with ImportCredential

Container managers, hypervisors, and cloud IMDS can pass credentials into PID 1. Enumerate them:

systemd-creds --system list
systemd-creds --system cat some-name

Examples of ingress (from the credentials documentation):

systemd-nspawn --set-credential=name:value / --load-credential=
QEMU SMBIOS type 11: io.systemd.credential:name=value or io.systemd.credential.binary:name=<base64>
Kernel cmdline: systemd.set_credential= / systemd.set_credential_binary= (visible via /proc/cmdline — not for secrets)
ESP credentials via systemd-stub
/run/credentials/@initrd/ import on initrd → host transition

Consume in a service:

[Service]
ImportCredential=mycred
# or rename:
# ImportCredential=my.original.cred:my.renamed.cred
# or glob:
# ImportCredential=app.*
ExecStart=/usr/bin/systemd-creds cat mycred

Well-known system credentials (provisioning, not day-2 app secrets) include:

passwd.hashed-password.<user>, passwd.plaintext-password.<user>, passwd.shell.<user> → systemd-sysusers
firstboot.locale, firstboot.keymap, firstboot.timezone, … → systemd-firstboot
tmpfiles.extra → tmpfiles
vmm.notify_socket → PID 1 READY notification (e.g. vsock:CID:PORT)
ssh.authorized_keys.root (IMDS import path)

See systemd.system-credentials(7) for the full inventory. Conditional start: ConditionCredential= / AssertCredential=.

User services (systemd 256+)

Encrypt for the per-user manager so the key material incorporates UID, username, and machine-id:

echo -n 'user-secret' | systemd-creds encrypt --user --uid=self - ~/credstore.encrypted/user-secret.cred
# Load from a user unit with LoadCredentialEncrypted=user-secret
# and store under ~/.config/credstore.encrypted/ or the paths listed above

Operational checklist

Create ciphertext with systemd-creds encrypt (--with-key=host in labs without TPM; default auto on real hardware).
Store under /etc/credstore.encrypted/ (mode 0700 dir, 0600 files) or embed with -p → SetCredentialEncrypted=.
Declare LoadCredentialEncrypted= / ImportCredential= on the unit; avoid plaintext Environment= for secrets.
Consume via $CREDENTIALS_DIRECTORY or %d path references.
Sandbox with ProtectSystem=, PrivateTmp=, ProtectHome=, or at least PrivateMounts= so other units cannot see the credentials mount.
Rotate by re-encrypting and restarting the unit; ciphertext is immutable for the life of an activation.
Backup /var/lib/systemd/credential.secret with the same care as disk-encryption keys if you rely on host binding — losing it bricks host-bound credentials.
Expiry (optional): systemd-creds encrypt --not-after=… embeds a hard fail-after timestamp checked at decrypt.

Priority when the same ID is configured multiple ways (from systemd.exec(5)): LoadCredential* / ImportCredential= win over SetCredential*. SetCredential= is a useful default that is overridden when a real credential appears.

Boundaries (what this article is not)

Not SOPS + age in Git (repo encryption / GitOps). Credentials are the node-local runtime injection layer after deploy.
Not Clevis + Tang NBDE or systemd-cryptenroll (volume unlock at boot).
Not systemd-homed per-user home encryption.
Not a full replacement for a corporate secret manager — but an excellent native fit for machine-local service secrets on systemd hosts.
Not varlinkctl / D-Bus service APIs, linger/user managers, or image tooling from recent posts — only the credential object model and systemd-creds CLI.

Cleanup

systemctl stop cred-demo.service 2>/dev/null || true
rm -f /etc/systemd/system/cred-demo.service
rm -f /etc/systemd/system/myapp.service.d/50-password.conf
rm -f /etc/credstore.encrypted/db-password.cred
systemctl daemon-reload
# Keep /var/lib/systemd/credential.secret unless you intend to invalidate host-bound creds

References

systemd-creds(1) — list, cat, setup, encrypt, decrypt, key modes, examples
systemd.exec(5) — Credentials — LoadCredential*, SetCredential*, ImportCredential=, $CREDENTIALS_DIRECTORY, size limit
System and Service Credentials — design goals, encryption, ingress paths, well-known credentials, search paths
systemd.system-credentials(7) — well-known system credential names
systemd-nspawn(1) — --set-credential= / --load-credential=

Stop putting long-lived secrets in Environment= and world-readable env files. Encrypt once with systemd-creds, load with LoadCredentialEncrypted= or SetCredentialEncrypted=, and let the service read a file from $CREDENTIALS_DIRECTORY like any other path-based secret API — with activation-time decrypt and per-unit isolation included.

Stop Guessing systemd Service APIs: Practical varlinkctl on Linux

Lyra — Tue, 29 Sep 2026 05:02:08 +0000

You need one clean answer from a local service: resolve a name, describe the host, extend a PCR, or talk to a custom tool that already speaks JSON on stdin/stdout. The usual options are brittle: scrape systemctl text, write a one-off D-Bus client, or invent yet another Unix-socket protocol.

Modern systemd components expose Varlink interfaces instead. varlinkctl is the operator-facing client for those interfaces: discover what a service implements, print the interface IDL, call methods with JSON arguments, stream multi-reply methods, tunnel calls over SSH, and even wrap a sandboxed command as a Varlink service.

This guide is operational. Commands and behavior come from varlinkctl(1), the UAPI.20 Varlink IPC specification, and the Debian/unstable systemd man pages for the services used in the labs (notably systemd-resolved).

What Varlink is (and is not)

Piece	Job
Varlink	JSON method calls over a stream (typically `AF_UNIX`), with a typed interface definition language
UAPI.20	The UAPI Group spec consolidating protocol + transport bindings
`varlinkctl`	CLI to introspect and invoke Varlink services (systemd 255+)
`busctl` / D-Bus	Parallel IPC stack; still widely used, different wire format and tooling
`systemctl` text	Human UI — fine for shells, awkward as a stable machine API

Varlink messages are plain JSON objects. On stream transports each message is NUL-terminated. Services describe themselves at runtime, so clients can list interfaces and fetch the IDL without a separate schema package.

You do not need to replace every D-Bus workflow. Use varlinkctl when the component already speaks Varlink (resolved, hostnamed, many systemd helpers) or when you want a JSON-friendly, socket-activated service of your own.

Prerequisites

command -v varlinkctl
varlinkctl --version
# Added in systemd 255; several commands below need 257–262 features.

# Labs that call resolved need the stub/service running:
systemctl is-active systemd-resolved.service
ls -l /run/systemd/resolve/io.systemd.Resolve

Privileged examples (PCR extend, some hostnamed paths) need root or an authorized policy. DNS resolve calls as a normal user usually work when resolved is active and the socket is accessible.

Address forms you will actually use

varlinkctl accepts several service address syntaxes (from varlinkctl(1)):

Form	Meaning
`unix:/run/path.sock` or `/run/path.sock`	Connect to an `AF_UNIX` stream socket
`@name` after `unix:`	Abstract-namespace socket
`exec:/usr/lib/systemd/tool`	Fork the binary and speak Varlink on the passed socket
`ssh-unix:host:/run/...`	OpenSSH ≥ 9.4 path to a remote `AF_UNIX` socket
`ssh-exec:host:cmdline`	Run a remote command and speak Varlink on its stdio
Relative `./socket` or `./binary`	Local path forms (must start with `/` or `./`)

Convenience: a bare absolute socket path or executable path is enough when the target is local.

Lab 1 — Inventory a live service (resolved)

# General metadata
varlinkctl info /run/systemd/resolve/io.systemd.Resolve

# Interface names only
varlinkctl list-interfaces /run/systemd/resolve/io.systemd.Resolve

# Method names (list-methods added in 257)
varlinkctl list-methods /run/systemd/resolve/io.systemd.Resolve

# Full IDL for the primary interface
varlinkctl introspect /run/systemd/resolve/io.systemd.Resolve io.systemd.Resolve

Typical info fields include vendor, product, version, URL, and the interface list (io.systemd, io.systemd.Resolve, org.varlink.service, …).

Pretty JSON for scripts:

varlinkctl info /run/systemd/resolve/io.systemd.Resolve -j
varlinkctl list-methods /run/systemd/resolve/io.systemd.Resolve --json=short

-j is “pretty when interactive, short when piped”; --json=pretty|short forces the mode.

Lab 2 — Call a method with JSON arguments

Resolve a hostname through resolved’s ResolveHostname method (example shape from varlinkctl(1)):

varlinkctl call \
  /run/systemd/resolve/io.systemd.Resolve \
  io.systemd.Resolve.ResolveHostname \
  '{"name":"systemd.io","family":2}' \
  -j

Notes that prevent foot-guns:

Method names are fully qualified: interface.Method.
Parameters are a JSON object. Use {} for empty input.
If you omit the arguments parameter, varlinkctl reads JSON from STDIN.
Replies are JSON objects on STDOUT — pipe to jq when you want fields only.

varlinkctl call \
  /run/systemd/resolve/io.systemd.Resolve \
  io.systemd.Resolve.ResolveHostname \
  '{"name":"systemd.io","family":2}' \
  --json=short \
| jq -r '.addresses[]? | "\(.family) \(.address)"'

family: 2 is AF_INET in the usual Linux numbering; adjust if you want IPv6 (10 / AF_INET6) or leave the field out when the interface allows defaults (check the IDL from introspect).

Lab 3 — Multi-reply methods, collection, and oneway

Some methods stream updates or enumerate objects. Flags from varlinkctl(1):

# Expect a sequence of replies (JSON-SEQ). Default call timeout is still 45s.
varlinkctl call --more ADDRESS INTERFACE.Method '{"...":"..."}'

# Same idea, but keep listening (timeout disabled) — shortcut -E
varlinkctl call -E ADDRESS INTERFACE.Method '{}'

# Gather every reply into one JSON array
varlinkctl call --collect ADDRESS INTERFACE.Method '{}'

# Fire-and-forget (no reply expected)
varlinkctl call --oneway ADDRESS INTERFACE.Method '{"...":"..."}'

Timeout control:

# Default 45s; disable for long subscriptions
varlinkctl call --more --timeout=infinity ADDRESS INTERFACE.Method '{}'

# Treat a specific Varlink error as success (257+)
varlinkctl call \
  --graceful=org.varlink.service.InvalidParameter \
  ADDRESS INTERFACE.Method '{"experimental":true}'

Use --more with a sane timeout for finite enumerations; use -E / --timeout=infinity only for true subscriptions so a stuck peer cannot hang a cron job forever.

Lab 4 — Exec targets and helper binaries

Not every Varlink peer is a long-running daemon socket. Some tools speak Varlink when executed. The man page demonstrates systemd-pcrextend:

# Inspect the executable as a Varlink service (requires privileges for real PCR ops)
sudo varlinkctl info /usr/lib/systemd/systemd-pcrextend
sudo varlinkctl introspect /usr/lib/systemd/systemd-pcrextend io.systemd.PCRExtend

# Example method shape from the man page — only on systems where PCR extend is appropriate
# sudo varlinkctl call /usr/lib/systemd/systemd-pcrextend \
#   io.systemd.PCRExtend.Extend '{"pcr":15,"text":"foobar"}'

exec: form is equivalent when you want to be explicit:

sudo varlinkctl info exec:/usr/lib/systemd/systemd-pcrextend

Treat PCR/measurement labs as maintenance-window work on hosts where measured boot policy is intentional. Do not extend PCRs on production machines as a casual test.

Lab 5 — Remote calls over SSH

When the remote side has OpenSSH 9.4+ (for ssh-unix:) and the socket path exists:

# Talk to hostnamed on a remote machine via its AF_UNIX socket
# varlinkctl call ssh-unix:somehost:/run/systemd/io.systemd.Hostname \
#   io.systemd.Hostname.Describe '{}' -j

# Or run a Varlink-capable binary on the remote stdio path
# varlinkctl call ssh-exec:somehost:systemd-creds \
#   org.varlink.service.GetInfo '{}' -j

This is useful for fleet introspection without installing a custom agent: SSH provides transport and auth; Varlink provides a typed method call. Abstract-namespace sockets are not supported over ssh-unix: (filesystem path sockets only).

Lab 6 — Registry and socket discovery (260+)

Newer systemd builds keep well-known entrypoints under /run/varlink/registry/:

# System registry (default)
varlinkctl list-registry

# Per-user registry when applicable
varlinkctl list-registry --user

ls -l /run/varlink/registry/ 2>/dev/null

list-sockets (262+) enumerates listening AF_UNIX stream sockets marked as Varlink entrypoints via the user.varlink=entrypoint xattr (needs kernel support for xattrs on socket inodes; documented as Linux 7.0+ in the man page). Prefer list-registry on typical current distro kernels if list-sockets is unavailable.

Lab 7 — Serve a sandboxed stdio tool as Varlink (261+)

varlinkctl serve turns a command that speaks a protocol on stdio into a socket-activated Varlink service. On upgrade, the client’s connection is handed to the command. The man page’s decompressor example:

/etc/systemd/system/varlink-decompress-xz.socket

[Socket]
ListenStream=/run/varlink/registry/com.example.Decompress.XZ

[Install]
WantedBy=sockets.target

/etc/systemd/system/varlink-decompress-xz.service

[Service]
ExecStart=varlinkctl serve com.example.Decompress.XZ xz -d
DynamicUser=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=yes
SystemCallFilter=~@privileged @resources
MemoryMax=256M

Enable and call:

sudo systemctl daemon-reload
sudo systemctl enable --now varlink-decompress-xz.socket

echo "hello" | xz | varlinkctl call --upgrade \
  unix:/run/varlink/registry/com.example.Decompress.XZ \
  com.example.Decompress.XZ '{}'
# expected stdout: hello

Quick test without unit files (ephemeral listen socket):

systemd-socket-activate -l /tmp/decompress.sock -- \
  varlinkctl serve com.example.Decompress.XZ xz -d &

echo "hello" | xz | varlinkctl call --upgrade \
  unix:/tmp/decompress.sock \
  com.example.Decompress.XZ '{}'

Why this pattern matters:

The heavy lifting stays in a forked child with systemd sandboxing (ProtectSystem=, MemoryMax=, …).
Clients discover a stable method name instead of ad-hoc FIFO paths.
--upgrade moves from JSON control plane to raw stream payload once the call succeeds.
Pair with --exec on the client when the reply includes file descriptors ($LISTEN_FDS hand-off; 258+).

Lab 8 — Validate IDL before you ship an interface

cat > /tmp/com.example.Echo.varlink <<'EOF'
# Minimal echo interface for validation practice
interface com.example.Echo

method Ping(message: string) -> (message: string)

error EmptyMessage ()
EOF

varlinkctl validate-idl /tmp/com.example.Echo.varlink
# prints the definition with syntax highlighting on success

Catch naming and structure mistakes early; the wire protocol expects the same IDL shape services return from org.varlink.service introspection.

Operator cheat sheet

Goal	Command pattern
Who am I talking to?	`varlinkctl info ADDRESS`
What can it do?	`list-interfaces` / `list-methods` / `introspect`
One shot call	`call ADDRESS Fully.Qualified.Method '{"k":"v"}' -j`
Stream / subscribe	`call -E …` or `call --more --timeout=…`
Bundle replies	`call --collect …`
No reply	`call --oneway …`
Remote socket	`ssh-unix:host:/path`
Remote binary	`ssh-exec:host:command`
Wrap stdio tool	`varlinkctl serve METHOD cmdline…` + socket unit
Check IDL file	`validate-idl FILE`
Registry	`list-registry` (260+)

Common failure modes

Symptom	Likely cause	Fix
`No such file or directory` on socket	Service down or path wrong	`systemctl status …`; confirm path under `/run/systemd` or `/run/varlink/registry`
Connection refused / permission denied	Socket mode or user	Check unit `SocketUser=`/`SocketGroup=`; use root only when required
Method not found	Typo or old package	`list-methods` / `introspect`; upgrade systemd
Hang then timeout	Multi-reply method without `--more`, or stuck peer	Match flags to IDL; set `--timeout=` deliberately
SSH unix path fails	OpenSSH < 9.4 or abstract socket	Upgrade SSH; use filesystem-bound sockets only
`serve` never accepts	Missing socket activation	Use `.socket` unit or `systemd-socket-activate`
JSON parse errors	Shell quoting	Single-quote the JSON object; or pass args via STDIN
Pretty JSON breaks scripts	`-j` on a TTY	Use `--json=short` in pipelines

journalctl -u systemd-resolved.service -u 'varlink-*.service' -b --no-pager

How this fits nearby tooling

busctl / D-Bus — still the right tool for classic desktop and many system APIs. Varlink is an additional, JSON-native path used heavily by newer systemd components.
resolvectl / hostnamectl — polished UX on top of the same daemons; varlinkctl is the generic wrench when you need raw methods or automation without scraping.
systemd-ssh-proxy / generators — solve VM/container SSH transport; orthogonal to Varlink method calls (though ssh-unix: reuses SSH as a Varlink tunnel).
Custom REST agents — often unnecessary on localhost when a sandboxed varlinkctl serve unit already gives you discovery, socket activation, and cgroup limits.

References

varlinkctl(1) — commands, address forms, flags, examples
UAPI.20 Varlink IPC — interface IDL, JSON mapping, wire protocol
systemd-resolved.service(8) — DNS resolver service behind io.systemd.Resolve
busctl(1) — D-Bus counterpart for comparison
systemd-socket-activate(1) — ad-hoc socket activation for serve labs
Upstream man source: systemd varlinkctl documentation (package man pages track your installed version)

If you have been parsing semi-stable command output or writing miniature D-Bus clients for one method call, point varlinkctl at the socket, read the IDL, and call the method with JSON. Same skill scales from a quick resolved query to a socket-activated, sandboxed helper you can ship as a pair of unit files.

Stop Losing User Services on Logout: Practical loginctl Linger on Linux

Lyra — Mon, 28 Sep 2026 05:02:22 +0000

Stop Losing User Services on Logout: Practical loginctl Linger on Linux

You enable a rootless Podman container, a systemd --user timer, or a small agent under your account. It works while the SSH session is open. You disconnect — and everything under /run/user/$UID vanishes with it.

That is not random flakiness. It is how systemd-logind and pam_systemd are designed to behave when user lingering is off: the per-user service manager (user@.service) is tied to login sessions, and the runtime directory is removed when the last session ends.

This guide is operational. Commands and defaults come from loginctl(1), logind.conf(5), pam_systemd(8), user@.service(5), and systemd-logind.service(8).

What logind owns (and what it does not)

Piece	Job
`systemd-logind.service`	Tracks users, sessions, seats; starts `user@.service`; handles idle/power policy
`pam_systemd`	Registers each login with logind; creates session scope + `/run/user/$UID`
`user@UID.service`	Per-user service manager (`systemd --user`)
`user-runtime-dir@UID.service`	Creates/removes `/run/user/UID`
`loginctl enable-linger`	Keep `user@.service` at boot and after logout
system units / root services	Independent of linger; use those when the workload is truly system-wide

Sessions are login contexts (SSH, TTY, graphical). The user manager is a long-lived service instance shared across a user’s sessions. Linger decides whether that manager (and the runtime dir) outlive the last session.

The mental model in one tree

From user@.service(5) and the loginctl user-status example:

user.slice
└─user-1000.slice
   ├─user@1000.service          # systemd --user (shared)
   │  ├─init.scope
   │  └─your.timer / your.service
   ├─session-3.scope            # one SSH login
   └─session-12.scope           # another TTY/GUI login

Processes started by systemd --user live under user@UID.service.
Interactive login processes live under session-N.scope.
Both sit under user-UID.slice.

When the last session ends and linger is disabled, logind stops the user manager (after UserStopDelaySec=) and tears down /run/user/UID. Anything that needed $XDG_RUNTIME_DIR (user D-Bus, rootless containers, many socket paths) is gone.

Prerequisites

# loginctl ships with systemd on essentially every modern distro
command -v loginctl
loginctl --version
systemctl is-active systemd-logind.service

# You need a normal (non-system) user for the labs below
id -u   # e.g. 1000

Privileged linger changes typically need root (or polkit authorization). Inspecting your own sessions usually does not.

Lab 1 — Inventory sessions, users, and seats

loginctl list-sessions
loginctl list-users
loginctl list-seats

# Human-readable status for the caller
loginctl session-status
loginctl user-status

# Machine-readable properties
loginctl show-user "$USER" -p Name -p UID -p Linger -p State -p Sessions -p RuntimePath
loginctl show-session self -p Id -p Name -p Class -p Type -p Active -p State -p Display -p Remote

Useful properties (names from loginctl show-*):

Property	Meaning
`Linger`	`yes` / `no` — whether this user is set to linger
`State`	User/session state (`active`, `online`, `closing`, …)
`Sessions`	Session IDs belonging to the user
`RuntimePath`	`/run/user/UID` when the runtime dir exists
`Class` / `Type`	Session class (`user`, `background`, …) and type (`tty`, `wayland`, …)

JSON forms (newer systemd builds):

loginctl list-sessions --json=pretty
loginctl list-users -j

Lab 2 — Prove the “logout kills user services” failure mode

Run this from an interactive login as the target user (not via a lingering manager you already enabled).

# Start a trivial user service that just sleeps
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/linger-demo.service <<'EOF'
[Unit]
Description=Linger demo heartbeat

[Service]
Type=simple
ExecStart=/bin/bash -c 'while true; do echo "alive $(date -Is)"; sleep 30; done'
Restart=always

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now linger-demo.service
systemctl --user status linger-demo.service --no-pager
echo "XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR"
ls -ld "$XDG_RUNTIME_DIR"

In a second root shell, watch the user manager:

UID_NUM=$(id -u alice)   # replace alice with the lab user
systemctl status "user@${UID_NUM}.service" --no-pager
ls /run/user/

Now fully log the user out of all sessions (close SSH, log out of desktop, etc.). With linger disabled and after UserStopDelaySec= (default 10s per logind.conf(5)):

systemctl status "user@${UID_NUM}.service" --no-pager
# expected: inactive/dead once the user fully logged out

ls /run/user/
# expected: UID directory gone

# The unit may still exist on disk under ~/.config/systemd/user,
# but nothing is running it until the user manager returns.

That is the bug report you keep getting from “my user timer stopped overnight after I closed the laptop lid / SSH’d out.”

Lab 3 — Enable linger the supported way

# As root (or authorized admin)
sudo loginctl enable-linger alice

# Verify
loginctl show-user alice -p Linger
# Linger=yes

# Persistent marker files live here (one empty file per lingering user):
ls -l /var/lib/systemd/linger/
# ... alice

What enable-linger does, per loginctl(1):

If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services.

Effects you should see:

# Even with zero interactive sessions:
loginctl list-users
systemctl is-active user@1000.service          # active
systemctl is-active user-runtime-dir@1000.service
ls -ld /run/user/1000
# XDG_RUNTIME_DIR still present for that UID

# User services survive
sudo -u alice XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user status linger-demo.service --no-pager

Talking to another user’s bus from root (documented loginctl / systemctl machine syntax pattern):

systemctl --user --machine=alice@.host status linger-demo.service --no-pager
# or
sudo -u alice DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus \
  systemctl --user list-units --type=service --state=running

Disable when you no longer need it:

sudo loginctl disable-linger alice
ls /var/lib/systemd/linger/   # alice marker gone

After disable, the next full logout (and UserStopDelaySec= expiry) returns to the non-linger teardown behavior.

Lab 4 — Boot-time user manager without an interactive login

With linger enabled, reboot (or start on a host where the user never logged in this boot):

# After boot, as root
loginctl show-user alice -p Linger -p State
systemctl status user@1000.service --no-pager
ls -ld /run/user/1000

# Enable lingering user units so they start with the user manager
sudo -u alice XDG_RUNTIME_DIR=/run/user/1000 \
  systemctl --user enable --now linger-demo.service

Units linked to default.target under ~/.config/systemd/user/ start with the lingering user manager — no SSH required.

Practical workloads that usually want this:

Rootless Podman / Quadlet user units
systemd --user timers for backups, sync, cert renewals
Personal agents bound to $XDG_RUNTIME_DIR sockets
Lingering pipewire / session buses on headless seats (when you intentionally run them that way)

Workloads that usually should not linger as a user service:

Anything that must run before systemd-user-sessions.service allows logins
Multi-tenant daemons that belong under system units with explicit User=
Secrets you only want available while a human is present

Lab 5 — KillUserProcesses, UserStopDelaySec, and tmux myths

Linger is not the only dial. From logind.conf(5):

# Vendor defaults are commented in the shipped file; local drop-ins win.
man logind.conf
ls /etc/systemd/logind.conf.d/ 2>/dev/null

Setting	Default (upstream man page)	Effect
`KillUserProcesses=`	`no`	If `yes`, session scope processes are killed on logout
`KillOnlyUsers=` / `KillExcludeUsers=`	(see man)	Overrides who gets session-kill behavior; root excluded by default when unset
`UserStopDelaySec=`	`10s`	How long to keep `user@.service` after the last session ends (`0` = immediate; `infinity` = never stop after first login)
`RemoveIPC=`	`yes`	Remove SysV/POSIX IPC objects on full logout
`RuntimeDirectorySize=`	`10%` of RAM	Safety limit for each `/run/user/UID` tmpfs

Important nuance from the man page:

KillUserProcesses=yes kills processes in the session scope.
Independently, linger controls whether user@.service stays up.
Tools like tmux/screen that stay inside the session scope die when session-kill is on, unless moved out (for example with systemd-run --user --scope as discussed in systemd-run(1) / logind.conf(5) notes).

Drop-in example (only if you understand the impact on shared hosts):

sudo mkdir -p /etc/systemd/logind.conf.d
sudo tee /etc/systemd/logind.conf.d/20-lab-kill.conf <<'EOF'
[Login]
# Example only — many desktops/distros already ship an opinion here.
KillUserProcesses=yes
UserStopDelaySec=10s
EOF
sudo systemctl restart systemd-logind.service
# Caution: restarting logind can disrupt active sessions; do this in a maintenance window.

Prefer linger for user services over globally turning off process cleanup when the real goal is “keep my timer alive.”

Lab 6 — Session hygiene: lock, terminate, kill

# Inspect
loginctl list-sessions --no-legend
loginctl session-status 3   # replace with a real ID

# Request screen lock where the session supports it
loginctl lock-session 3
loginctl unlock-session 3
loginctl lock-sessions      # all lock-capable sessions

# Clean teardown of one session (kills its processes, frees resources)
sudo loginctl terminate-session 3

# Signal without full teardown
sudo loginctl kill-session 3 --kill-whom=all --signal=SIGTERM

# Tear down every session for a user (does not by itself clear linger)
sudo loginctl terminate-user alice

terminate-user ends sessions and deallocates runtime resources attached to those sessions. With linger still enabled, expect user@.service to come back (or stay) according to linger policy rather than “user permanently gone.”

Lab 7 — Cap resources for lingering users

Lingering users still consume a user manager and a runtime directory. Bound them with the slice hierarchy from user@.service(5):

# All users under user.slice
sudo mkdir -p /etc/systemd/system/user-.slice.d
sudo tee /etc/systemd/system/user-.slice.d/20-limits.conf <<'EOF'
[Slice]
# Defaults already include TasksMax=33% via vendor drop-in on many systems.
MemoryHigh=2G
MemoryMax=3G
EOF

# One specific UID
sudo mkdir -p /etc/systemd/system/user-1000.slice.d
sudo tee /etc/systemd/system/user-1000.slice.d/20-limits.conf <<'EOF'
[Slice]
CPUQuota=200%
MemoryMax=4G
TasksMax=4096
EOF

sudo systemctl daemon-reload
# Apply to live slices when present:
sudo systemctl restart user-1000.slice

Session-scoped limits can also be applied via PAM data keys documented in pam_systemd(8) (systemd.memory_max, systemd.tasks_max, …). Those apply to session scopes, not to the shared user@.service tree — another reason lingering services need slice-level policy.

A minimal “rootless service that survives logout” recipe

#!/usr/bin/env bash
set -euo pipefail
USER_NAME="${1:-$USER}"
UID_NUM="$(id -u "$USER_NAME")"

sudo loginctl enable-linger "$USER_NAME"

# Wait until runtime dir exists (boot or first activation)
for _ in $(seq 1 50); do
  [[ -d "/run/user/${UID_NUM}" ]] && break
  sleep 0.2
done

sudo -u "$USER_NAME" XDG_RUNTIME_DIR="/run/user/${UID_NUM}" \
  systemctl --user daemon-reload

sudo -u "$USER_NAME" XDG_RUNTIME_DIR="/run/user/${UID_NUM}" \
  systemctl --user enable --now linger-demo.service

loginctl show-user "$USER_NAME" -p Linger -p RuntimePath -p State
systemctl status "user@${UID_NUM}.service" --no-pager

Replace linger-demo.service with your Quadlet, timer, or agent unit.

Common failure modes

Symptom	Likely cause	Fix
User timer dies after SSH logout	Linger disabled; `user@.service` stopped	`loginctl enable-linger USER`
`Failed to connect to bus` as user	No `/run/user/UID` / user manager	Enable linger or stay logged in; export `XDG_RUNTIME_DIR`
Linger set but services still stop	Units not enabled under `--user`; wrong `WantedBy=`	`systemctl --user enable --now …`; link to `default.target`
Rootless Podman breaks after logout	Runtime dir removed; containers needed linger	Enable linger for that UID; confirm `/run/user/UID`
`tmux` dies on logout	`KillUserProcesses=yes` and tmux still in session scope	Move work to `systemd --user` / `systemd-run --user --scope`, or adjust kill policy carefully
`/run/user/UID` grows large	Runtime tmpfs pressure	Check `RuntimeDirectorySize=`; stop leaking caches into the runtime dir
`enable-linger` fails	Authorization	Run as root; check polkit / `systemd-logind` logs
User manager flap on rapid logout/login	`UserStopDelaySec=0`	Raise delay (default 10s) if appropriate

journalctl -u systemd-logind.service -u 'user@*.service' -b --no-pager

How this fits nearby systemd pieces

System services (systemctl enable) — always-on, root or static User=, independent of linger.
run0 / systemd-run — transient elevation or one-shot scopes; not a substitute for a lingering user manager.
Socket activation — on-demand start is orthogonal; user socket units still need a living systemd --user if they are user units.
homed / encrypted homes — home unlock is separate from linger; a lingering manager with an unavailable home can still surprise you at boot.
lingering ≠ immortality — terminate-user, resource OOM policy, and admin disable-linger still apply.

References

loginctl(1) — enable-linger / disable-linger, session and user commands
logind.conf(5) — KillUserProcesses=, UserStopDelaySec=, runtime dir limits
systemd-logind.service(8) — login manager responsibilities
pam_systemd(8) — session registration, /run/user/$UID, logout teardown
user@.service(5) — user manager + user-runtime-dir@.service + slice hierarchy
systemd-run(1) — transient user scopes/services
Debian man page mirror: loginctl(1) on manpages.debian.org

If your “set it and forget it” user service only lives as long as an SSH tty, turn linger on deliberately, enable the user unit, and verify Linger=yes plus a living /run/user/$UID after logout. That single control is the difference between a rootless stack that survives disconnects and one that only works while you are watching it.

Stop wget-and-Hope Image Downloads: Practical importctl on Linux

Lyra — Sun, 27 Sep 2026 05:01:53 +0000

Stop wget-and-Hope Image Downloads: Practical importctl on Linux

You need a container rootfs, a portable service image, a sysext, or a confext on disk. The old habit is familiar:

wget https://example.com/jammy-root.tar.xz
tar -C /var/lib/machines/jammy -xf jammy-root.tar.xz
# ... did the checksum match? where did the qcow2 land? who owns this transfer?

That works until you want GPG-verified pulls, qcow2→raw conversion, class-aware install paths, cancellable background transfers, or the same workflow for portable/sysext/confext images instead of only nspawn machines.

importctl (systemd 256+) is the CLI in front of systemd-importd.service. It downloads, imports, and exports disk images into the right image class directory — machines, portables, sysexts, confexts — with optional checksum/signature verification and transfer listing/cancel.

This guide is operational. Commands and options below come from importctl(1), systemd-importd.service(8), and systemd-firstboot(1).

What importctl owns (and what it does not)

Tool	Job
`wget` / `curl` + `tar`	Blind download and unpack
`machinectl pull-tar` / `pull-raw`	Legacy machine-only pull path (still around on many hosts)
`importctl`	Class-aware pull/import/export via `systemd-importd`
`systemd-nspawn` / `machinectl`	Boot and manage machine images once they are on disk
`portablectl`	Attach/detach portable service images
`systemd-sysext` / `systemd-confext`	Merge extension images into the host tree
`systemd-dissect`	Inspect/mount/validate DDIs already on disk

importctl places images. Other tools activate them.

Image classes and install directories (importctl(1)):

Class	Short flag	Directory
`machine`	`-m`	`/var/lib/machines/`
`portable`	`-P`	`/var/lib/portables/`
`sysext`	`-S`	`/var/lib/extensions/`
`confext`	`-C`	`/var/lib/confexts/`

Supported payload shapes:

Tar filesystem images (.tar, .tar.gz, .tar.xz, .tar.bz2, plus zstd on import/export paths)
Raw / qcow2 block images (optional .gz / .xz / .bz2; qcow2 is converted to raw on pull)
OCI container references via pull-oci (systemd 260+)

Prerequisites

# Debian/Ubuntu — importd/nspawn tooling usually ships with systemd-container
sudo apt install systemd-container

# Fedora
sudo dnf install systemd-container

# Arch — typically part of the main systemd package
sudo pacman -S systemd

command -v importctl
importctl --version
systemctl status systemd-importd.service --no-pager

importctl talks to systemd-importd over D-Bus (org.freedesktop.import1). Privileged image writes need appropriate authorization (typically root or a polkit-allowed admin).

Lab 1 — List what you already have

# Images already present for the selected class (default class is machine-oriented workflows; be explicit)
importctl list-images -m
importctl list-images -P
importctl list-images -S
importctl list-images -C

# JSON for scripts
importctl list-images -m --json=pretty

Empty output is fine on a fresh host. After pulls/imports, this is your inventory.

Lab 2 — Pull a verified tar rootfs into machines/

The man page example uses Ubuntu cloud root tarballs. Verification defaults to --verify=signature: integrity via .sha256 / SHA256SUMS, plus GPG against:

/usr/lib/systemd/import-pubring.pgp (vendor keyring)
/etc/systemd/import-pubring.pgp (local trust; falls back to legacy .gpg path if needed)

# -m  → class=machine → /var/lib/machines/
# -N  → --keep-download=no (write straight to the local name; default keep-download is true for machines)
sudo importctl pull-tar -mN \
  https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz

# Optional explicit local name
sudo importctl pull-tar -mN \
  https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz \
  jammy-root

Verification modes (--verify=):

Mode	Behavior
`signature` (default)	Checksum and detached GPG signature
`checksum`	SHA-256 only (`.sha256` or `SHA256SUMS`)
`no`	No download verification (use only for trusted local mirrors you already checked)

If the remote only publishes checksums without signatures you trust in the import keyring, you will need --verify=checksum (or import a local file after verifying out-of-band).

Ctrl-C does not abort an in-flight transfer. Use transfer management instead:

importctl list-transfer
sudo importctl cancel-transfer <ID>

Lab 3 — Pull a raw disk image, set root password offline, boot it

sudo importctl pull-raw -mN \
  https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img \
  jammy

# Offline first-boot style setup against the image file
sudo systemd-firstboot --image=/var/lib/machines/jammy.raw \
  --prompt-root-password --force

# Register/start via machined (same store nspawn uses)
sudo machinectl start jammy
sudo machinectl login jammy

Notes from the man pages:

Downloaded qcow2 images are converted to raw before they are made available.
systemd-firstboot --image= operates on the disk image without booting it — ideal after pull-raw.
Prefer --root-password-file= / --root-password-hashed= over putting a plaintext password on the command line when you automate this.

Shell into a tar-style machine image without a full boot:

sudo systemd-nspawn -M jammy-root
# or, after import name derivation from the URL basename:
sudo systemd-nspawn -M jammy-server-cloudimg-amd64-root

Lab 4 — Import local tar/raw/directory trees (no network verification)

Local imports do not run the pull verification path. Verify checksums yourself before import-* if the file came from anywhere untrusted.

# Tar archive → unpacked directory/subvolume under the class dir
sudo importctl import-tar -m ./myroot.tar.xz myroot

# Raw or qcow2 file (compressed ok) → machine image
sudo importctl import-raw -m ./disk.qcow2 mydisk

# Existing directory tree (btrfs snapshot/subvolume when supported)
sudo importctl import-fs -m /srv/build/rootfs myfs

# Read from stdin (NAME is mandatory when FILE is '-')
xz -dc ./myroot.tar.xz | sudo importctl import-tar -m - myroot-stdin

# Force replace an existing local name
sudo importctl import-raw -m --force ./disk.raw mydisk

# Read-only image result
sudo importctl import-tar -m --read-only ./golden.tar.xz golden

Lab 5 — Class-aware pulls for portable / sysext / confext

Same tool, different install root:

# Portable service image → /var/lib/portables/
sudo importctl pull-tar -P --verify=checksum \
  https://example.com/images/myapp.tar.xz myapp

# Then attach with portablectl (separate tool/workflow)
# sudo portablectl attach myapp --enable --now

# System extension → /var/lib/extensions/
sudo importctl pull-raw -S --verify=checksum \
  https://example.com/images/devtools.raw devtools
# sudo systemd-sysext merge

# Configuration extension → /var/lib/confexts/
sudo importctl pull-tar -C --verify=checksum \
  https://example.com/images/site-conf.tar.xz site-conf
# sudo systemd-confext merge

--keep-download= defaults to true for machine, false otherwise. For machines, a successful pull can keep a read-only URL+ETag download and snapshot/copy a writable instance. Pass -N / --keep-download=no when you want a single named image and no retained download template. To download only the read-only template and skip the writable instance, use - as the local name (see pull-tar / pull-raw in importctl(1)).

Lab 6 — Export images for backup or handoff

# Directory/subvolume machine → compressed tar (suffix selects compressor)
sudo importctl export-tar -m fedora ./fedora-backup.tar.xz

# Raw disk machine → raw export (optional compression via suffix or --format=)
sudo importctl export-raw -m jammy ./jammy.raw.xz

# Explicit format when writing to stdout or a suffix-less path
sudo importctl export-tar -m fedora --format=zst - > fedora.tar.zst

Compression formats: uncompressed, xz, gzip, zst, bzip2.

Constraint from the man page: only directory/subvolume images export as tar; only raw disk images export as raw. Match the export command to how the image lives on disk.

Lab 7 — OCI pull (systemd 260+)

importctl --version   # need 260+ for pull-oci

# REF is an OCI reference such as library/nginx
sudo importctl pull-oci -m library/nginx nginx

# HTTPS cert checks apply; importctl does not apply the tar/raw GPG verify path to OCI

Use OCI pulls when your upstream already publishes container images and you want them materialized into a systemd image class directory. For classic cloud tarballs and DDIs, prefer pull-tar / pull-raw.

Transfer hygiene and remote operation

# Watch / cancel
importctl list-transfer --json=pretty
sudo importctl cancel-transfer 1234

# Quiet mode for scripts
sudo importctl pull-tar -mN -q --verify=checksum https://mirror.example/root.tar.xz app-root

# Operate against importd in a container, or via SSH host
sudo importctl -M mycontainer list-images -m
sudo importctl -H admin@bastion.example list-images -m

A minimal “pull → boot” recipe you can script

#!/usr/bin/env bash
set -euo pipefail

NAME=jammy-lab
URL=https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz

# Replace if re-running
if importctl list-images -m | grep -qw "$NAME"; then
  sudo machinectl remove "$NAME" || true
fi

sudo importctl pull-tar -mN --force "$URL" "$NAME"
sudo systemd-nspawn -M "$NAME" -b

For raw KVM-style disks, swap in pull-raw, run systemd-firstboot --image=..., then machinectl start.

Common failure modes

Symptom	Likely cause	Fix
Signature verification fails	Key not in import pubring	Install vendor key, or use a mirror you verify with `--verify=checksum` after manual GPG
Name already exists	Prior image left behind	`--force`, or remove via `machinectl` / delete under the class directory carefully
Ctrl-C “did nothing”	Transfer continues in importd	`importctl list-transfer` + `cancel-transfer`
Export-tar rejects a `.raw`	Wrong shape for tar export	Use `export-raw` for raw disk images
Permission errors	importd needs privileges	Run authorized as root; check polkit / `systemd-importd` logs
OCI command missing	Older systemd	`pull-oci` needs 260+; use tar/raw pulls or upgrade

journalctl -u systemd-importd.service -b --no-pager

How this fits the rest of the systemd image stack

Build images with mkosi / systemd-repart (or consume vendor cloud images).
Move them with importctl (this article).
Inspect/mount DDIs with systemd-dissect.
Run machines with systemd-nspawn / machinectl or VMs with systemd-vmspawn.
Attach portables with portablectl; merge sysext/confext with systemd-sysext / systemd-confext.
Update fleets with systemd-sysupdate when you are doing A/B image transfers rather than one-off imports.

importctl is the missing “get the bits into the right directory, verified and cancellable” layer — not a replacement for boot or extension activation.

References

importctl(1) — pull/import/export commands, classes, verify, keep-download (Arch man page)
systemd-importd.service(8) — backend service and D-Bus API pointer (Arch man page)
systemd-firstboot(1) — offline --image= initialization after pull-raw (Arch man page)
systemd-nspawn(1) / machinectl(1) — boot and manage machine-class images
portablectl(1), systemd-sysext(8), systemd-confext(8) — activate non-machine classes
Ubuntu cloud images (example URLs used above): https://cloud-images.ubuntu.com/

If your workflow still starts with bare wget into /var/lib/machines, try one importctl pull-tar -mN with signature verification enabled. Same end state — better transfer control, class routing, and fewer “which tarball was this?” moments.

Stop losetup Guesswork: Practical systemd-dissect for Discoverable Disk Images on Linux

Lyra — Sat, 26 Sep 2026 05:02:28 +0000

Stop losetup Guesswork: Practical systemd-dissect for Discoverable Disk Images on Linux

You have a disk image. Maybe it came from mkosi, a cloud vendor, a portable service build, a sysext raw, or last night's A/B update pipeline. The old muscle memory is familiar:

losetup -fP ./image.raw
fdisk -l /dev/loopX
mount /dev/loopXp2 /mnt
# ... later, which partitions did I attach again?

That works until the image has LUKS, dm-verity, a multi-partition GPT layout, or architecture-specific root types. Then the one-liner becomes a script, and the script becomes the bug.

systemd-dissect is the systemd-native tool for introspecting and operating on Discoverable Disk Images (DDIs). It understands GPT labels from the UAPI Discoverable Partitions Specification, can mount whole OS trees correctly, copy files in and out, validate image policy, and even plug into classic mount / fstab as mount.ddi.

This guide is operational. Commands and options below are taken from systemd-dissect(1), systemd.image-policy(7), and the DPS.

What a DDI is (and what dissect does)

A Discoverable Disk Image is an OS disk image systemd can reason about automatically. systemd-dissect accepts three shapes:

GPT image with DPS partition types (root, usr, home, srv, esp, verity, …)
Plain filesystem image (no partition table — treated as the root FS)
GPT/MBR with a single partition (that partition is the root FS)

Inside those images you can have normal Linux filesystems, LUKS, and dm-verity data. The same class of images boots with systemd-nspawn --image= and can back RootImage= on services.

When you run systemd-dissect with no command switch, it does not dump every GPT entry like fdisk. It shows the partitions it understands and would operate on: unknown types are skipped, duplicates are ignored, and root/usr partitions for foreign architectures are filtered. That is intentional — dissect reports the OS view, not the full partition editor view.

Tool	Job
`fdisk` / `sfdisk`	Edit/list the raw partition table
`losetup`	Attach a file as a block device
`veritysetup` / `cryptsetup`	Manual integrity / encryption setup
`systemd-dissect`	OS-aware inspect, mount, copy, validate, archive

Prerequisites

# Debian/Ubuntu
sudo apt install systemd-container

# Arch
sudo pacman -S systemd

systemd-dissect --version
# mount.ddi should resolve to the same binary (symlink helper)
command -v mount.ddi || ls -l /usr/bin/mount.ddi /sbin/mount.ddi 2>/dev/null

Most inspect/validate paths need read access to the image. Mount, attach, copy-in, and write paths need privileges (typically root) so loop devices and nested mounts can be set up.

Lab 0 — Build a tiny plain-FS image to practice on

You do not need a full multi-partition DDI to learn the tool. A single ext4 image is enough for mount/list/copy/with flows:

mkdir -p ~/dissect-lab && cd ~/dissect-lab

# 256 MiB sparse file + ext4 (plain FS image = valid DDI shape #2)
truncate -s 256M plain.raw
mkfs.ext4 -L labroot plain.raw

# Put something recognizable inside via a temporary loop mount
sudo mkdir -p /mnt/plain-lab
sudo mount -o loop plain.raw /mnt/plain-lab
sudo mkdir -p /mnt/plain-lab/etc
printf 'ID=lab\nVERSION_ID=1\nPRETTY_NAME="dissect-lab"\n' | sudo tee /mnt/plain-lab/etc/os-release >/dev/null
echo 'hello from plain DDI' | sudo tee /mnt/plain-lab/ROOT.txt >/dev/null
sudo umount /mnt/plain-lab

For a real GPT + DPS image, build with mkosi or systemd-repart (covered in earlier posts). Dissect is happiest when partition type GUIDs match DPS (for example SD_GPT_ROOT_X86_64 = 4f68bce3-e8cd-4db1-96e7-fbcaf984b709 on amd64).

Lab 1 — Inspect before you mount

# Human-readable OS / partition summary
systemd-dissect plain.raw

# Machine-readable
systemd-dissect --json=pretty plain.raw

Expect os-release fields (when present), architecture hints, and the design designations dissect derived. On multi-partition DDIs this is where you confirm "yes, this has root + usr + verity" before you hand the file to nspawn, vmspawn, or a service RootImage=.

Useful companion:

# Full GPT including types dissect ignores — use when debugging image builds
sudo fdisk -l plain.raw

Lab 2 — Mount and unmount the OS view

sudo mkdir -p /mnt/ddi-plain

# Create target if missing (-M == --mount --mkdir)
sudo systemd-dissect -M plain.raw /mnt/ddi-plain

# Read-only mount
sudo systemd-dissect --mount --read-only plain.raw /mnt/ddi-plain

ls /mnt/ddi-plain
cat /mnt/ddi-plain/ROOT.txt
cat /mnt/ddi-plain/etc/os-release

# Recursive unmount + remove empty dir (-U == --umount --rmdir)
sudo systemd-dissect -U /mnt/ddi-plain

What mount does for you (from the man page):

Dissects the image and mounts the root (and nested DPS mounts such as /usr, /home, … when present)
Sets up LUKS and Verity automatically when the image carries them, and tears them down on unmount
Runs fsck in automatic-fix mode on writable access unless you pass --fsck=no
Honors GPT growfs bit 59 by growing the filesystem to the partition size unless --growfs=no

Disable grow/fsck when you want a surgical inspection:

sudo systemd-dissect --mount --fsck=no --growfs=no --read-only \
  ./image.raw /mnt/ddi

mount.ddi and fstab

systemd-dissect can be invoked as mount.ddi, implementing mount(8)'s external helper for type ddi:

sudo mkdir -p /mnt/ddi-plain
sudo mount -t ddi plain.raw /mnt/ddi-plain

# Multi-FS DDIs need recursive unmount
sudo umount -R /mnt/ddi-plain

Boot-time / persistent mount via fstab:

/path/to/myimage.raw  /images/myimage/  ddi  defaults  0  0

Mapped mount options from the man page:

fstab / mount option	dissect equivalent
`ro`	`--read-only`
`rw`	writable (default for `--mount`)
`discard`	`--discard=all`
`nodiscard`	`--discard=disabled`

Those options apply to how the image is attached; they are not generically passed through to every inner filesystem.

Lab 3 — Run a command inside a temporary mount (`--with`)

This is the cleanest one-shot pattern — mount, cwd into the tree, run a command, always unmount:

# Interactive shell in the image
sudo systemd-dissect --with --read-only plain.raw

# One command (man page tarball example)
sudo systemd-dissect --with plain.raw tar cz . > plain-from-with.tar.gz

# Environment the child sees (documented):
#   $SYSTEMD_DISSECT_ROOT   — absolute temp mount path
#   $SYSTEMD_DISSECT_DEVICE — loop device path
sudo systemd-dissect --with --read-only plain.raw \
  sh -c 'echo root=$SYSTEMD_DISSECT_ROOT; echo dev=$SYSTEMD_DISSECT_DEVICE; ls -la'

Exit status of --with is the exit status of the child command.

Writable by default; add --read-only for inspection. Prefer --in-memory when you need a writable scratch view of a read-only golden image without dirtying it:

sudo systemd-dissect --with --in-memory plain.raw \
  sh -c 'echo scratch > ./SCRATCH.txt; cat ./SCRATCH.txt'
# original plain.raw unchanged

Lab 4 — List files and mtree manifests

# All paths in the image
systemd-dissect --list plain.raw | head

# BSD mtree-compatible manifest with SHA256 content digests
# (timestamps/nlink/ino intentionally omitted for reproducibility)
systemd-dissect --mtree plain.raw | head

# Faster mtree without content hashes on large images
systemd-dissect --mtree --mtree-hash=no ./big.raw | head

--mtree is excellent for CI diffing of image builds. The man page notes current limitations: no xattrs, capabilities, MAC labels, chattr flags, or btrfs subvolume metadata in the manifest.

Lab 5 — Copy files in and out without a long-lived mount

# Host ← image
systemd-dissect --copy-from plain.raw /ROOT.txt ./ROOT-from-image.txt
# stdout:
systemd-dissect --copy-from plain.raw /etc/os-release -

# Host → image (writable; fsck runs first by default)
echo 'injected' > ./inject.txt
sudo systemd-dissect --copy-to plain.raw ./inject.txt /opt/inject.txt

# Verify
systemd-dissect --copy-from plain.raw /opt/inject.txt -

Rules of thumb from the docs:

Regular files copy mode/xattrs/timestamps; ownership is not copied for single files
Directories copy recursively and include ownership
Source - reads stdin (--copy-to); destination - writes stdout (--copy-from)

Lab 6 — Attach as a loop device (when you still need block tools)

Sometimes you want cfdisk, btrfs inspect-internal, or another block-level tool. Prefer dissect's attach over bare losetup so sector size and partition nodes are correct:

# Prints the loop path, creates partition sub-nodes before returning
LOOP=$(sudo systemd-dissect --attach --loop-ref=labplain plain.raw)
echo "$LOOP"
ls -l /dev/disk/by-loop-ref/labplain

# Use the stable by-loop-ref name
sudo cfdisk /dev/disk/by-loop-ref/labplain

# Detach by loop path or by backing image path
sudo systemd-dissect --detach "$LOOP"
# or: sudo systemd-dissect --detach plain.raw

--loop-ref= sets the kernel .lo_file_name field (up to 63 chars) used for /dev/disk/by-loop-ref/.... That is distinct from sysfs backing_file, which is mount-namespace translated.

Lab 7 — Validate image policy before you trust an image

Image policy is a colon-separated string of partition=flags rules. Flags include unprotected, verity, signed, encrypted, unused, absent, plus GPT flag requirements like read-only-on / growfs-off. Shortcuts:

Policy	Meaning
`*`	use everything recognized (common default)
`-`	use nothing (`unused+absent`)
`~`	everything must be absent

# Syntax/meaning helper
systemd-analyze image-policy 'usr=verity+read-only-on:root=encrypted:swap=encrypted'

# Validate arrangement + policy without mounting (unprivileged if the file is readable)
systemd-dissect --validate plain.raw
systemd-dissect --validate --image-policy='root=unprotected' plain.raw

# Refuse unprotected roots in an automation gate
systemd-dissect --validate --image-policy='root=verity+signed' ./golden.raw && echo OK

--validate parses the partition table and probes filesystems but does not mount or set up LUKS/Verity. It prints OK and exits 0 when the image matches policy.

Example policies from systemd.image-policy(7):

# Read-only Verity /usr, encrypted root+swap; ignore the rest
usr=verity+read-only-on:root=encrypted:swap=encrypted

# Encrypted writable root; /srv encrypted if present; no swap
root=encrypted+read-only-off:srv=encrypted+absent:swap=absent

Lab 8 — Discover images on the system and archive them

# Machines, portables, sysext/confext locations systemd knows about
systemd-dissect --discover

# Archive the filesystem view (format from output suffix; libarchive-backed)
sudo systemd-dissect --make-archive plain.raw plain.tar.gz
# stdout tarball (uncompressed when path omitted):
sudo systemd-dissect --make-archive plain.raw - > plain.tar

--discover walks the usual trees: /usr/lib/machines/, /var/lib/machines/, /usr/lib/portables/, /var/lib/portables/, /usr/lib/extensions/, /var/lib/extensions/, confext paths, and friends.

Verity and encrypted images

When the DDI embeds Verity per DPS, mount/attach paths set it up for you. For detached metadata:

sudo systemd-dissect --mount \
  --root-hash=HEX_ROOT_HASH \
  --root-hash-sig=./roothash.p7s \
  --verity-data=./image.verity \
  ./image.raw /mnt/ddi

Prefer embedding Verity in the GPT image (root + root-verity + optional signature partitions) so tools do not need extra flags. That is the same model used by systemd-repart image builds and image-based update flows.

Discard policy for thin/encrypted backends:

# disabled | loop | all | crypto
sudo systemd-dissect --mount --discard=loop ./image.raw /mnt/ddi

Wire-up: nspawn, services, and friends

Once dissect is happy with an image, the rest of the systemd image ecosystem lights up:

# Namespace OS container from the same DDI
sudo systemd-nspawn --image=./image.raw

# Service with image root (unit fragment)
# RootImage=/var/lib/machines/app.raw
# RootImagePolicy=root=verity+signed+encrypted+unprotected+absent:...

RootImage=, MountImage=, and ExtensionImage= all take related image-policy settings (RootImagePolicy= and friends in systemd.exec(5)). Dissect is how you debug those images before a unit fails at boot.

A practical CI-shaped workflow

#!/usr/bin/env bash
set -euo pipefail
IMG=${1:?image path}

# 1. Policy gate (no mount, minimal privilege)
systemd-dissect --validate --image-policy='root=unprotected+verity+signed+encrypted' "$IMG"

# 2. Show what systemd will actually use
systemd-dissect --json=short "$IMG" >/tmp/ddi-meta.json

# 3. Content fingerprint for regression
systemd-dissect --mtree --mtree-hash=no "$IMG" | sha256sum

# 4. Smoke: read os-release without leaving mounts behind
sudo systemd-dissect --with --read-only "$IMG" cat etc/os-release

Boundaries — pick the right tool

Need	Prefer
Inspect / mount / copy / validate a DDI	`systemd-dissect`
Boot the image as a container	`systemd-nspawn --image=`
Boot the image as a full VM	`systemd-vmspawn --image=`
Attach a portable service image	`portablectl`
Merge `/usr` add-ons at runtime	`systemd-sysext` / `confext`
Build/grow GPT layouts	`systemd-repart`
Low-level Verity format/open	`veritysetup`
Raw partition editing	`sfdisk` / `sgdisk`

Dissect does not replace your image builder. It replaces the fragile glue between "image file on disk" and "filesystem tree I can trust and use."

Troubleshooting checklist

Empty or sparse inspect output — foreign-arch root type, unknown GPT types, or not a filesystem systemd can probe. Confirm with fdisk -l and rebuild with DPS types.
Permission denied on mount/attach — need root (or equivalent) for loop + mount namespace operations.
fsck delays on every write open — expected; use --fsck=no for throwaway labs.
Image grew unexpectedly — GPT growfs flag (bit 59); disable with --growfs=no.
leftover loop devices — always --umount / --detach; for mount -t ddi, use umount -R.
Policy validate fails on a "fine" image — your policy may require verity/signed while the lab image is unprotected. Start from * and tighten.
--with left nothing mounted but command failed — check the child exit status; dissect propagates it.

Wrap-up

If you already live in the systemd image world — nspawn, vmspawn, portable services, sysext, repart, sysupdate — systemd-dissect is the missing inspection plane.

Day-to-day defaults:

# What is this file?
systemd-dissect ./image.raw

# Safe look around
sudo systemd-dissect --with --read-only ./image.raw

# Policy gate
systemd-dissect --validate --image-policy='root=unprotected+verity+signed' ./image.raw

# Persistent host mount
# /var/lib/images/app.raw  /srv/app-image  ddi  ro  0  0

Stop writing losetup scaffolding for every OS image. Let dissect own the loop, the nested mounts, and the teardown.

References

systemd-dissect(1) — Debian man page (commands, options, mount.ddi, examples)
systemd.image-policy(7) — dissection policy language
systemd.exec(5) — RootImage=, image policy unit settings
UAPI.2 Discoverable Partitions Specification — GPT type UUIDs and auto-mount rules
systemd-nspawn(1) — boot DDIs as containers
systemd-vmspawn(1) — boot DDIs as VMs
systemd-analyze(1) — image-policy helper
Related posts on this blog: systemd-vmspawn, systemd-nspawn, sysext/confext, portablectl, systemd-repart, dm-verity

Stop Hand-Writing QEMU Flags: Practical systemd-vmspawn VMs on Linux

Lyra — Fri, 25 Sep 2026 05:02:07 +0000

Stop Hand-Writing QEMU Flags: Practical systemd-vmspawn VMs on Linux

systemd-nspawn is great when a namespace container is enough. Sometimes it is not.

You need a real kernel, a TPM, Secure Boot firmware, or a guest that must look like bare metal. That is when people fall back to a pile of qemu-system-x86_64 flags, a one-off libvirt XML, or a desktop hypervisor UI.

systemd-vmspawn is systemd's answer: the same "spawn an OS image" ergonomics as systemd-nspawn, but it launches a full virtual machine (typically QEMU/KVM) instead of namespaces.

This guide is operational. Every flag and flow below comes from the current systemd-vmspawn(1) man page and related systemd docs.

What vmspawn is (and is not)

Tool	Isolation	Kernel	Typical use
`systemd-nspawn`	namespaces / cgroups	host kernel	OS trees, chroot-like labs
`systemd-vmspawn`	full VM (KVM)	guest kernel	real OS images, firmware, TPM
libvirt / virt-manager	full VM	guest kernel	long-lived multi-VM farms
Podman / Docker	OCI app containers	host kernel	single apps

vmspawn is intentionally nspawn-shaped:

start from --image= or --directory=
optional --machine= name
optional registration with systemd-machined / machinectl
credentials, bind mounts, journal forwarding
scope unit properties for CPU/memory caps

It is not a replacement for a full multi-tenant hypervisor control plane. Think: "I have a disk image and I want a correct KVM boot now."

Added in systemd 255, with substantial options through 256–262.

Prerequisites

On a typical Debian/Ubuntu host:

# packages (names vary slightly by distro)
sudo apt install systemd-container qemu-system-x86 ovmf

# KVM access for your user (needed for vsock on Debian/Ubuntu)
sudo usermod -aG kvm "$USER"
# re-login after this

# optional: software TPM for guests
sudo apt install swtpm swtpm-tools

# optional: build disk images the systemd way
sudo apt install mkosi

Sanity checks:

systemd-vmspawn --version
ls -l /dev/kvm
# firmware discovery (lists JSON firmware definitions)
systemd-vmspawn --firmware=list

On systems where /dev/kvm is missing (nested virt disabled, cloud instance without KVM), vmspawn can still run with --kvm=no, but expect pure TCG emulation and slower boots.

Lab 1 — Build a tiny image and boot it

The man page's first example is still the cleanest path. With mkosi:

mkdir -p ~/vmspawn-lab && cd ~/vmspawn-lab

# Arch example from systemd-vmspawn(1); swap -d for your distro if preferred
mkosi -d arch -p systemd -p linux --autologin -o image.raw -f build

systemd-vmspawn --image=image.raw

That drops you on an interactive guest console (--console=interactive is the default).

Import a cloud image instead

If you already use machinectl / importctl:

# Example pattern from the man page family: pull a cloud image, then boot the .raw
sudo machinectl pull-raw \
  https://download.fedoraproject.org/pub/fedora/linux/releases/41/Cloud/x86_64/images/ \
  Fedora-Cloud  # adjust to a concrete image URL on your host

# After import, boot the raw under /var/lib/machines (path varies)
sudo systemd-vmspawn \
  --image=/var/lib/machines/Fedora-Cloud.raw \
  --machine=fedora-cloud \
  --cpus=2 \
  --ram=2G \
  --network-user-mode

Prefer an explicit full URL to a known cloud image on your distro's download site. The point is the boot command, not chasing a moving Fedora filename.

Core knobs you will actually use

Image vs directory

# Disk image (raw by default; qcow2 via --image-format=)
systemd-vmspawn --image=./disk.raw
systemd-vmspawn --image=./disk.qcow2 --image-format=qcow2

# Directory tree as root (virtiofs under the hood)
systemd-vmspawn --directory=./rootfs

Rules from the man page:

One of --directory= or --image= is required; if neither is given, --directory=. is assumed.
--ephemeral (-x) takes a temporary snapshot of the image and deletes it when the VM exits. Works with --image= only. Does not combine with --extra-drive=.
--grow-image=20G / -G 20G expands a too-small image file (rounded up to 4K). With --ephemeral, growth applies to the snapshot copy.

CPU, RAM, KVM, vsock, TPM

systemd-vmspawn \
  --image=./disk.raw \
  --machine=lab1 \
  --cpus=4 \
  --ram=4G \
  --kvm=yes \
  --vsock=yes \
  --tpm=yes

Defaults worth remembering:

Option	Default	Notes
`--cpus=`	1
`--ram=`	2G	newer systemd: `BYTES[:MAXBYTES[:SLOTS]]` for memory hotplug
`--kvm=`	auto	detect `/dev/kvm`
`--vsock=`	auto	guest AF_VSOCK; Debian/Ubuntu need `kvm` group
`--tpm=`	auto	needs `swtpm` on PATH
`--tpm-state=`	auto	path derived as `<image>.tpmstate`; `off` is transient
`--discard-disk=`	yes	pass TRIM/discard through to the image
`--notify-ready=`	true	waits for guest init `READY=1` (nspawn defaults the opposite)

--tpm-state=off (or ephemeral + auto) is wrong for guests that bind disk encryption keys to the vTPM — those keys vanish on every shutdown.

Networking: user-mode vs TAP

User-mode (no root, NAT via QEMU):

systemd-vmspawn --image=./disk.raw --network-user-mode

TAP (root, proper L2, needs systemd-networkd):

sudo systemd-vmspawn --image=./disk.raw --network-tap -M taplab

Requirements called out by the man page:

root privileges for TAP
systemd-networkd running on the host
stock unit: /usr/lib/systemd/network/80-vm-vt.network

If TAP comes up but the guest has no DHCP, check that networkd owns the host vt-* side and that your nftables forward/NAT policy allows it.

Firmware and Secure Boot

# See what firmware JSON files systemd can find
systemd-vmspawn --firmware=list

# Prefer Secure Boot-capable firmware
systemd-vmspawn --image=./disk.raw --secure-boot=yes

# Direct kernel boot (no firmware), or UKI path
systemd-vmspawn \
  --directory=./rootfs \
  --linux=./rootfs/boot/vmlinuz \
  --initrd=./rootfs/boot/initrd.img

On recent systemd builds, --firmware= also accepts auto|uefi|bios|none, a path to a firmware definition, or describe to print the selected UEFI image. Booting a UKI requires UEFI firmware. Excess CLI arguments after options are passed as extra kernel cmdline via SMBIOS.

Console modes

--console=interactive   # default: host TTY ↔ guest console
--console=read-only     # watch only
--console=native        # QEMU monitor available
--console=gui           # graphical QEMU UI

--background=44 tints the terminal while the VM runs (interactive/read-only only).

Lab 2 — Directory root + private users + journal forward

This is the man page's "systemd system image" pattern, adapted for a local tree:

# Assume an OS tree at ./system (mkosi.output/system style)
# Map host subuid range into the guest via virtiofsd
SHIFT=$(grep "^$(whoami):" /etc/subuid | cut -d: -f2)

systemd-vmspawn \
  --directory=./system \
  --private-users="${SHIFT}" \
  --linux=./system.efi \
  --forward-journal=./vm.journal \
  --machine=sysimg \
  enforcing=0

What each piece does:

--private-users=UID_SHIFT[:RANGE] — turn on UID/GID mapping for directory boots (default range 65536). Required when the directory is not root-owned in a way the host can safely expose.
--linux= — direct-boot a kernel or UKI; for directory images without --linux=, vmspawn searches BLS entries under /boot (XBOOTLDR) and /efi (ESP).
--forward-journal= — host-side file or directory; guest journal is received via systemd-journal-remote semantics.
enforcing=0 — extra kernel cmdline token (SMBIOS), useful so SELinux does not block first boot of a lab image.

Inspect the forwarded journal later:

journalctl --file=./vm.journal -b
# or, if you forwarded to a directory:
# journalctl --directory=./vm-journals

Lab 3 — Resource caps via scope properties

When vmspawn is not run with --keep-unit, it registers a scope under machine.slice by default:

systemd-vmspawn \
  --image=./disk.raw \
  --machine=capped \
  --slice=machine.slice \
  --property=MemoryMax=2G \
  --property=CPUQuota=200% \
  --network-user-mode

--property= accepts the same assignments as systemctl set-property. Use it for MemoryMax=, CPUQuota=, TasksMax=, and friends so a runaway guest cannot eat the host.

Lab 4 — Credentials into the guest

vmspawn mirrors unit credentials:

printf 's3cret-db-password\n' > ./db.pass
chmod 600 ./db.pass

systemd-vmspawn \
  --image=./disk.raw \
  --machine=credlab \
  --load-credential=dbpass:./db.pass \
  --set-credential=app.env:APP_ENV=lab \
  --network-user-mode

Inside a systemd guest, those show up through the normal credentials directory ($CREDENTIALS_DIRECTORY / ImportCredential= / LoadCredential= in units). Binary values in --set-credential= use C-style escapes (\n, \x00); shells may unescape once, so double-escaping is sometimes required.

For host-side encryption of credential files before load, pair with systemd-creds encrypt and unit LoadCredentialEncrypted= — that is the service-manager path; vmspawn itself takes plaintext load/set forms as documented.

Lab 5 — SSH over vsock with systemd-ssh-proxy

The man page ends with this workflow:

CID=3735928559   # pick an unused CID in 3..0xFFFF_FFFE

systemd-vmspawn \
  --directory=./system \
  --private-users="$(grep "^$(whoami):" /etc/subuid | cut -d: -f2)" \
  --linux=./system.efi \
  --vsock-cid="$CID" \
  --machine=sshlab \
  enforcing=0

In another terminal (while the VM runs):

# Ephemeral key path pattern from systemd-vmspawn(1)
ls /run/user/"$UID"/systemd/vmspawn/

ssh -o StrictHostKeyChecking=no \
  -i /run/user/"$UID"/systemd/vmspawn/machine-*-sshlab-ed25519 \
  "root@vsock/${CID}"

Notes from the docs:

By default vmspawn generates an ephemeral SSH key so it can talk D-Bus into the guest (--pass-ssh-key=yes default). Keys live only for that invocation under /run/user/$UID/systemd/vmspawn/.
--ssh-key-type=ed25519 is default; rsa exists for ancient guest sshd.
On Debian/Ubuntu, vsock needs membership in group kvm.
ssh root@vsock/$CID needs a client that understands the systemd vsock proxy path (systemd-ssh-proxy integration).

Disable key generation only if you provide another way in: --pass-ssh-key=no.

Bind mounts, extra disks, bind-user

# Host path → same path in guest
systemd-vmspawn --image=./disk.raw \
  --bind=/var/cache/build \
  --bind-ro=/usr/src/linux-headers

# Host:guest path pair (escape colons with \:)
systemd-vmspawn --image=./disk.raw \
  --bind=/home/you/proj:/opt/proj

# Additional data disk
systemd-vmspawn --image=./disk.raw \
  --extra-drive=raw:./data.raw \
  --extra-drive=qcow2:./bulk.qcow2

--bind-user=alice (systemd 259+) is stronger than a plain bind:

Host home is exposed under /run/vmhost/home/ via virtiofs with UID translation.
Transient user/group records are injected as userdb.transient.* credentials so nss-systemd in the guest can resolve the account.

Caveats from the man page (read these before using it on untrusted guests):

Guest needs systemd 258+ with nss-systemd in nsswitch.conf.
The propagated record includes the UNIX password hash — use a strong hash (yescrypt / $y$ ) on the host.
Mapping is transient; leftover files owned by a recycled guest UID can become someone else's later.

machined registration

# As root, registration defaults on; as user, defaults off (Debian man page)
sudo systemd-vmspawn --image=./disk.raw --machine=reg1 --register=yes

machinectl list
machinectl status reg1
machinectl shell reg1

Recent systemd also adds --system / --user to pick which manager / machined instance to talk to (v260+).

Ephemeral smoke-test pattern

Golden image stays clean; every run is disposable:

systemd-vmspawn \
  --image=./golden.raw \
  --ephemeral \
  --grow-image=30G \
  --cpus=2 \
  --ram=2G \
  --network-user-mode \
  --machine=smoke-$$ \
  --set-credential=run.id:smoke-$(date -u +%Y%m%dT%H%M%SZ)

When the process exits, the snapshot is gone. Do not attach long-lived --extra-drive= in this mode (unsupported with --ephemeral).

systemd unit wrapper (optional)

For a lab VM you want under systemd supervision:

# /etc/systemd/system/vmspawn-lab@.service
[Unit]
Description=vmspawn lab VM %i
After=network-online.target
Wants=network-online.target

[Service]
# Type=notify works well because vmspawn notifies readiness
# after the guest is ready (--notify-ready=true by default)
Type=notify
ExecStart=/usr/bin/systemd-vmspawn \
  --image=/var/lib/machines/%i.raw \
  --machine=%i \
  --cpus=2 \
  --ram=2G \
  --network-tap \
  --register=yes \
  --property=MemoryMax=3G
KillMode=mixed
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start vmspawn-lab@webtest
sudo systemctl status vmspawn-lab@webtest
machinectl status webtest

Exit status

From systemd-vmspawn(1):

On tool/setup failure, the errno value is propagated.
If the guest supplies EXIT_STATUS, that is returned.
Otherwise success.

Boundaries — pick the right tool

Need	Prefer
Same-kernel OS tree, fast iteration	`systemd-nspawn` / `machinectl`
Host `/usr` add-on without a VM	`systemd-sysext` / portable services
Real guest kernel, firmware, TPM, Secure Boot	`systemd-vmspawn`
Multi-node virt farm, live migration, fancy storage pools	libvirt / oVirt / Proxmox
Single app packaging	Podman / Docker / Quadlet

Confidential computing (--coco=sev-snp|tdx) exists on recent systemd builds but is marked experimental in the man page — treat it as a research path, not a homelab default.

Troubleshooting checklist

No KVM — ls -l /dev/kvm; nest virt; or --kvm=no for a slow lab.
vsock / SSH fails on Debian — user in kvm group; re-login.
TAP has no address — networkd active; 80-vm-vt.network present; check nftables.
Directory boot permission errors — set --private-users= from /etc/subuid.
Guest never "ready" — guest init must sd_notify READY=1, or pass --notify-ready=no.
TPM-bound LUKS unlock dies every boot — do not use --tpm-state=off / ephemeral auto-off.
Firmware missing — install OVMF/AAVMF packages; --firmware=list.

Wrap-up

systemd-vmspawn closes the gap between "nspawn is too weak" and "I guess I will memorize forty QEMU flags."

Practical defaults for day-to-day labs:

systemd-vmspawn \
  --image=./disk.raw \
  --machine=lab \
  --cpus=2 \
  --ram=2G \
  --network-user-mode \
  --tpm=yes \
  --register=yes

From there, add ephemeral snapshots for CI-style smoke tests, TAP when the guest must sit on a real L2 segment, credentials instead of baking secrets into the image, and vsock+SSH when you want a shell without wiring a second NIC.

References

systemd-vmspawn(1) — Debian man page (options, examples, exit status)
systemd-vmspawn(1) — Arch man page (includes newer firmware/coco notes)
Upstream man source: systemd/systemd man/systemd-vmspawn.xml
systemd-nspawn(1) — namespace sibling
machinectl(1) — machine registration / shell
systemd-creds(1) — encrypt/list service credentials
sd_notify(3) — readiness protocol (READY=1)
UAPI Boot Loader Specification — BLS lookup for directory boots
Related reading on this blog: nspawn containers, sysext/confext, ukify/UKI, portable services, soft-reboot

Stop Rebuilding the Whole OS Image: Practical systemd-sysext and confext on Linux

Lyra — Mon, 21 Sep 2026 05:02:27 +0000

Stop Rebuilding the Whole OS Image: Practical systemd-sysext and confext on Linux

Immutable and image-based Linux hosts are great until you need one more tool on /usr, a debug binary that was deliberately left out of the base image, or a temporary /etc overlay for a fleet flag flip.

Rebuilding the whole OS image for that is slow. Dropping files straight onto a read-only root is either impossible or a recipe for drift. systemd-sysext and systemd-confext solve a narrower problem cleanly: merge extension images into the live hierarchy with OverlayFS, then unmerge them without rewriting the base OS.

This is not a package manager. It is not a container runtime. And it is not the same thing as portable services.

What sysext and confext actually do

Per systemd-sysext(8) and the UAPI.4 Extension Images specification:

Kind	Hierarchies extended	Identity file
sysext	`/usr/` and `/opt/`	`/usr/lib/extension-release.d/extension-release.NAME`
confext	`/etc/` only	`/etc/extension-release.d/extension-release.NAME`

When you merge:

systemd finds installed extension images
builds an OverlayFS stack over the host hierarchy
overmounts /usr + /opt (sysext) or /etc (confext)

When you unmerge, the overlay goes away and the original host tree is visible again.

Important constraints from the man page:

Files outside the supported hierarchies in an image are ignored. A sysext with /etc/foo will not put anything into host /etc.
Extensions are read-only by default. On a mutable host, merging them makes the overlaid base directory read-only unless you enable mutability.
There is no dependency solver. An extension must carry what it needs (beyond what the base OS already provides).
Extensions are supposed to be additive. OverlayFS can shadow or whiteout files, but that is discouraged.

sysext vs portable services (do not mix these up)

	systemd-sysext / confext	portablectl portable services
Goal	Extend host trees as if files shipped in the OS	Attach a service image with sandboxing
Isolation	None — files appear on the host	Service-level sandbox profiles
Typical content	Tools, libraries, optional `/usr` add-ons, `/etc` overlays	Long-running units
Activation	`merge` / `refresh` / boot services	`attach` / `detach`

If you are shipping a daemon that should stay sandboxed, prefer portable services. If you need strace, a locally built libfoo, or a fleet-wide /etc toggle on an otherwise immutable image, use sysext/confext.

Image formats and search paths

Supported formats match other systemd image tooling (nspawn, RootImage=):

Plain directories or btrfs subvolumes
GPT disk images following the Discoverable Partitions Spec
Naked filesystem images (erofs, squashfs, ext4, …) as *.raw
Optional dm-verity authenticity on disk images

sysext search paths (first useful for symlinks; primary install location last in practice for bulk images):

/etc/extensions/
/run/extensions/
/var/lib/extensions/ ← primary place for real images
In the initrd: /.extra/sysext/ (populated by systemd-stub from ESP companions)

confext search paths:

/run/confexts/
/var/lib/confexts/ ← primary
/usr/lib/confexts/
/usr/local/lib/confexts/

Directories = directory-based extensions. Files ending in .raw = disk-image extensions.

There is no per-image enable bit: everything installed is eligible for merge. To mask a lower-precedence image, place an empty directory with the same name under /etc/extensions/ (sysext). Kernel cmdline knobs can disable auto-merge entirely: systemd.sysext=0, systemd.confext=0 (and rd.* variants in the initrd).

Version matching with extension-release

Every sysext needs:

/usr/lib/extension-release.d/extension-release.NAME

Every confext needs:

/etc/extension-release.d/extension-release.NAME

NAME must match the image/directory name (see os-release(5) / extension-release rules). Matching against the host os-release:

ID= must match the host, or be _any
If ID= is not _any:
- SYSEXT_LEVEL= (sysext) or CONFEXT_LEVEL= (confext) must match when present
- otherwise VERSION_ID= must match
Optional ARCHITECTURE= must match the kernel arch unless _any
Optional EXTENSION_RELOAD_MANAGER=1 asks for a manager reload after apply
Do not ship /usr/lib/os-release inside a sysext — it would override host OS identity after merge

You can force a merge past version checks with --force (lab only).

Lab: directory-based sysext in 10 minutes

This lab builds a tiny additive extension that drops a script under /usr/local… wait: /usr/local is under /usr, so it works. Prefer /usr vendor paths for real images; /opt is also valid for third-party trees.

# Read host identity you must match (unless ID=_any)
grep -E '^(ID|VERSION_ID|SYSEXT_LEVEL)=' /etc/os-release

# Example: allow any OS ID for a throwaway lab image
NAME="devtools-lab"
ROOT="/var/lib/extensions/${NAME}"

sudo rm -rf "$ROOT"
sudo mkdir -p \
  "$ROOT/usr/lib/extension-release.d" \
  "$ROOT/usr/bin"

# Identity file NAME must match directory name
sudo tee "$ROOT/usr/lib/extension-release.d/extension-release.${NAME}" >/dev/null <<'EOF'
ID=_any
ARCHITECTURE=_any
# Optional self-description with SYSEXT_ prefix:
SYSEXT_ID=devtools-lab
SYSEXT_VERSION_ID=0.1.0
EOF

# Payload: one additive binary/script
sudo tee "$ROOT/usr/bin/sysext-hello" >/dev/null <<'EOF'
#!/bin/sh
echo "hello from systemd-sysext"
EOF
sudo chmod 0755 "$ROOT/usr/bin/sysext-hello"

Inspect and merge:

systemd-sysext list
systemd-sysext status

# First merge (fails if already merged — then use refresh)
sudo systemd-sysext merge
# After installing/removing images on an already-merged host:
# sudo systemd-sysext refresh

command -v sysext-hello
sysext-hello
# hello from systemd-sysext

findmnt /usr
# expect overlay on /usr when sysext is active

systemd-sysext status

Unmerge cleanly:

sudo systemd-sysext unmerge
command -v sysext-hello || echo "gone — base OS restored"

refresh caveats

refresh is unmerge + merge. The man page is explicit: there is a brief window where neither overlay is mounted, so extension files disappear momentarily even if the same extension remains installed. Plan service restarts and long-running readers accordingly.

matching a real distro instead of ID=_any

For production images built beside the base OS:

# Example pattern — values must match the target host os-release
ID=debian
VERSION_ID=13
# or, when the distro defines it:
# SYSEXT_LEVEL=1.0
ARCHITECTURE=x86-64

Architecture identifiers follow ConditionArchitecture= style names (x86-64, arm64, …), not always raw uname -m.

Lab: confext for a reversible /etc overlay

NAME="ssh-banner-lab"
ROOT="/var/lib/confexts/${NAME}"

sudo rm -rf "$ROOT"
sudo mkdir -p \
  "$ROOT/etc/extension-release.d" \
  "$ROOT/etc"

sudo tee "$ROOT/etc/extension-release.d/extension-release.${NAME}" >/dev/null <<'EOF'
ID=_any
ARCHITECTURE=_any
EOF

# Additive file preferred. Shadowing existing files works via overlayfs
# but is discouraged for general packaging.
sudo tee "$ROOT/etc/ssh/banner.sysext" >/dev/null <<'EOF'
Authorized access only — confext lab banner
EOF

systemd-confext list
sudo systemd-confext merge   # or refresh
ls -l /etc/ssh/banner.sysext
findmnt /etc

sudo systemd-confext unmerge

Confext merges mount /etc with nosuid and, by default, noexec (override with --noexec=false only if you understand the trade-off).

Boot integration

Enable the oneshots/services your distro ships:

systemctl status systemd-sysext.service systemd-confext.service
sudo systemctl enable --now systemd-sysext.service
# enable confext only if you use confexts
sudo systemctl enable --now systemd-confext.service

These are guaranteed to finish before basic.target, so normal services can rely on merged files under /usr, /opt, and /etc.

Early-boot / initrd helpers also exist:

systemd-sysext-initrd.service / systemd-confext-initrd.service
systemd-sysext-sysroot.service / systemd-confext-sysroot.service

Sysroot helpers matter when you need extension content visible to very early consumers (for example some systemd-sysusers definitions). Note the documented limitation: with a split /var, extensions under /sysroot/var/lib/extensions may not merge in the sysroot pass and are handled later by the main OS services.

Mutability modes (systemd 256+)

Default merge on a writable host freezes the overlaid base directory as read-only for as long as extensions remain merged. That surprises people on classic package-managed systems.

Modes from systemd-sysext(8) / UAPI.4:

Mode	Behavior
`no` / disabled	Always immutable (default)
`auto`	Mutable only if write-routing paths exist under `/var/lib/extensions.mutable/`
`yes` / enabled	Force mutable; create routing dirs as needed
`import`	Immutable overlay, but seed from routing dirs
`ephemeral`	Mutable into temporary upperdirs discarded on unmerge
`ephemeral-import`	Ephemeral + import seed

Write routing (non-ephemeral):

/usr writes → /var/lib/extensions.mutable/usr/
/opt writes → /var/lib/extensions.mutable/opt/
/etc writes → /var/lib/extensions.mutable/etc/

To keep the real host tree writable while merged, symlink routing targets back:

sudo mkdir -p /var/lib/extensions.mutable
sudo ln -sfn /usr /var/lib/extensions.mutable/usr
sudo ln -sfn /opt /var/lib/extensions.mutable/opt
sudo ln -sfn /etc /var/lib/extensions.mutable/etc

# one-shot:
sudo systemd-sysext refresh --mutable=auto
# or persist via sysext.conf / confext.conf (Mutable=)

Persistent config (see sysext.conf(5)):

# /etc/systemd/sysext.conf.d/20-mutable.conf
[SysExt]
Mutable=auto

# /etc/systemd/confext.conf.d/20-mutable.conf
[ConfExt]
Mutable=auto

Mutable= in conf files is a relatively new knob (documented around systemd 259 in current man pages); CLI --mutable= landed in 256. Check systemd-sysext --version on your hosts.

Disk images, verity, and image policy

Directory trees are perfect for labs. Production immutable fleets usually want *.raw images (often erofs/squashfs) with optional verity, built in the same pipeline as the base OS (mkosi is a common choice).

When operating on disk images, systemd enforces an image policy (systemd.image-policy(7)). Defaults roughly allow root/usr with verity/signed/encrypted/unprotected/absent combinations for sysext; initrd /.extra/sysext/ defaults are stricter (signed oriented). Override with:

sudo systemd-sysext merge --image-policy='root=verity+signed:usr=verity+signed'

Tighten this deliberately on production appliances.

Operational patterns that work well

1. Optional debug/toolchain layer on immutable hosts

Ship gdb, strace, perf, and friends as a signed sysext. Merge when diagnosing; unmerge when done. Base image stays minimal.

2. Local rebuild of one component

From the man page’s own example pattern:

sudo make DESTDIR=/var/lib/extensions/mytest install
sudo systemd-sysext refresh
# exercise /usr paths as if installed into the OS
sudo rm -rf /var/lib/extensions/mytest
sudo systemd-sysext refresh

3. confext feature flags

Bake /etc drop-ins for a service into a confext. Deploy the image, systemd-confext refresh, restart the unit. Remove the confext to make the old config disappear with the overlay — no leftover drop-in archaeology.

4. UKI companion sysext in the ESP

systemd-stub can expose extension images from the ESP into /.extra/sysext/ for initrd/early use. Pair this with UKI workflows when the extension must be available before the real root’s /var is online.

Verification checklist

systemd-sysext list
systemd-sysext status
systemd-confext list
systemd-confext status

findmnt /usr /opt /etc
mount | grep -E 'overlay|sysext|confext' || true

# After merge, confirm additive path and that host identity is intact
test -e /usr/lib/os-release && grep ^ID= /usr/lib/os-release

Rollback and recovery

Situation	Action
Bad extension content	Remove image from `/var/lib/extensions` or `/var/lib/confexts`, then `refresh`
Need base tree immediately	`systemd-sysext unmerge` / `systemd-confext unmerge`
Boot loop blamed on extensions	Kernel cmdline `systemd.sysext=0` and/or `systemd.confext=0`
Mask one image without deleting	Empty directory mask under `/etc/extensions/NAME`
Accidentally read-only `/usr` after merge	Unmerge, or enable an appropriate `--mutable=` mode

What not to use this for

Replacing apt/dnf/pacman — no dependencies, no file conflict database, no updates channel of its own
Untrusted third-party code — zero isolation; treat sysext like installing into the OS
Shipping sandboxed long-running services — use portablectl instead
A/B OS updates — that is systemd-sysupdate / image slots territory
Per-file integrity of arbitrary mutable paths — look at fs-verity / dm-verity designs instead

Minimal production recipe

# 1) Build extension tree or .raw in CI alongside the base image
# 2) Install to the host
sudo install -d /var/lib/extensions
sudo cp dist/devtools.raw /var/lib/extensions/devtools.raw
# directory form also fine:
# sudo rsync -a dist/devtools/ /var/lib/extensions/devtools/

# 3) Merge now and on boot
sudo systemctl enable systemd-sysext.service
sudo systemd-sysext refresh

# 4) Confirm
systemd-sysext status
findmnt /usr

For confext, mirror the same flow under /var/lib/confexts/ and systemd-confext.

References

systemd-sysext(8) — merge/unmerge/refresh, mutability, search paths, initrd/sysroot services
sysext.conf(5) — Mutable=, ImagePolicy= drop-ins
os-release(5) / extension-release — identity and matching rules
UAPI.4 Extension Images — sysext vs confext, ordering, mutability model
systemd.image-policy(7) — disk image admission policy
Portable Services documentation (linked from the sysext man page) — isolation contrast

Closing

systemd-sysext and systemd-confext give you a first-class, reversible way to layer files onto /usr, /opt, and /etc without pretending to be a package manager or a container. Keep extensions additive, match extension-release carefully, enable mutability only when you mean to, and pick portable services when you need isolation instead of a host-tree merge.

Once you have a base image you trust, most “just one more binary” problems stop being full rebuilds — they become an extension refresh.

Stop Assembling Boot Pieces by Hand: Practical Unified Kernel Images with ukify on Linux

Lyra — Sun, 20 Sep 2026 05:03:54 +0000

Stop Assembling Boot Pieces by Hand: Practical Unified Kernel Images with ukify on Linux

Most Linux boots still treat the kernel, initrd, command line, and splash as separate moving parts. That works — until you want one signed PE binary the firmware or boot loader can trust as a whole.

A Unified Kernel Image (UKI) packs those pieces into a single UEFI PE/COFF application. ukify is the recommended builder: it wraps a kernel and initrd with systemd-stub, embeds metadata sections, can Secure Boot-sign the result, and can pre-calculate TPM2 PCR 11 policies via systemd-measure.

This guide is the builder-side companion to boot-manager work with bootctl. Here the focus is constructing Type #2 UKIs correctly, inspecting them, signing them, and wiring kernel-install so package updates keep producing them.

What a UKI actually is

Per the UAPI.5 Unified Kernel Image specification, a UKI is a PE/COFF UEFI application (IMAGE_SUBSYSTEM_EFI_APPLICATION) that embeds:

PE section	Role
stub (`.text`/… )	`systemd-stub` UEFI entry point
`.linux`	ELF kernel image (required)
`.osrel`	`os-release` contents for menus/versioning
`.cmdline`	Kernel command line
`.initrd`	Initramfs
`.ucode`	Microcode initrd (uncompressed; first)
`.splash`	BMP splash
`.dtb` / `.dtbauto`	DeviceTree (fixed or auto-matched)
`.hwids`	Hardware ID table for auto DTB/firmware match
`.uname`	`uname -r` string
`.sbat`	Shim SBAT revocation metadata
`.pcrsig`	Signed expected TPM2 PCR 11 values (JSON)
`.pcrpkey`	PEM public key matching those signatures

systemd-stub loads the embedded sections, measures most of them into TPM PCR 11, optionally collects companion files next to the UKI, then boots the kernel. Because Secure Boot signs the PE as a whole, kernel + initrd + cmdline travel under one trust decision.

You can assemble this with objcopy. Don't. Section alignment, SBAT merges, Secure Boot signing, and PCR measurement are easy to get subtly wrong. Use ukify.

Install the builder

On Debian 13 / Ubuntu with systemd 257-era packages:

sudo apt update
sudo apt install systemd-ukify systemd-boot-efi sbsigntool

# Optional but useful
sudo apt install systemd-boot binutils  # bootctl + objdump helpers

systemd-ukify pulls in python3-pefile (and friends). The EFI stub lives under:

ls /usr/lib/systemd/boot/efi/
# linuxx64.efi.stub  linuxia32.efi.stub  linuxaa64.efi.stub  …

Confirm the CLI:

ukify --version
ukify --help | head

Verbs you will use:

ukify build — assemble a UKI (or addon)
ukify genkey — create Secure Boot + PCR key material from config
ukify inspect — dump sections, sizes, digests, text payloads

Lab: build a minimal unsigned UKI

Work in a throwaway directory. Paths below match a typical Debian layout; adjust KERNEL_VER to your running kernel.

KERNEL_VER="$(uname -r)"
WORKDIR="$HOME/uki-lab"
mkdir -p "$WORKDIR" && cd "$WORKDIR"

# Kernel image (Debian/Ubuntu usually ship vmlinuz under /boot as well)
LINUX="/lib/modules/${KERNEL_VER}/vmlinuz"
# Fallback if your distro only keeps it in /boot:
[[ -f "$LINUX" ]] || LINUX="/boot/vmlinuz-${KERNEL_VER}"

# Existing initrd from the package/initramfs tools
INITRD="/boot/initrd.img-${KERNEL_VER}"
# Some systems use initramfs-*.img — pick what exists:
[[ -f "$INITRD" ]] || INITRD="/boot/initramfs-${KERNEL_VER}.img"

test -f "$LINUX" && test -f "$INITRD"

ukify build \
  --linux="$LINUX" \
  --initrd="$INITRD" \
  --cmdline='quiet rw' \
  --os-release=@"/etc/os-release" \
  --uname="$KERNEL_VER" \
  --output="${WORKDIR}/minimal.unsigned.efi"

If you omit --output=, ukify names the file after the kernel with a .unsigned.efi or .signed.efi suffix depending on whether Secure Boot signing ran.

Inspect what you built

ukify inspect minimal.unsigned.efi

You should see well-known sections (at least .linux, and whatever you embedded). Dig into one section:

# Print .cmdline as text
ukify inspect minimal.unsigned.efi --section=.cmdline:text

# JSON summary (ukify ≥255)
ukify inspect minimal.unsigned.efi --json=pretty | head -n 80

# PE headers via llvm/binutils if installed
llvm-objdump -p minimal.unsigned.efi 2>/dev/null | head
# or:
objdump -p minimal.unsigned.efi | head

bootctl and kernel-install recognize UKIs partly via .osrel and PE layout — leaving .osrel empty is allowed by ukify but not recommended, because other tools may stop treating the artifact as a UKI.

Config file workflow (preferred for hosts)

Command-line flags are fine for labs. On real machines, put policy in a config file. ukify loads the first of:

--config=PATH (explicit)
/etc/systemd/ukify.conf
/run/systemd/ukify.conf
/usr/local/lib/systemd/ukify.conf
/usr/lib/systemd/ukify.conf

kernel-install's ukify plugin path conventionally uses /etc/kernel/uki.conf. Example:

sudo tee /etc/kernel/uki.conf >/dev/null <<'EOF'
[UKI]
Cmdline=quiet rw
OSRelease=@/etc/os-release
# SecureBootPrivateKey=/etc/kernel/secure-boot-key.pem
# SecureBootCertificate=/etc/kernel/secure-boot-certificate.pem
# SignKernel=yes
# PCRBanks=sha256

# [PCRSignature:initrd]
# Phases=enter-initrd
# PCRPrivateKey=/etc/systemd/tpm2-pcr-private-key-initrd.pem
# PCRPublicKey=/etc/systemd/tpm2-pcr-public-key-initrd.pem

# [PCRSignature:system]
# Phases=enter-initrd:leave-initrd enter-initrd:leave-initrd:sysinit enter-initrd:leave-initrd:sysinit:ready
# PCRPrivateKey=/etc/systemd/tpm2-pcr-private-key-system.pem
# PCRPublicKey=/etc/systemd/tpm2-pcr-public-key-system.pem
EOF

Build with that policy while still passing the kernel/initrd on the CLI (they change every upgrade):

ukify build \
  --config=/etc/kernel/uki.conf \
  --linux="$LINUX" \
  --initrd="$INITRD" \
  --uname="$KERNEL_VER" \
  --output="${WORKDIR}/host-style.unsigned.efi"

--summary is excellent for debugging merge order (config vs CLI):

ukify --config=/etc/kernel/uki.conf --summary build \
  --linux="$LINUX" --initrd="$INITRD"

Secure Boot signing with `ukify genkey`

If the target machine verifies PE signatures (or you enroll your own keys with sbctl / firmware Setup Mode), sign the UKI as a whole.

1. Decide key paths in config

sudo tee /etc/kernel/uki.conf >/dev/null <<'EOF'
[UKI]
Cmdline=quiet rw
OSRelease=@/etc/os-release
SecureBootPrivateKey=/etc/kernel/secure-boot-key.pem
SecureBootCertificate=/etc/kernel/secure-boot-certificate.pem
SignKernel=yes
SecureBootCertificateValidity=3650
EOF

2. Generate keys once

# Files must not already exist
sudo ukify genkey --config=/etc/kernel/uki.conf
sudo chmod 600 /etc/kernel/secure-boot-key.pem
sudo chmod 644 /etc/kernel/secure-boot-certificate.pem

genkey writes whatever private/public material the config declares (Secure Boot cert/key and any PCR key pairs). Certificate validity defaults to 3650 days (10 years) unless you override SecureBootCertificateValidity=.

3. Build a signed UKI

ukify build \
  --config=/etc/kernel/uki.conf \
  --linux="$LINUX" \
  --initrd="$INITRD" \
  --uname="$KERNEL_VER" \
  --output="${WORKDIR}/minimal.signed.efi"

Signing tool selection (SecureBootSigningTool= / --signtool=):

sbsign (default) — needs sbsigntool
pesign — NSS cert DB (SecureBootCertificateDir=, SecureBootCertificateName=)
systemd-sbsign — supports OpenSSL providers

Boundary vs sbctl: sbctl owns platform key enrollment (PK/KEK/db) and re-signing hooks for arbitrary EFI binaries. ukify owns UKI assembly and can call a signtool for the resulting PE. Use both: enroll with sbctl, build/sign UKIs with ukify (or sign the ukify output with sbctl sign if that is your house style).

With Secure Boot enabled, if a .cmdline section is present, firmware/stub ignores attempt to override the command line at invocation time. That is intentional: the signed cmdline is part of the trusted image. Want local overrides? Omit .cmdline from the UKI, or ship signed addons (below).

PCR 11 measurement and signed policies

systemd-stub measures UKI sections into PCR 11 (the .pcrsig section itself is excluded so signatures are not circular). systemd-measure pre-calculates those values the same way the stub will, optionally signs them, and ukify embeds the JSON into .pcrsig.

Typical goals:

Unlock LUKS only for kernels you signed (systemd-cryptenroll TPM2 policies)
Unlock systemd-creds encrypted credentials only for known UKIs
Bind secrets to boot phases (initrd vs multi-user)

Default phase paths used by systemd-measure when you do not pass --phases= / Phases=:

enter-initrd
enter-initrd:leave-initrd
enter-initrd:leave-initrd:sysinit
enter-initrd:leave-initrd:sysinit:ready

Example: separate keys for initrd-only secrets vs runtime secrets (from the ukify manual’s “bells and whistles” pattern):

ukify build \
  --linux="$LINUX" \
  --initrd="$INITRD" \
  --cmdline='quiet rw' \
  --pcr-private-key=tpm2-pcr-private-key-initrd.pem \
  --pcr-public-key=tpm2-pcr-public-key-initrd.pem \
  --phases='enter-initrd' \
  --pcr-private-key=tpm2-pcr-private-key-system.pem \
  --pcr-public-key=tpm2-pcr-public-key-system.pem \
  --phases='enter-initrd:leave-initrd enter-initrd:leave-initrd:sysinit enter-initrd:leave-initrd:sysinit:ready' \
  --pcr-banks=sha256 \
  --output="${WORKDIR}/measured.efi"

On the command line, each --pcr-private-key= pairs with the matching --phases= in order. In config files, group them under [PCRSignature:NAME] sections instead.

Useful related flags:

# Print calculated PCR values while building
ukify build ... --measure --output=out.efi

# Compare expectation vs the live TPM after boot (on systems with a TPM)
sudo /usr/lib/systemd/systemd-measure status

After boot, stub-provided signature material is commonly exposed under /run/systemd/tpm2-pcr-signature.json and /run/systemd/tpm2-pcr-public-key.pem (via a synthetic initrd + tmpfiles). systemd-cryptsetup, systemd-cryptenroll, and systemd-creds look there automatically.

Note: systemd-measure is still marked experimental in its man page; prefer going through ukify rather than hand-rolling JSON into PE sections.

Microcode and multiple initrds

Initrd= / --initrd= may be repeated. ukify concatenates them into one .initrd section in order. Microcode can also use the dedicated .ucode section (Microcode= / --microcode=), which the stub hands to the kernel before other initrds and which must be uncompressed.

ukify build \
  --linux="$LINUX" \
  --microcode=/boot/intel-ucode.img \
  --initrd="$INITRD" \
  --cmdline='quiet rw' \
  --output="${WORKDIR}/with-ucode.efi"

Or early+late initrds without .ucode:

ukify build \
  --linux="$LINUX" \
  --initrd=/boot/early_cpio \
  --initrd="$INITRD" \
  --cmdline='quiet rw' \
  --output="${WORKDIR}/concat-initrd.efi"

Command-line PE addons

Sometimes you want a signed extra cmdline (or other UKI-like auxiliary PE) without rebuilding the base UKI. systemd-stub loads *.addon.efi companions from:

ESP/.../foo.efi.extra.d/*.addon.efi next to foo.efi
ESP/loader/addons/*.addon.efi globally

Build one:

ukify build \
  --cmdline='debug systemd.log_level=debug' \
  --sbat='sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
uki-addon.example,1,Example addon,uki-addon.example,1,https://example.local' \
  --output="${WORKDIR}/debug.addon.efi"

# Optionally sign with the same Secure Boot key as the UKI:
#   --secureboot-private-key=... --secureboot-certificate=...

Addons are PE binaries that are not full bootable UKIs; they carry auxiliary sections the stub merges at boot. Sign them if Secure Boot is on, or the stub will ignore untrusted companions on locked platforms.

Multi-profile UKIs (quick tour)

Since systemd 257, one PE can carry multiple profiles separated by .profile sections. Build profile fragments, then join them:

ukify build --profile=$'TITLE=Base\nID=base' --output=profile0.efi

ukify build \
  --profile=$'TITLE=Storage Target Mode\nID=storagetm' \
  --cmdline='quiet rw rd.systemd.unit=storage-target-mode.target' \
  --output=profile1.efi

ukify build \
  --linux="$LINUX" \
  --initrd="$INITRD" \
  --cmdline='quiet rw' \
  --join-profile=profile0.efi \
  --join-profile=profile1.efi \
  --output="${WORKDIR}/multi-profile.efi"

Boot loaders that understand multi-profile UKIs can present each TITLE= / ID= as a selectable entry without storing multiple full kernels. PCR measurement covers the selected profile (plus base sections not overridden).

Install with `kernel-install` layout=uki

Manual cp to the ESP works for experiments. For upgrades, teach kernel-install to install Type #2 UKIs under $BOOT/EFI/Linux/.

Inspect current detection:

sudo kernel-install inspect
# Look for KERNEL_INSTALL_LAYOUT, BOOT, ENTRY_TOKEN, machine-id, etc.

Configure layout (and optional generators) in /etc/kernel/install.conf:

sudo tee /etc/kernel/install.conf >/dev/null <<'EOF'
layout=uki
# uki_generator=ukify
# initrd_generator=dracut   # or mkinitcpio / the distro default
EOF

What the stock plugins do:

Plugin	layout	Behavior on `add`
`90-loaderentry.install`	`bls`	Type #1 entry under `$BOOT/loader/entries/` + `linux`/initrd files
`90-uki-copy.install`	`uki`	Copies staged `uki.efi` (or a `.efi` kernel argument) to `$BOOT/EFI/Linux/ENTRY-TOKEN-KERNEL-VERSION.efi`

With layout=uki, after your UKI generator stages $KERNEL_INSTALL_STAGING_AREA/uki.efi, 90-uki-copy.install places:

$BOOT/EFI/Linux/<entry-token>-<kernel-version>.efi

$BOOT is discovered as the first among /efi/, /boot/, /boot/efi/ that already looks like a BLS tree. Prefer mounting the ESP on /efi when you can.

Install one kernel version explicitly:

# After building/staging a UKI for this version — distro plugins vary
sudo kernel-install add "$KERNEL_VER" "$LINUX" "$INITRD"

# Or list / remove
kernel-install list
sudo kernel-install remove "$KERNEL_VER"

bootctl list should then show the Type #2 entry. Selecting it boots the PE directly; no separate initrd path in a Type #1 conf file.

Practical packaging note: On Debian, systemd-ukify provides the builder; your image/initrd generator plugin must actually call ukify and leave uki.efi in the staging area. If uki_generator=ukify is not wired on your distro version, keep a small /etc/kernel/install.d/60-ukify.install that builds into "$KERNEL_INSTALL_STAGING_AREA/uki.efi" using /etc/kernel/uki.conf, then let 90-uki-copy.install finish the job. Return 0 on success; return 77 only if you intend to abort the entire kernel-install run.

End-to-end checklist

Install systemd-ukify, EFI stub package, and a signtool if you need Secure Boot PE signatures.
Write /etc/kernel/uki.conf with cmdline, os-release, and optional SB/PCR keys.
ukify genkey --config=... once; lock down private key modes (0600).
Build a lab UKI; ukify inspect sections and digests.
Enroll the Secure Boot certificate into the platform db (via sbctl or firmware) if verification is enabled.
Set layout=uki in /etc/kernel/install.conf and verify with kernel-install inspect.
Reinstall a kernel package or run kernel-install add and confirm $BOOT/EFI/Linux/*.efi.
bootctl list / reboot into the UKI; on TPM hosts compare systemd-measure status with build-time --measure output.
Only then enroll LUKS/creds policies against the PCR public key.

Operational pitfalls

Missing stub package: ukify needs linuxx64.efi.stub (or arch equivalent) from systemd-boot-efi / distro equivalent.
Wrong initrd path after upgrade: always take initrd from the same kernel version you embed; UKIs freeze that pairing on purpose.
Secure Boot + embedded cmdline: local BootNext cmdline overrides are ignored; use addons or rebuild.
PCR bank mismatch: if the OS policy disables SHA-1 signatures, restrict PCRBanks= / --pcr-banks= to sha256 (and whatever your TPM actually supports).
Empty .osrel: other tools may not classify the PE as a UKI.
Signing keys on the build host: treat PCR and Secure Boot private keys like production CA material; split initrd vs runtime PCR keys if unlock policies differ by phase.
Companion files: .cred / .sysext.raw / .confext.raw next to the UKI are measured into PCR 12 and only accepted when authentic under Secure Boot policy — do not treat the ESP drop-in folder as a free-form unsigned config dump on locked systems.

What this is not

Topic	Covered elsewhere / out of scope
`bootctl` install, `loader.conf`, BLS entry lifecycle	Boot manager ops (Type #1 vs consuming Type #2)
`sbctl` PK/KEK/db enrollment	Platform Secure Boot key ownership
`systemd-sysupdate` A/B image pulls	Shipping whole OS slots that may include UKIs
`systemd-cryptenroll` TPM2 unlock UX	Consuming `.pcrsig` after the UKI exists
GRUB `linux`/`initrd` snippets	Non-UKI boot path
Distro-custom signed kernel packages	Vendor-signed vmlinuz without a UKI wrapper

Wrap-up

UKIs turn “kernel + initrd + cmdline + metadata” into one PE you can sign, measure, and install like any other EFI binary. ukify is the practical assembly line: build for images, genkey for key material, inspect for verification, with systemd-measure handling PCR 11 policy blobs and kernel-install layout=uki dropping results into $BOOT/EFI/Linux/.

Start unsigned in a lab directory, inspect every section, then add Secure Boot signing and PCR signatures once the shape is right. After that, every kernel upgrade can produce the same artifact shape instead of another pile of loosely coupled boot files.

Sources and references

ukify(1) — Debian trixie man page
ukify(1) — man7
systemd-stub(7)
systemd-measure(1)
kernel-install(8)
UAPI.5 Unified Kernel Image specification
UAPI Boot Loader Specification (Type #1 text entries vs Type #2 UKIs)
Shim SBAT documentation
Debian package metadata: systemd-ukify 257.x (apt-cache show systemd-ukify)

Stop Fighting GRUB Menus: Practical systemd-boot with bootctl on Linux

Lyra — Sat, 19 Sep 2026 05:03:18 +0000

Stop Fighting GRUB Menus: Practical systemd-boot with bootctl on Linux

If your boot story still starts with a hand-edited GRUB config, a mystery ESP path, and “I hope the new kernel shows up,” it is time to switch mental models.

systemd-boot (sd-boot) is a small UEFI boot manager. It does not try to be a full OS. It reads standard Boot Loader Specification entries from the ESP (and optional XBOOTLDR partition), shows a menu, and hands control to a kernel or UKI. Day-to-day management is done with bootctl from the running OS—not by grepping shell scripts under /etc/grub.d.

This guide is operational: install, inspect, set defaults, wire kernel packages through kernel-install, tune loader.conf, enable early entropy, and use boot counting so a bad kernel does not trap you.

Scope note: This is about the boot manager and entry lifecycle. It is not a Secure Boot key enrollment guide (use sbctl / firmware tools for that), not A/B image update policy (systemd-sysupdate), and not userspace-only restart (systemctl soft-reboot).

What you need

UEFI firmware (systemd-boot is UEFI-only)
An ESP (GPT type C12A7328-F81F-11D2-BA4B-00A0C93EC93B), typically VFAT
Optional but useful: an Extended Boot Loader partition (XBOOTLDR, GPT type BC13C2FF-59E6-4262-A352-B275FD6F7172) on the same disk
Packages (Debian/Ubuntu naming):
- systemd-boot — integration/services
- systemd-boot-efi — EFI binaries
- systemd-boot-tools — bootctl and related tools

# Debian/Ubuntu-style
sudo apt install systemd-boot systemd-boot-efi systemd-boot-tools

On Fedora/RHEL-family systems the bits usually ship closer to the main systemd/systemd-boot packages; the commands below are the same.

Mental model: ESP, $BOOT, Type #1 vs Type #2

The UAPI Boot Loader Specification defines two entry styles:

Type	Where	What it is
#1	`$BOOT/loader/entries/*.conf`	Text snippets pointing at `linux` + `initrd` (or `efi` / `uki`)
#2	`$BOOT/EFI/Linux/*.efi` (and ESP)	Unified Kernel Images: one PE binary with stub + kernel + initrd + metadata

$BOOT is:

XBOOTLDR if it exists
otherwise the ESP

Recommended mounts (from the BLS):

Mount $BOOT at /boot
If ESP and XBOOTLDR are both present, mount the ESP at /efi
Avoid nesting ESP under /boot/efi when you can—it complicates automount setups

bootctl discovers these paths:

bootctl status
bootctl --print-esp-path
bootctl --print-boot-path   # XBOOTLDR if present, else ESP

kernel-install uses the same discovery order for installing kernels into $BOOT.

Inspect before you touch anything

bootctl status
bootctl list
bootctl is-installed

Useful things status reports:

Firmware / Secure Boot state (as the loader sees it)
Which loader booted you
ESP / boot partition paths
Current default / oneshot / selected entry IDs
Feature flags the loader advertised via EFI variables

list shows Boot Loader Spec entries plus other discovered options (Windows, firmware setup, EFI shell when present).

Install or update systemd-boot on the ESP

# First install: copy sd-boot into the ESP and register it with firmware
sudo bootctl install

# Later: refresh installed copies when package binaries are newer
sudo bootctl update

# Confirm
bootctl is-installed

What install does (from bootctl(1)):

Installs the systemd-boot EFI binary into the ESP (including the removable-path fallback EFI/BOOT/BOOT*.EFI)
Adds the loader to the firmware boot order (unless you pass --no-variables)

Signed-file note: if a *.efi.signed sibling exists, install/update prefer it—useful when your distro ships pre-signed bootloader binaries for Secure Boot.

Graceful hosts (images, chroots, odd firmware):

sudo bootctl update --graceful

That ignores some failure modes (missing ESP write, foreign loader already present) instead of hard-failing.

To remove systemd-boot from the ESP and firmware list:

sudo bootctl remove

Configure the menu: loader.conf

Create or edit $ESP/loader/loader.conf (often /efi/loader/loader.conf or /boot/loader/loader.conf depending on layout):

# /efi/loader/loader.conf
timeout 5
default @saved
editor no
console-mode keep
random-seed-mode with-system-token
auto-firmware yes

Key knobs from loader.conf(5):

Setting	Practical meaning
`timeout`	Seconds before default boots. `0` / `menu-hidden` = no menu unless you hold a key. `menu-force` = always show, no timeout
`default`	Glob for default entry, or `@saved` to remember last choice in an EFI variable
`editor`	Allow editing kernel cmdline at the menu. Disable on untrusted physical access
`console-mode`	`keep` (default), `auto`, `max`, or a numeric mode
`random-seed-mode`	`off`, `with-system-token` (default), or `always`
`auto-entries`	Show/hide auto-discovered foreign loaders
`auto-firmware`	Show “Reboot into firmware” entry

You can also change timeout/default at runtime without editing files:

# Persistent default (EFI variable)
sudo bootctl set-default '<entry-id-or-glob>'

# Next boot only
sudo bootctl set-oneshot '<entry-id-or-glob>'

# Menu timeout
sudo bootctl set-timeout 5
sudo bootctl set-timeout-oneshot 30

Special IDs resolved by bootctl: @default, @oneshot, @current, @saved.

From a running system you can also ask systemd to reboot into a specific entry or force the menu once:

systemctl reboot --boot-loader-entry='<entry-id>'
systemctl reboot --boot-loader-menu=5
systemctl reboot --firmware-setup

Those depend on the Boot Loader Interface that systemd-boot implements.

Type #1 entries you can read and reason about

A minimal BLS Type #1 snippet looks like this (paths relative to the filesystem that holds the snippet):

# $BOOT/loader/entries/6a9857a393724b7a981ebb5b8495b9ea-6.12.0-amd64.conf
title      Debian GNU/Linux
sort-key   debian
machine-id 6a9857a393724b7a981ebb5b8495b9ea
version    6.12.0-amd64
architecture x64
options    root=UUID=6d3376e4-fc93-4509-95ec-a21d68011da2 ro quiet
linux      /6a9857a393724b7a981ebb5b8495b9ea/6.12.0-amd64/linux
initrd     /6a9857a393724b7a981ebb5b8495b9ea/6.12.0-amd64/initrd.img-6.12.0-amd64

Rules that matter in production:

Filenames are identifiers, not UI titles (restricted charset: alnum, +, -, _, .)
Prefer including entry-token/machine-id + kernel version in the filename to avoid multi-boot clashes
title comes from PRETTY_NAME when kernel-install generates the file
Multiple initrd / options lines are allowed and are combined in order
On EFI, Linux images should be EFI PE stubs (CONFIG_EFI_STUB)

You rarely hand-write these on a package-managed host. You let kernel-install do it.

Let kernel-install own kernel placement

# See what the host will do for the running kernel
kernel-install inspect

# Install current or explicit kernel
sudo kernel-install add "$(uname -r)" /boot/vmlinuz-$(uname -r) /boot/initrd.img-$(uname -r)

# Or every kernel under /usr/lib/modules (where layout supports it)
sudo kernel-install add-all

# Remove an old version cleanly
sudo kernel-install remove 6.11.0-amd64

Layout selection (install.conf / env):

layout=bls — Type #1 under $BOOT/loader/entries/ + files under $BOOT/<entry-token>/<version>/
layout=uki — Type #2 copy into $BOOT/EFI/Linux/<entry-token>-<version>.efi
layout=auto — UKI if the image is a UKI; else BLS if entries already look BLS-like

Entry token selection (bootctl install --entry-token= / kernel-install --entry-token= / /etc/kernel/entry-token):

machine-id — best for multiple parallel installs of the same OS on one disk
os-id / os-image-id — stable names, but collide if two installs share the same ID
auto — read /etc/kernel/entry-token if present, else fall through machine-id → IMAGE_ID → ID

Inspect discovery:

kernel-install list
bootctl list
ls "$(bootctl -x)/loader/entries"

Type #2 UKIs (when you want one signed PE file)

Unified Kernel Images live as:

$BOOT/EFI/Linux/<name>.efi

bootctl can classify them:

bootctl kernel-identify /path/to/vmlinuz.efi
bootctl kernel-inspect  /path/to/vmlinuz.efi

Possible identify results: uki, addon, pe, unknown.

Building UKIs is usually ukify + (optionally) systemd-measure for PCR policies—that is a full article on its own. For day-2 ops with systemd-boot you mainly need:

UKIs land in EFI/Linux/
bootctl list sees them
Secure Boot signatures match your enrolled keys (again: sbctl territory)

Early boot entropy: random-seed

systemd-boot can pass a solid random seed into the OS very early by combining:

/loader/random-seed on the ESP
a persistent LoaderSystemToken EFI variable

Initialize both:

sudo bootctl random-seed

Default random-seed-mode with-system-token only credits the ESP seed when the system token exists—this avoids identical seeds when the same disk image is cloned to many machines without unique tokens.

If the random-seed file is marked immutable (read-only attribute), systemd-boot neither updates nor uses it—by design, because a never-updated seed would be identical every boot.

Boot counting: automatic fallback after a bad kernel

systemd-boot supports boot counting: entry filenames encode tries-left / tries-done. On failure to reach a “good” boot, the loader prefers older entries.

Userspace side:

systemd-bless-boot-generator detects counting is active
systemd-bless-boot.service runs once the boot is considered successful (boot-complete.target path)
It renames the entry to drop the counters (“bless” = permanently good)

Manual checks:

/usr/lib/systemd/systemd-bless-boot status
# good | bad | indeterminate | clean | dirty

You can force:

sudo /usr/lib/systemd/systemd-bless-boot good
sudo /usr/lib/systemd/systemd-bless-boot bad

Use bad when you know the current entry is toxic and you want it skipped next time without burning remaining tries.

This is the practical answer to “new kernel panic-loops and I have no serial console on site.”

Housekeeping: unlink and cleanup

# Remove one entry and unreferenced payloads
sudo bootctl unlink '<entry-id-or-glob>'

# Dry run first
sudo bootctl --dry-run unlink '*-6.11.*'

# Remove orphaned files for this entry token that no entry references
sudo bootctl cleanup
sudo bootctl --dry-run cleanup

Prefer kernel-install remove for package-shaped kernels so plugins (depmod, loaderentry, uki-copy) stay consistent. Use bootctl unlink/cleanup for orphans and one-off images.

Multi-boot and foreign OS entries

Out of the box, systemd-boot can surface:

Microsoft Windows Boot Manager
macOS boot manager
EFI shell
Reboot into firmware setup

Toggle discovery with auto-entries / auto-firmware in loader.conf. For “reboot into Windows tonight”:

bootctl list   # find the Windows entry id
sudo bootctl set-oneshot auto-windows
# or the concrete id shown by list
systemctl reboot

A sane baseline checklist

ESP mounted where bootctl --print-esp-path expects it; XBOOTLDR at /boot if you use one.
bootctl install once; bootctl update after systemd-boot package upgrades.
loader.conf: short timeout, editor no on exposed machines, default glob or @saved.
kernel-install layout matches how you ship kernels (bls vs uki).
bootctl random-seed on first provisioning of bare metal (and unique per machine).
Boot counting + bless-boot enabled if you want automatic bad-kernel fallback.
Verify after every kernel install:

bootctl list
bootctl status
kernel-install inspect

Secure Boot: sign systemd-boot and kernels/UKIs with your enrolled keys—do not confuse “loader installed” with “firmware will execute it.”

Rollback / coexistence notes

Switching to systemd-boot does not require wiping GRUB in the same second. Firmware boot order decides who runs. Keep a known-good GRUB entry until bootctl list and a test reboot look right.
Switching away: bootctl remove clears systemd-boot copies and firmware entries it owns; restore your previous loader deliberately.
Soft-reboot and sysupdate operate after or around the bootloader; they do not replace it.
Portable services / sysext extend the running OS; they are not boot entries.

References

GRUB still has a place on legacy BIOS and some exotic setups. On ordinary UEFI servers and workstations, systemd-boot + Boot Loader Spec entries + bootctl is the smaller, auditable path: plain-text or UKI entries on a known partition, defaults in EFI variables, and a userspace tool that tells you the truth before you reboot.

Stop Leaning on setuid sudo Alone: Practical run0 Privilege Elevation on Linux

Lyra — Fri, 18 Sep 2026 05:03:41 +0000

Stop Leaning on setuid sudo Alone: Practical run0 Privilege Elevation on Linux

sudo still works. It also still depends on a classic pattern: a setuid helper, caller environment inheritance (unless tightly filtered), and a policy language that lives outside the service manager.

systemd 256+ ships a different elevation tool: run0.

run0 is not a rebranded sudoers parser. It is a multi-call alias of systemd-run that starts your command as a fresh transient service, authenticates through polkit, allocates an independent pseudo-TTY when you are on a terminal, and never relies on setuid/setgid bits to gain power. That makes it especially interesting on hardened hosts where NoNewPrivileges= is on the table, and on fleets that already treat systemd as the source of truth for process isolation.

This guide is written against the run0(1) / systemd-run(1) manuals and a live systemd 257 install (Debian trixie). Newer releases add more knobs; stick to the man page on your box when in doubt.

What problem it solves

Operator elevation usually fails in one of these ways:

The elevated process still carries too much of the caller’s environment and session assumptions.
Authentication and the privileged program share the same TTY in ways that are hard to reason about.
setuid helpers stop working (or become undesirable) when the system manager enforces NoNewPrivileges=yes.
One-off root work is invisible to cgroup accounting, journal unit filters, and slice placement.

run0 attacks those points directly:

No credential inheritance from the caller into the elevated command — the service manager forks a clean service context.
polkit authentication, with the prompt isolated from the terminal when possible.
Independent PTY for the invoked command (default when stdio is a TTY).
No SetUID/SetGID implementation path.
Sessions go through the systemd-run0 PAM stack.

If you already wrote sudoers.d policy, keep it for command allowlists that polkit does not express well. If you already use plain systemd-run to sandbox one-offs, keep that for non-interactive isolation. run0 sits in the middle: interactive (or scripted) privilege change that looks like sudo, but executes like a managed unit.

Requirements

systemd ≥ 256 for run0 itself (--pty / --pipe / shell prompt prefix land in 257 on the manuals used here).
Working polkit (polkitd) for non-root callers.
Permission to manage units — by default that is polkit action org.freedesktop.systemd1.manage-units.

Check what you have:

systemctl --version | head -n1
command -v run0
ls -l "$(command -v run0)"
run0 --version | head -n1

On a typical install you should see something like:

systemd 257 (257.x)
/usr/bin/run0 -> systemd-run

Confirm polkit is alive:

systemctl is-active polkit
pkaction --action-id org.freedesktop.systemd1.manage-units

On Debian/Ubuntu-style defaults, that action is:

allow_any / allow_inactive: auth_admin
allow_active (active local session): auth_admin_keep

And the stock admin identity rule often nominates unix-group:sudo as the administrative group. That does not mean run0 reads /etc/sudoers. It means polkit and sudo frequently share the same human admin group name.

Mental model

you (uid=1000)
   │  run0 [options] command...
   ▼
polkit: org.freedesktop.systemd1.manage-units
   │  auth_admin / auth_admin_keep
   ▼
PID 1 starts a transient .service (often under user.slice for run0)
   │  fresh execution context
   │  optional independent PTY
   │  PAM stack: systemd-run0
   ▼
command runs as root (default) or --user=/--group=

Important implementation detail from the manual: run0 is a symbolic link to systemd-run. Invoking the binary as run0 selects the elevation personality; invoking it as systemd-run selects the general transient-unit tool.

1) Everyday elevation

Interactive root shell (no command → shell):

run0

One-shot root command:

run0 id
run0 cat /etc/shadow | head -n1   # careful: still privileged output
run0 systemctl restart chrony

Run a specific shell explicitly:

run0 --setenv=SHELL=/bin/bash
# or
run0 /bin/bash -l

Notes that surprise sudo veterans:

With no command, the shell defaults to the originating user’s shell, not the target user’s shell (local case).
With --machine=, the fallback shell is /bin/sh.
Unlike sudo -i folklore, run0 always uses login-shell semantics for shells it starts (manual wording on current docs: always login shell semantics regardless of -i-style habits from sudo).

2) Become another user, not only root

# service account shell / command
run0 --user=www-data --group=www-data id
run0 -u postgres -g postgres --chdir=/var/lib/postgresql psql --version

# interactive as that user
run0 -u deploy

Working directory rules:

Root elevation: default cwd is the client’s current directory.
Non-root target user: default cwd is that user’s home directory.
Override with -D / --chdir=:

run0 -D /etc/nginx nginx -t
run0 -u backup -D /var/backups /usr/local/bin/backup-now

3) See the unit, not just the exit code

Because elevation is a transient service, you can observe it like any other unit.

# In one terminal, start a longer session:
run0 --unit=admin-shell.service

# Elsewhere:
systemctl status admin-shell.service --no-pager
systemctl show admin-shell.service -p MainPID -p User -p Slice -p ControlGroup
journalctl -u admin-shell.service -n 50 --no-pager

Useful properties to pass through --property= (same assignment syntax as systemctl set-property):

# Hard ceiling example for a risky maintenance command
run0 --property=MemoryMax=512M --property=TasksMax=100 \
  --description="compress logs" \
  tar -C /var/log -czf /root/logs.tgz .

# CPU niceness without a full policy rewrite
run0 --nice=10 updatedb

Place the session in a slice:

run0 --slice=system-admin.slice bash
# or inherit/nest relative to the caller’s slice
run0 --slice-inherit --slice=maint bash

Default slice for run0 is user.slice (per run0(1)), which is worth remembering if you expected system.slice from bare systemd-run --system habits.

4) TTY vs pipe mode

From systemd 257:

Mode	Flag	When to use
Pseudo-TTY	`--pty`	Interactive programs, shells, anything that wants a terminal
Pass-through stdio	`--pipe`	Pipelines and scripts where FDs should flow through
Auto	(default)	If stdin/stdout/stderr are all TTYs → PTY; otherwise → pipe

# Force pipeline-friendly behavior
run0 --pipe sh -c 'gzip -c </var/log/syslog' > /tmp/syslog.gz

# Force a PTY even if you are unsure about detection
run0 --pty htop

Security note from the manual: TTY isolation is a core feature. Do not casually hand raw terminal control to untrusted programs when overriding defaults.

5) Environment: what you get and what you set

run0 inherits the system manager environment, not your full interactive profile dump. On top of that it sets:

Variable	Meaning
`$TERM`	Copied from caller (override with `--setenv`)
`$SUDO_USER`	Originating username
`$SUDO_UID`	Originating UID
`$SUDO_GID`	Originating primary GID
`$SHELL_PROMPT_PREFIX`	Default superhero emoji prefix when supported (257+)

Pass more with repeated --setenv:

run0 --setenv=EDITOR=vim --setenv=LANG=C.UTF-8 visudo -c
run0 --setenv=HTTP_PROXY   # value taken from caller env when =value omitted

Prompt cosmetics:

run0 --shell-prompt-prefix='(root) '
# or disable
run0 --shell-prompt-prefix= bash
# or via env for defaults
SYSTEMD_RUN_SHELL_PROMPT_PREFIX='' run0

Background tint (visual “you are elevated” cue):

# default: reddish as root, yellowish as other UID
run0 --background=44 bash    # blue
run0 --background= bash      # disable tint

6) polkit: where policy actually lives

run0 does not consult sudoers for the elevation decision. The gate is polkit’s unit-management action:

pkaction --verbose --action-id org.freedesktop.systemd1.manage-units

Expect a description like “Manage system services or other units” and defaults requiring administrative authentication.

Debian’s stock rule file commonly includes:

// /usr/share/polkit-1/rules.d/50-default.rules (vendor file — do not edit in place)
polkit.addAdminRule(function(action, subject) {
    return ["unix-group:sudo"];
});

Local overrides belong under /etc/polkit-1/rules.d/ (or the distribution’s documented local path). Example lab-only pattern for a dedicated admin group active on the local console — tighten this for production:

// /etc/polkit-1/rules.d/60-admin-manage-units.rules
// Example only: prefer narrow groups + active-session checks in real fleets.
polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.systemd1.manage-units" &&
        subject.isInGroup("wheel") &&
        subject.local && subject.active) {
        return polkit.Result.YES;
    }
});

After changing rules, reload/restart polkit according to your distro (systemctl restart polkit is the blunt approach; some setups pick up rules automatically).

--no-ask-password skips interactive auth prompts. That is appropriate for already-authorized automation contexts, not as a way to bypass policy.

7) Why this pairs with NoNewPrivileges=

From systemd-system.conf(5):

# /etc/systemd/system.conf.d/10-nnp.conf  # powerful; understand the blast radius
[Manager]
NoNewPrivileges=yes

When true, PID 1 and its children cannot gain new privileges through execve(2) via setuid/setgid bits or file capabilities. Classic sudo is exactly that class of helper. run0 is documented as the elevation approach that still works in environments where setuid support is unavailable, because the service manager starts the privileged unit instead of a setuid binary flipping credentials in-process.

Do not flip NoNewPrivileges= globally on a general-purpose distro without a test plan — lots of legacy tooling still assumes setuid. Use it where you intentionally build a setuid-free image.

8) Containers and machines

# Elevate inside a local machine/container known to machined
run0 --machine=webtest systemctl status nginx --no-pager
run0 --machine=webtest

This is the same --machine= plumbing family as systemd-run / machinectl, not SSH.

9) Practical operator cookbook

Drop into root for a change window

run0 --unit=change-window.service --description="2026-09-18 change window"
# work...
# exit
journalctl -u change-window.service --since "10 min ago" --no-pager

Restart a unit without keeping a root shell

run0 systemctl try-restart caddy.service

Edit a root-owned file with your $EDITOR semantics

run0 --setenv=EDITOR --setenv=TERM --chdir=/etc/ssh \
  sh -c '"$EDITOR" sshd_config'
sshd -t   # still verify as root if needed: run0 sshd -t

Bounded maintenance job

run0 \
  --unit=apt-maintenance.service \
  --property=Nice=10 \
  --property=MemoryHigh=1G \
  --property=MemoryMax=2G \
  --property=TasksMax=300 \
  --description="apt update && upgrade" \
  bash -lc 'apt-get update && apt-get -y upgrade'

Verify identity and origin fields inside the session

run0 bash -lc 'id; printf "SUDO_USER=%s SUDO_UID=%s SUDO_GID=%s\n" \
  "$SUDO_USER" "$SUDO_UID" "$SUDO_GID"'

You should see root (or the target user) for id, and your original account in the SUDO_* fields.

10) Failure modes and debugging

Symptom	Likely cause	What to check
Immediate auth failure	polkit denied / no admin group	`pkaction`, group membership, active session
Works on console, fails over SSH	agent / session class differences	run on a real active session; install a polkit agent if needed
Command not found inside run0	clean service `PATH`	use absolute paths or `--setenv=PATH=`
Interactive TUI garbled	pipe mode instead of PTY	pass `--pty`
Unit left failed	command non-zero exit	`systemctl reset-failed` / `journalctl -u …`
“Looks like sudo but ignore sudoers”	expected	policy is polkit + unit properties, not `sudoers`

Debug trail:

journalctl -u polkit.service -n 100 --no-pager
journalctl -t run0 -n 50 --no-pager  # if tagged on your build
systemctl list-units 'run-*.service' --all --no-pager

11) What run0 is not

Not a full sudoers replacement for per-command argument filtering, NOPASSWD host-by-host matrices, or existing enterprise sudo policy distributions.
Not the same article as “sandbox any one-off with systemd-run” — that tool is broader (timers, scopes, user manager, etc.) and is not specifically the elevation UX.
Not pkexec with a different name — both use polkit, but run0 is explicitly a systemd transient-service elevation path with unit properties, slices, and journal integration.
Not a substitute for service hardening (ProtectSystem=, CapabilityBoundingSet=, Landlock, seccomp). Elevation gets you in; hardening still belongs on long-running units.

Migration cheat sheet

Old habit	run0-shaped habit
`sudo -s`	`run0`
`sudo -u postgres -i`	`run0 -u postgres`
`sudo -E cmd`	selective `run0 --setenv=NAME` (prefer explicit)
`sudo nice -n 10 cmd`	`run0 --nice=10 cmd`
`sudo systemd-run …`	often just `run0 --property=…`
audit via sudo logs only	also `journalctl -u <unit>` + polkit events

Keep sudo installed while you migrate muscle memory. Many images will ship both for years.

Rollout suggestion

Confirm systemd ≥ 256 and polkit healthy.
Practice run0 id and run0 shells on a lab user in the admin group.
Teach absolute paths and --setenv instead of blanket env preservation.
Move break-glass interactive work to run0 --unit=… so sessions are nameable in the journal.
Only then consider narrowing sudoers or building setuid-free images that rely on service-manager elevation.

References

run0(1) — privilege elevation; multi-call alias of systemd-run (systemd 256+)
systemd-run(1) — transient services/scopes and shared options
systemd-system.conf(5) — NoNewPrivileges= manager setting
polkit(8) / pkaction(1) — authorization actions and admin rules
Debian policy file org.freedesktop.systemd1.policy — manage-units defaults (auth_admin / auth_admin_keep)
Vendor rules such as /usr/share/polkit-1/rules.d/50-default.rules — admin group mapping (often unix-group:sudo)
Local verification base for this article: systemd 257.13 on Debian trixie, run0 -> systemd-run, polkitd present

sudo taught a generation of admins to borrow root carefully. run0 keeps the careful part, drops the setuid helper, and makes the elevated session a first-class systemd citizen. If your hosts already boot, isolate, and heal through units, elevating through units is the consistent next step.

Stop Patching the Live Root: Practical A/B OS Updates with systemd-sysupdate

Lyra — Sun, 13 Sep 2026 05:03:44 +0000

Stop Patching the Live Root: Practical A/B OS Updates with systemd-sysupdate

Package managers are great at evolving a mutable system in place. They are a poorer fit when you want the opposite model: ship a whole, known-good OS image, keep the currently running root untouched while the next version lands beside it, then flip roles on reboot.

That is the job of systemd-sysupdate.

It is the image-update tool that pairs naturally with GPT discoverable partitions, dm-verity roots, UKIs on the ESP, systemd-repart spare slots, and portable/container disk images. You describe transfers (source → target). The tool enumerates versions, verifies downloads, writes into free A/B (or A/B/C…) slots, and only then finalizes names/labels so a reboot can pick up the new set.

This article is a practical operator guide based on the current systemd-sysupdate(8) and sysupdate.d(5) manuals. The feature is still marked experimental upstream—pin versions, read the man pages for your release, and expect setting names to evolve—but the core model has been stable enough since systemd 251 to use deliberately.

What problem it solves

Typical in-place upgrade pain:

Mid-upgrade reboot leaves a half-applied userspace.
Kernel, initrd, rootfs, and verity data can drift out of lockstep.
“Rollback” means hoping the package manager’s undo path still works.
Mutable / makes fleet drift inevitable.

systemd-sysupdate flips the model:

Keep multiple concurrent versions of each resource (files, directories/subvolumes, or GPT partitions).
Download the next version into empty slots while the current version keeps running.
Bind several resources together with a shared version id (@v) so root + verity + UKI move as one logical release.
Activate by booting the new entry point (usually the new UKI / boot loader entry), not by rewriting the live root underneath PID 1.

It can update:

the host OS online from inside itself
offline disk images via --image=
container images, portable service images, and other file/dir trees the same way

Mental model: transfers + versions

Each resource you update is one transfer file under:

/etc/sysupdate.d/
/run/sysupdate.d/
/usr/local/lib/sysupdate.d/
/usr/lib/sysupdate.d/

Current docs name these *.transfer (older trees and some man page editions still show *.conf—check man 5 sysupdate.d on your box).

Every transfer has three sections:

[Transfer] — policy (min version, protect running version, verify, optional features)
[Source] — where versions come from
[Target] — where versions are installed

Sources and targets are typed:

Source type	Typical target	Auth / integrity
`url-file`	`regular-file` or `partition`	SHA256SUMS + optional SHA256SUMS.gpg
`url-tar`	`directory` or `subvolume`	same
`regular-file`	`regular-file` or `partition`	no remote auth
`tar`	`directory` / `subvolume`	local only
`directory` / `subvolume`	`directory` / `subvolume`	local copy

HTTP(S) catalogs are deliberately simple: a GNU sha256sum-format SHA256SUMS next to the payloads, optionally signed as SHA256SUMS.gpg. Payloads are always checked against the hashes. Verify= (default yes) controls whether the manifest signature is validated against the import keyring (/usr/lib/systemd/import-pubring.pgp and /etc/systemd/import-pubring.pgp, with older .gpg names still documented).

Version extraction uses match patterns. The mandatory wildcard is @v. Useful extras include partition UUID/flags (@u, @f, @a, @g, @r), file mode/size/time, boot-assessment counters (@d / @l), and more.

A complete OS update: root + verity + UKI

The manuals’ canonical example is still the clearest. Updating “foobarOS” to version 47 means three coordinated transfers of the same @v:

foobarOS_47.root.xz → empty GPT root partition (x86-64 type UUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709)
foobarOS_47.verity.xz → empty GPT root verity partition (2c7357ed-ebd2-46d9-aec1-23d437ec2bf5)
foobarOS_47.efi → $BOOT/EFI/Linux/foobarOS_47.efi (Boot Loader Spec Type #2 UKI)

Empty partition slots are marked with GPT label _empty. After a successful write, labels become versioned names like foobarOS_47 / foobarOS_47_verity.

Critical ordering rule: transfers run in alphabetical filename order. Download+write happens first for all transfers; final rename/relabel happens second, still in that order, with disk sync points. Put the boot entry point last (for example 90-uki.transfer) so a crash never leaves a bootable UKI pointing at incomplete root/verity backing.

Example transfer set

# /etc/sysupdate.d/50-root.transfer
[Transfer]
ProtectVersion=%A
Verify=yes

[Source]
Type=url-file
Path=https://download.example.com/foobarOS
MatchPattern=foobarOS_@v.root.xz

[Target]
Type=partition
Path=auto
MatchPattern=foobarOS_@v
# Prefer DPS type names when your systemd supports them; otherwise use the GPT UUID.
TypeUUID=root-x86-64
InstancesMax=2
ReadOnly=yes

# /etc/sysupdate.d/60-verity.transfer
[Transfer]
ProtectVersion=%A
Verify=yes

[Source]
Type=url-file
Path=https://download.example.com/foobarOS
MatchPattern=foobarOS_@v.verity.xz

[Target]
Type=partition
Path=auto
MatchPattern=foobarOS_@v_verity
TypeUUID=root-x86-64-verity
InstancesMax=2
ReadOnly=yes

# /etc/sysupdate.d/90-uki.transfer
[Transfer]
ProtectVersion=%A
Verify=yes

[Source]
Type=url-file
Path=https://download.example.com/foobarOS
MatchPattern=foobarOS_@v.efi

[Target]
Type=regular-file
Path=/EFI/Linux
PathRelativeTo=boot
MatchPattern=foobarOS_@v.efi
Mode=0644
InstancesMax=2

Notes that save real debugging time:

ProtectVersion=%A expands from IMAGE_VERSION= in /etc/os-release and refuses to delete/overwrite the currently booted image version while making room for the next one. %w (VERSION_ID=) and %B (BUILD_ID=) are related alternatives.
Path=auto on partition targets means “the block device that backs the booted root.”
PathRelativeTo=boot (also esp, xbootldr, root, …) anchors file targets relative to the right firmware/boot volume instead of hard-coding mount points.
InstancesMax= should usually match across the whole transfer set. With two root slots you effectively keep current + next.
Partition targets must already exist. systemd-sysupdate will not create GPT entries. Pair this with systemd-repart so first boot (or image build) materializes enough _empty slots of the right types.

Server-side layout

On the download host:

https://download.example.com/foobarOS/SHA256SUMS
https://download.example.com/foobarOS/SHA256SUMS.gpg
https://download.example.com/foobarOS/foobarOS_47.root.xz
https://download.example.com/foobarOS/foobarOS_47.verity.xz
https://download.example.com/foobarOS/foobarOS_47.efi
https://download.example.com/foobarOS/foobarOS_48.root.xz
...

Generate the manifest with GNU sha256sum (binary mode is recommended in the man page even on Linux):

cd /srv/foobarOS
sha256sum --binary foobarOS_*.root.xz foobarOS_*.verity.xz foobarOS_*.efi > SHA256SUMS
gpg --detach-sign --armor -o SHA256SUMS.gpg SHA256SUMS

Optional freshness brake: include a special BEST-BEFORE-YYYY-MM-DD entry in SHA256SUMS. Past that date, the listing is rejected.

Day-2 commands operators actually use

Install the tool first. On Debian/Ubuntu it typically ships in the systemd-container package; some other distros ship it with core systemd. Confirm:

command -v systemd-sysupdate
man 8 systemd-sysupdate
man 5 sysupdate.d

Enumerate and inspect

# Default command is list
systemd-sysupdate list
systemd-sysupdate list 48          # detail one version + required transfers
systemd-sysupdate --json=pretty list

See if anything newer exists

if ver=$(systemd-sysupdate check-new); then
  echo "candidate: $ver"
else
  echo "already current or no catalog"
fi

Exit status 0 means a newer installable version exists; its id is printed on stdout.

Install / stage

# Newest available
systemd-sysupdate update

# Explicit version
systemd-sysupdate update 48

# Download now, install later (newer systemd)
systemd-sysupdate acquire 48
systemd-sysupdate update --offline 48

# Install then reboot immediately (host OS only)
systemd-sysupdate update --reboot

acquire (added in 260 on current docs) separates bandwidth work from the finalization window. Older releases only have update.

Space management

systemd-sysupdate vacuum
systemd-sysupdate --instances-max=3 update

vacuum deletes or empties old instances until InstancesMax= is satisfied. update/acquire already call this implicitly when they need a free slot.

Pending activation

# Newer installed than running IMAGE_VERSION= ?
systemd-sysupdate pending && echo "reboot to activate"

# Reboot only if pending
systemd-sysupdate reboot

pending compares the newest installed version id against IMAGE_VERSION= in /etc/os-release. That is your “update written, not yet booted” signal.

Components (update planes that move independently)

Resources that must always ship together belong in the same sysupdate.d/ directory as multiple transfers.

Resources that may move on different cadences get components:

systemd-sysupdate components
systemd-sysupdate --component=addon list
systemd-sysupdate --component=addon update

That switches the search path to /etc/sysupdate.<name>.d/ (and the /run + /usr/lib twins). Do not split root/verity/UKI across components if they must stay version-locked.

Newer systemd also adds optional features, enable-feature / disable-feature, enable-component / disable-component, and cleanup for orphaned filesystem installs. Use them when your image product actually exposes optional payload sets; ignore them for a minimal A/B OS.

Offline image maintenance

# Apply transfers defined inside a disk image, writing partitions in-image
systemd-sysupdate --image=/var/lib/machines/edge.raw update

# Or point at an explicit definitions directory
systemd-sysupdate --definitions=/srv/transfers.d --image=/srv/appliance.img update

This is especially useful for factory-updating nspawn machines, portable service images, or appliance RAW files from a build host.

Timers: download by day, reboot by night

Do not couple “fetch” and “bounce the fleet” into one unit.

Current unit names (check your release; older trees used systemd-sysupdate.service / .timer without the -update infix):

# Pull updates on a schedule
systemctl enable --now systemd-sysupdate-update.timer

# Optionally reboot later when a newer installed version is pending
systemctl enable --now systemd-sysupdate-reboot.timer

systemctl list-timers 'systemd-sysupdate*'
systemctl status systemd-sysupdate-update.service

Why separate?

Downloads can be frequent and opportunistic.
Reboots need change windows, draining, and monitoring.
pending + reboot timer gives you “already staged, activate overnight” without re-hitting the network.

Some releases also ship systemd-sysupdate-auto-enable.service to auto-enable suggested components/features before each update. Leave it off unless you explicitly want that product behavior.

Partition prep with systemd-repart

Remember: sysupdate fills slots; it does not invent them.

A minimal repart mental model for two root + two verity slots:

# /etc/repart.d/10-root-a.conf
[Partition]
Type=root
Label=_empty
SizeMinBytes=4G
SizeMaxBytes=4G

# /etc/repart.d/11-root-b.conf
[Partition]
Type=root
Label=_empty
SizeMinBytes=4G
SizeMaxBytes=4G

# /etc/repart.d/20-verity-a.conf
[Partition]
Type=root-verity
Label=_empty
SizeMinBytes=128M
SizeMaxBytes=128M

# /etc/repart.d/21-verity-b.conf
[Partition]
Type=root-verity
Label=_empty
SizeMinBytes=128M
SizeMaxBytes=128M

Exact Type= aliases and verity pairing rules come from the UAPI Discoverable Partitions Specification and systemd-repart(8). Factory images often bake these empty slots at build time; devices can also grow them on first boot from free disk space.

Verification checklist after first staging

# 1) Catalog vs installed
systemd-sysupdate list

# 2) os-release identity the tool uses for pending/protect
grep -E '^(IMAGE_ID|IMAGE_VERSION|VERSION_ID)=' /etc/os-release

# 3) Partition labels / empty slots
lsblk -o NAME,PARTTYPENAME,PARTLABEL,SIZE,FSTYPE
# or
sfdisk -d /dev/disk/by-id/… | grep -E 'label:|type='

# 4) UKIs present on $BOOT
bootctl status
ls -l /efi/EFI/Linux 2>/dev/null || ls -l /boot/EFI/Linux

# 5) After update, before reboot
systemd-sysupdate pending; echo exit:$?

If list sees remote versions but update will not move:

GPG verify failing (Verify=yes + missing key in import-pubring)
no free _empty / reclaimable slot (InstancesMax / partition count)
mismatched @v sets (root 48 present remotely but verity 48 missing → incomplete version)
MinVersion= filtering candidates
ProtectVersion= preventing reclamation of the only spare that still holds the running image

Boundaries: when not to use this

Tool	Job
apt/dnf/zypper	Mutable package OS, rich dependency solves, classic servers
ostree / rpm-ostree / bootc	Content-addressed tree deployments with their own client stack
systemd-sysupdate	Whole-file / whole-partition A/B transfers driven by simple HTTPS manifests
systemd-repart	Create/grow GPT layout and factories—not the updater
dm-verity / fs-verity	Integrity of what you already have—not distribution
portable services / sysext	Add software onto a base OS—not replacing the base OS image
soft-reboot	Userspace restart without firmware/kernel; complementary activation tactic, not an image transporter

If you are mostly pinning a few packages on Debian stable, stay with APT. If you are building appliances, edge nodes, kiosks, or verified image fleets where “version 48” means a concrete root+verity+UKI tuple, sysupdate is the systemd-native answer.

Safer rollout pattern

Build images with IMAGE_ID= / IMAGE_VERSION= baked into /etc/os-release.
Publish payloads + SHA256SUMS (+ .gpg) on HTTPS.
Ensure GPT has ≥2 slots per updated partition type (repart).
Deploy matching sysupdate.d transfers; put UKI/entry-point last alphabetically.
systemd-sysupdate list and check-new on a canary.
update on canary → pending → controlled reboot → health checks.
Enable update timer fleet-wide; keep reboot timer tighter or manual.
Keep one known-good UKI and partition set around (InstancesMax=3 helps) until the new version proves itself.
Treat the whole stack as experimental in change control: read the man page for your systemd version before copying flags from the internet.

References

systemd-sysupdate(8) — commands, timers, --image=, components, pending/reboot https://manpages.debian.org/unstable/systemd-container/systemd-sysupdate.8.en.html https://man7.org/linux/man-pages/man8/systemd-sysupdate.8.html
sysupdate.d(5) — transfer files, resource types, match patterns, InstancesMax=, verify/keyring https://manpages.debian.org/unstable/systemd-container/sysupdate.d.5.en.html https://man7.org/linux/man-pages/man5/sysupdate.d.5.html
UAPI Discoverable Partitions Specification (root / root-verity GPT types) https://uapi-group.org/specifications/specs/discoverable_partitions_specification/
UAPI Boot Loader Specification (Type #2 UKI paths under $BOOT/EFI/Linux) https://uapi-group.org/specifications/specs/boot_loader_specification/
systemd-repart(8) — creating empty partition slots for A/B https://manpages.debian.org/unstable/systemd-repart/systemd-repart.8.en.html
os-release(5) — IMAGE_VERSION= used by pending and %A https://man.archlinux.org/man/os-release.5.en
GNU sha256sum(1) — SHA256SUMS manifest format https://man7.org/linux/man-pages/man1/sha256sum.1.html

Bottom line: stop treating OS updates as a long transaction against the live root. Define versioned transfers, give yourself empty slots, verify the catalog, stage beside the running system, and reboot on purpose. That is systemd-sysupdate—simple HTTPS manifests, A/B (or better) slots, and activation you control.

Stop Shipping Full Containers for Host Extensions: Practical systemd Portable Services with portablectl

Lyra — Sat, 12 Sep 2026 05:03:01 +0000

Stop Shipping Full Containers for Host Extensions: Practical systemd Portable Services with portablectl

You already know the two common answers for shipping extra software onto a Linux host:

Full OS containers (systemd-nspawn, LXC) — their own PID 1, their own network stack, their own lifecycle.
App containers (Podman/Docker) — OCI images, separate runtime, often a separate mental model from systemctl.

There is a third option that sits in a useful gap: systemd portable services.

A portable service is just an OS tree (directory, btrfs subvolume, or .raw disk image) that carries:

the service binary and its libraries
one or more systemd unit files
a normal os-release file

You attach that image with portablectl. systemd copies the matching units onto the host, pins them to RootDirectory= / RootImage=, and applies a security profile. From then on the payload is a normal host unit: systemctl start, journald, cgroups, timers, sockets — same tools you already use.

This post is a practical operator guide: build a minimal image, attach it, pick a profile, upgrade with reattach, keep state on the host, and tear it down cleanly.

What portable services are (and are not)

From the Portable Services design doc:

They do not invent a new image format. Directory trees and GPT/raw images already work.
They do not run your app as PID 1. It is a normal service process under host systemd.
They do not fully isolate like Docker by default. The point is host integration with optional lockdown, not a separate world.
They do bundle dependencies and apply stricter default sandboxing via profiles.

Approach	Runs as	Root FS	Typical control plane
Host package / unit	host service	host root	`systemctl`
Portable service	host service	image root (`RootImage=` / `RootDirectory=`)	`portablectl` + `systemctl`
`systemd-nspawn -b`	container PID 1	image root	`machinectl` / nspawn
Podman/Docker	container runtime	OCI layers	`podman` / `docker`

Use portable services when the software should feel like a host extension (agent, exporter, appliance sidecar, “super-privileged container” style workload) while still shipping its own userspace tree.

Prerequisites

systemd 239+ with portable support (portablectl, systemd-portabled.service). On many modern distros this ships with systemd; Arch’s man page documents systemd 261-era behavior.
Root (or polkit) for attach/detach.
Ability to build a Linux userspace tree (debootstrap, dnf --installroot=, or mkosi).

Check the CLI:

command -v portablectl
portablectl --version
systemctl status systemd-portabled.service --no-pager

If portablectl is missing, install/enable the portable bits for your distro (package naming varies; on some systems the binary historically lived under /usr/lib/systemd/ before landing on $PATH).

Image search paths

portablectl list looks in:

/var/lib/portables/ (preferred store for large images)
/etc/portables/
/run/portables/
/usr/local/lib/portables/
/usr/lib/portables/

Recommendation from portablectl(1): put real images under /var/lib/portables/ and only put symlinks in /etc/portables/ or /run/portables/.

Unit name prefix rules (this trips people up)

When you attach foobar_47.11.raw, the default unit prefix is foobar (filename without .raw, truncated at the first _).

Only units whose names match that prefix followed by ., -, or @ are copied. Examples that match:

foobar.service
foobar-agent.service
foobar@.service
foobar.socket / foobar.timer / foobar.path / foobar.target

Units that do not match the prefix are ignored. You can override prefixes on the command line after the image name.

Optional hardening in the image’s os-release:

PORTABLE_PREFIXES=foobar
PORTABLE_SCOPE=system

PORTABLE_PREFIXES= documents and constrains allowed prefixes (especially useful with authenticated/verity images). PORTABLE_SCOPE= can be system, user, or any (default implies system).

Lab: build a minimal directory image

You do not need a full distro. The design doc’s minimal tree is enough for a static binary. Here is a small directory image you can attach without building a GPT disk.

sudo mkdir -p /var/lib/portables
IMG=/var/lib/portables/minagent_1.0.0
sudo rm -rf "$IMG"
sudo mkdir -p \
  "$IMG/usr/bin" \
  "$IMG/usr/lib/systemd/system" \
  "$IMG/usr/lib" \
  "$IMG/etc" \
  "$IMG/proc" "$IMG/sys" "$IMG/dev" "$IMG/run" "$IMG/tmp" "$IMG/var/tmp"

# Placeholder "daemon": write a heartbeat then sleep forever.
# Prefer a real static binary in production; this is a lab stand-in.
sudo tee "$IMG/usr/bin/minagentd" >/dev/null <<'EOF'
#!/bin/sh
echo "minagentd starting on $(hostname) at $(date -Is)"
while true; do
  echo "minagentd heartbeat $(date -Is)"
  sleep 30
done
EOF
sudo chmod 0755 "$IMG/usr/bin/minagentd"

sudo tee "$IMG/usr/lib/systemd/system/minagent.service" >/dev/null <<'EOF'
[Unit]
Description=Minimal portable agent example
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/minagentd
# Writable state lives on the host, not inside a read-only image.
StateDirectory=minagent
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo tee "$IMG/usr/lib/os-release" >/dev/null <<'EOF'
NAME="minagent portable"
ID=minagent
VERSION_ID=1.0.0
PORTABLE_PREFIXES=minagent
PORTABLE_SCOPE=system
EOF

# Required mount points / bind targets (empty is fine; host over-mounts them).
sudo : > "$IMG/etc/resolv.conf"
sudo : > "$IMG/etc/machine-id"

Important reality check for `/bin/sh` labs

A pure static tree is ideal. A #!/bin/sh script needs a shell and dynamic loader inside the image (or a static busybox). For a quick lab on the same distro family, clone a thin root with debootstrap --variant=minbase or copy the interpreter + libs. Production images are usually built with mkosi / installroot so the tree is self-contained.

A debootstrap-shaped lab skeleton:

# Debian/Ubuntu-shaped example — adjust suite/mirror for your host.
sudo debootstrap --variant=minbase bookworm /var/lib/portables/minagent_1.0.0 http://deb.debian.org/debian
# Then install your unit + binary paths as above, and set os-release PORTABLE_* fields.

Or follow the official walkthrough repo that builds a C daemon with mkosi: systemd/portable-walkthrough.

Inspect before you attach

sudo portablectl inspect /var/lib/portables/minagent_1.0.0
sudo portablectl inspect --cat /var/lib/portables/minagent_1.0.0

You should see os-release metadata and the matching unit list (minagent.service).

Attach, enable, start

# Persistent attach (units under /etc/systemd/system.attached/)
sudo portablectl attach --profile=default --enable --now /var/lib/portables/minagent_1.0.0

# Or temporary until reboot:
# sudo portablectl attach --runtime --profile=default --now /var/lib/portables/minagent_1.0.0

What attach does (portablectl(1) / design doc):

Copies matching .service / .socket / .target / .timer / .path units into /etc/systemd/system.attached/ (or /run/... with --runtime).
Writes a drop-in (20-portable.conf) with RootDirectory= or RootImage=, plus Environment=PORTABLE=... and LogExtraFields=PORTABLE=....
Links a profile drop-in (10-profile.conf).
Symlinks the image into the portable search path if needed.
Reloads the manager (unless --no-reload).

Verify:

sudo portablectl is-attached minagent_1.0.0
sudo portablectl list
systemctl status minagent.service --no-pager
systemctl cat minagent.service
ls -la /etc/systemd/system.attached/minagent.service.d/

Expected attachment states include: detached, attached, attached-runtime, enabled, enabled-runtime, running, running-runtime.

Profiles: the real product feature

Profiles are host-local drop-ins under:

/usr/lib/systemd/portable/profile/<name>/service.conf
/etc/systemd/portable/profile/<name>/ for custom profiles

Shipped profiles:

Profile	Intent
`default`	Medium lockdown; journal + D-Bus + IP networking allowed
`nonetwork`	Like default, but `PrivateNetwork=yes` and `IPAddressDeny=any`
`strict`	Tightest stock profile: no caps, AF_UNIX only, no net, low `TasksMax`
`trusted`	Minimal restrictions; effectively full host trust

Select at attach time:

sudo portablectl attach --profile=strict /var/lib/portables/minagent_1.0.0

Excerpt from the stock default profile (systemd v256 tree):

[Service]
MountAPIVFS=yes
BindReadOnlyPaths=/dev/log /run/systemd/journal/socket /run/systemd/journal/stdout
BindReadOnlyPaths=/etc/machine-id
BindReadOnlyPaths=-/etc/resolv.conf
BindReadOnlyPaths=/run/dbus/system_bus_socket
DynamicUser=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictNamespaces=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native
# plus a reduced CapabilityBoundingSet= ...

Excerpt from strict:

[Service]
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX
PrivateNetwork=yes
IPAddressDeny=any
NoNewPrivileges=yes
TasksMax=4
# ... same Protect*/Private* baseline as default ...

Operator rule: the image vendor ships code; the host admin chooses the profile. That split is intentional.

Need host files or sockets? Use normal unit directives in the image unit (or a host drop-in via systemctl edit):

[Service]
BindReadOnlyPaths=/run/myservice.sock
BindPaths=/var/lib/minagent-extra

Writable data with immutable images

All profiles except trusted lean on ProtectSystem=strict. Keep the image read-only and put mutable data on the host with StateDirectory= / CacheDirectory= / LogsDirectory= / RuntimeDirectory= in the unit.

# After the unit has run once under DynamicUser/StateDirectory:
ls -la /var/lib/minagent
journalctl -u minagent.service -n 50 --no-pager

Optional:

sudo portablectl read-only minagent_1.0.0 yes

Upgrade with less downtime: `reattach`

Ship a new version as minagent_1.0.1 (underscore versioning). reattach detaches the old attachment and attaches the new image; with --now it restarts updated units instead of a blunt stop+start cycle, which helps preserve runtime state such as the file descriptor store when applicable.

# Build/copy new tree or .raw named minagent_1.0.1
sudo portablectl reattach --profile=default --now /var/lib/portables/minagent_1.0.1
sudo portablectl is-attached minagent_1.0.1

Partial matching: only the part before the first _ must match for upgrade pairing.

Keep the image path stable while attached. Moving the file breaks RootImage= / RootDirectory= in the generated drop-in.

Extensions (sysext/confext layering)

Since v249, portablectl attach --extension PATH ... can stack OverlayFS layers: a shared base runtime image plus thin app extensions. Extension images need matching extension-release metadata (ID= + SYSEXT_LEVEL=/VERSION_ID= or confext equivalents). Same extensions, same order, are required on detach.

# Conceptual pattern from the design doc:
sudo portablectl attach \
  --extension foobar_0.7.23.raw \
  debian-runtime_11.1.raw \
  foobar

Use this when many agents share one distro runtime and you only want to ship the app layer repeatedly.

Logging fields you can query

Portable attach injects structured journal fields such as:

PORTABLE=
PORTABLE_NAME_AND_VERSION= (from IMAGE_ID/ID + IMAGE_VERSION/VERSION_ID/BUILD_ID)
with extensions: PORTABLE_ROOT=, PORTABLE_EXTENSION=, PORTABLE_*_NAME_AND_VERSION=

journalctl FIELD=PORTABLE=minagent_1.0.0 -n 20 --no-pager
# or filter by unit as usual
journalctl -u minagent.service -f

Instantiation

No special portable API: ship foobar@.service and instantiate after attach.

sudo portablectl attach foobar_0.7.23.raw
sudo systemctl enable --now foobar@edge-a.service
sudo systemctl enable --now foobar@edge-b.service

Detach and clean up

# Stop + disable + detach
sudo portablectl detach --enable --now minagent_1.0.0

# systemd v256+: also remove host state/logs/cache/runtime dirs for the service
sudo portablectl detach --clean --now minagent_1.0.0

# Remove only a search-path symlink/image entry (does not follow and delete the target if it is a symlink)
sudo portablectl remove minagent_1.0.0

Confirm:

sudo portablectl is-attached minagent_1.0.0   # detached
systemctl status minagent.service --no-pager  # not-found / inactive as expected

Copy mode and image policy notes

--copy=copy|symlink|auto|mixed controls whether attached units/profiles/images are copied or symlinked. Raw disk images force copy when symlink is impossible. mixed (v256+) symlinks profile drop-ins but copies units/images.
On attach, systemd can generate an image policy that pins the content discovered at attach time so a later swap cannot silently drop protections such as dm-verity without reinstall. See systemd.image-policy(7).

Operational checklist

Build a self-contained tree or .raw with matching unit prefix + os-release.
portablectl inspect before attach.
Choose profile deliberately (default / nonetwork / strict / custom / trusted).
Prefer StateDirectory= over writable images.
Store images under /var/lib/portables/ and version with _.
Upgrade via reattach --now.
Tear down with detach --now and --clean when you want host state gone.
Treat portable units like any other units for cgroup limits: systemctl set-property minagent.service MemoryMax=512M CPUQuota=50%.

Boundaries (so this does not blur into other tools)

Not a replacement for Podman Quadlet app containers when you need OCI registries, rootless stacks, and Kubernetes-shaped workflows.
Not a replacement for systemd-nspawn -b when you need a full guest OS with its own init and machine lifecycle.
Not a substitute for whole-disk integrity (dm-verity) or per-file authenticity (fs-verity); those compose with portable images when you ship .raw + verity.
Not soft-reboot / image A/B host update logic; portable services extend a live host, they do not replace host OS update strategy.

Sources and references

Portable Services Introduction (systemd.io)
portablectl(1) — Arch man pages
systemd-portabled.service(8)
Walkthrough for Portable Services — Lennart Poettering
systemd/portable-walkthrough example repo
Stock profiles in systemd source: default/service.conf, strict/service.conf, nonetwork/service.conf
Related: mkosi, Discoverable Partitions Specification

Wrap-up

Portable services are the “missing middle” for Linux operators who want bundled dependencies and profile-based sandboxing without leaving the systemd control plane. Build an image, portablectl attach --profile=... --enable --now, manage it with systemctl and journald, upgrade with reattach, and detach without scattering random unit files by hand.

If your next component is “basically a host service, but I refuse to pollute the host rootfs with its library tree,” try a portable image before you reach for a full guest OS container.

DEV Community: Lyra

Stop Putting Secrets in Environment Variables: Practical systemd-creds on Linux

Why not just use an env var?

Prerequisites

The five unit settings (and when to use each)

Lab 1 — Plain LoadCredential with a transient unit

Lab 2 — Encrypt with the host key

Lab 3 — LoadCredentialEncrypted in a real unit drop-in

Lab 4 — Paste ciphertext into the unit with SetCredentialEncrypted

Lab 5 — Wire apps that do not know $CREDENTIALS_DIRECTORY yet

Lab 6 — Propagate system credentials with ImportCredential

User services (systemd 256+)

Operational checklist

Boundaries (what this article is not)

Cleanup

References

Stop Guessing systemd Service APIs: Practical varlinkctl on Linux

What Varlink is (and is not)

Prerequisites

Address forms you will actually use

Lab 1 — Inventory a live service (resolved)

Lab 2 — Call a method with JSON arguments

Lab 3 — Multi-reply methods, collection, and oneway

Lab 4 — Exec targets and helper binaries

Lab 5 — Remote calls over SSH

Lab 6 — Registry and socket discovery (260+)

Lab 7 — Serve a sandboxed stdio tool as Varlink (261+)

Lab 8 — Validate IDL before you ship an interface

Operator cheat sheet

Common failure modes

How this fits nearby tooling

References

Stop Losing User Services on Logout: Practical loginctl Linger on Linux

Stop Losing User Services on Logout: Practical loginctl Linger on Linux

What logind owns (and what it does not)

The mental model in one tree

Prerequisites

Lab 1 — Inventory sessions, users, and seats

Lab 2 — Prove the “logout kills user services” failure mode

Lab 3 — Enable linger the supported way

Lab 4 — Boot-time user manager without an interactive login

Lab 5 — KillUserProcesses, UserStopDelaySec, and tmux myths

Lab 6 — Session hygiene: lock, terminate, kill

Lab 7 — Cap resources for lingering users

A minimal “rootless service that survives logout” recipe

Common failure modes

How this fits nearby systemd pieces

References

Stop wget-and-Hope Image Downloads: Practical importctl on Linux

Stop wget-and-Hope Image Downloads: Practical importctl on Linux

What importctl owns (and what it does not)

Prerequisites

Lab 1 — List what you already have

Lab 2 — Pull a verified tar rootfs into machines/

Lab 3 — Pull a raw disk image, set root password offline, boot it

Lab 4 — Import local tar/raw/directory trees (no network verification)

Lab 5 — Class-aware pulls for portable / sysext / confext

Lab 6 — Export images for backup or handoff

Lab 7 — OCI pull (systemd 260+)

Transfer hygiene and remote operation

A minimal “pull → boot” recipe you can script

Common failure modes

How this fits the rest of the systemd image stack

References

Stop losetup Guesswork: Practical systemd-dissect for Discoverable Disk Images on Linux

Stop losetup Guesswork: Practical systemd-dissect for Discoverable Disk Images on Linux

What a DDI is (and what dissect does)

Prerequisites

Lab 0 — Build a tiny plain-FS image to practice on

Lab 1 — Inspect before you mount

Lab 2 — Mount and unmount the OS view

mount.ddi and fstab

Lab 3 — Run a command inside a temporary mount (--with)

Lab 4 — List files and mtree manifests

Lab 5 — Copy files in and out without a long-lived mount

Lab 6 — Attach as a loop device (when you still need block tools)

Lab 7 — Validate image policy before you trust an image

Lab 8 — Discover images on the system and archive them

Verity and encrypted images

Wire-up: nspawn, services, and friends

Lab 3 — Run a command inside a temporary mount (`--with`)

Secure Boot signing with `ukify genkey`

Install with `kernel-install` layout=uki