DEV Community

Cover image for Stop Hand-Writing QEMU Flags: Practical systemd-vmspawn VMs on Linux
Lyra
Lyra

Posted on

Stop Hand-Writing QEMU Flags: Practical systemd-vmspawn VMs on Linux

#linux #systemd #devops #opensource

Stop Hand-Writing QEMU Flags: Practical systemd-vmspawn VMs on Linux

systemd-nspawn is great when a namespace container is enough. Sometimes it is not.

You need a real kernel, a TPM, Secure Boot firmware, or a guest that must look like bare metal. That is when people fall back to a pile of qemu-system-x86_64 flags, a one-off libvirt XML, or a desktop hypervisor UI.

systemd-vmspawn is systemd's answer: the same "spawn an OS image" ergonomics as systemd-nspawn, but it launches a full virtual machine (typically QEMU/KVM) instead of namespaces.

This guide is operational. Every flag and flow below comes from the current systemd-vmspawn(1) man page and related systemd docs.

What vmspawn is (and is not)

Tool Isolation Kernel Typical use
systemd-nspawn namespaces / cgroups host kernel OS trees, chroot-like labs
systemd-vmspawn full VM (KVM) guest kernel real OS images, firmware, TPM
libvirt / virt-manager full VM guest kernel long-lived multi-VM farms
Podman / Docker OCI app containers host kernel single apps

vmspawn is intentionally nspawn-shaped:

  • start from --image= or --directory=
  • optional --machine= name
  • optional registration with systemd-machined / machinectl
  • credentials, bind mounts, journal forwarding
  • scope unit properties for CPU/memory caps

It is not a replacement for a full multi-tenant hypervisor control plane. Think: "I have a disk image and I want a correct KVM boot now."

Added in systemd 255, with substantial options through 256–262.

Prerequisites

On a typical Debian/Ubuntu host:

# packages (names vary slightly by distro)
sudo apt install systemd-container qemu-system-x86 ovmf

# KVM access for your user (needed for vsock on Debian/Ubuntu)
sudo usermod -aG kvm "$USER"
# re-login after this

# optional: software TPM for guests
sudo apt install swtpm swtpm-tools

# optional: build disk images the systemd way
sudo apt install mkosi
Enter fullscreen mode Exit fullscreen mode

Sanity checks:

systemd-vmspawn --version
ls -l /dev/kvm
# firmware discovery (lists JSON firmware definitions)
systemd-vmspawn --firmware=list
Enter fullscreen mode Exit fullscreen mode

On systems where /dev/kvm is missing (nested virt disabled, cloud instance without KVM), vmspawn can still run with --kvm=no, but expect pure TCG emulation and slower boots.

Lab 1 — Build a tiny image and boot it

The man page's first example is still the cleanest path. With mkosi:

mkdir -p ~/vmspawn-lab && cd ~/vmspawn-lab

# Arch example from systemd-vmspawn(1); swap -d for your distro if preferred
mkosi -d arch -p systemd -p linux --autologin -o image.raw -f build

systemd-vmspawn --image=image.raw
Enter fullscreen mode Exit fullscreen mode

That drops you on an interactive guest console (--console=interactive is the default).

Import a cloud image instead

If you already use machinectl / importctl:

# Example pattern from the man page family: pull a cloud image, then boot the .raw
sudo machinectl pull-raw \
  https://download.fedoraproject.org/pub/fedora/linux/releases/41/Cloud/x86_64/images/ \
  Fedora-Cloud  # adjust to a concrete image URL on your host

# After import, boot the raw under /var/lib/machines (path varies)
sudo systemd-vmspawn \
  --image=/var/lib/machines/Fedora-Cloud.raw \
  --machine=fedora-cloud \
  --cpus=2 \
  --ram=2G \
  --network-user-mode
Enter fullscreen mode Exit fullscreen mode

Prefer an explicit full URL to a known cloud image on your distro's download site. The point is the boot command, not chasing a moving Fedora filename.

Core knobs you will actually use

Image vs directory 

# Disk image (raw by default; qcow2 via --image-format=)
systemd-vmspawn --image=./disk.raw
systemd-vmspawn --image=./disk.qcow2 --image-format=qcow2

# Directory tree as root (virtiofs under the hood)
systemd-vmspawn --directory=./rootfs
Enter fullscreen mode Exit fullscreen mode

Rules from the man page:

  • One of --directory= or --image= is required; if neither is given, --directory=. is assumed.
  • --ephemeral (-x) takes a temporary snapshot of the image and deletes it when the VM exits. Works with --image= only. Does not combine with --extra-drive=.
  • --grow-image=20G / -G 20G expands a too-small image file (rounded up to 4K). With --ephemeral, growth applies to the snapshot copy.

CPU, RAM, KVM, vsock, TPM 

systemd-vmspawn \
  --image=./disk.raw \
  --machine=lab1 \
  --cpus=4 \
  --ram=4G \
  --kvm=yes \
  --vsock=yes \
  --tpm=yes
Enter fullscreen mode Exit fullscreen mode

Defaults worth remembering:

Option Default Notes
--cpus= 1
--ram= 2G newer systemd: BYTES[:MAXBYTES[:SLOTS]] for memory hotplug
--kvm= auto detect /dev/kvm
--vsock= auto guest AF_VSOCK; Debian/Ubuntu need kvm group
--tpm= auto needs swtpm on PATH
--tpm-state= auto path derived as <image>.tpmstate; off is transient
--discard-disk= yes pass TRIM/discard through to the image
--notify-ready= true waits for guest init READY=1 (nspawn defaults the opposite)

--tpm-state=off (or ephemeral + auto) is wrong for guests that bind disk encryption keys to the vTPM — those keys vanish on every shutdown.

Networking: user-mode vs TAP

User-mode (no root, NAT via QEMU):

systemd-vmspawn --image=./disk.raw --network-user-mode
Enter fullscreen mode Exit fullscreen mode

TAP (root, proper L2, needs systemd-networkd):

sudo systemd-vmspawn --image=./disk.raw --network-tap -M taplab
Enter fullscreen mode Exit fullscreen mode

Requirements called out by the man page:

  • root privileges for TAP
  • systemd-networkd running on the host
  • stock unit: /usr/lib/systemd/network/80-vm-vt.network

If TAP comes up but the guest has no DHCP, check that networkd owns the host vt-* side and that your nftables forward/NAT policy allows it.

Firmware and Secure Boot 

# See what firmware JSON files systemd can find
systemd-vmspawn --firmware=list

# Prefer Secure Boot-capable firmware
systemd-vmspawn --image=./disk.raw --secure-boot=yes

# Direct kernel boot (no firmware), or UKI path
systemd-vmspawn \
  --directory=./rootfs \
  --linux=./rootfs/boot/vmlinuz \
  --initrd=./rootfs/boot/initrd.img
Enter fullscreen mode Exit fullscreen mode

On recent systemd builds, --firmware= also accepts auto|uefi|bios|none, a path to a firmware definition, or describe to print the selected UEFI image. Booting a UKI requires UEFI firmware. Excess CLI arguments after options are passed as extra kernel cmdline via SMBIOS.

Console modes 

--console=interactive   # default: host TTY ↔ guest console
--console=read-only     # watch only
--console=native        # QEMU monitor available
--console=gui           # graphical QEMU UI
Enter fullscreen mode Exit fullscreen mode

--background=44 tints the terminal while the VM runs (interactive/read-only only).

Lab 2 — Directory root + private users + journal forward

This is the man page's "systemd system image" pattern, adapted for a local tree:

# Assume an OS tree at ./system (mkosi.output/system style)
# Map host subuid range into the guest via virtiofsd
SHIFT=$(grep "^$(whoami):" /etc/subuid | cut -d: -f2)

systemd-vmspawn \
  --directory=./system \
  --private-users="${SHIFT}" \
  --linux=./system.efi \
  --forward-journal=./vm.journal \
  --machine=sysimg \
  enforcing=0
Enter fullscreen mode Exit fullscreen mode

What each piece does:

  • --private-users=UID_SHIFT[:RANGE] — turn on UID/GID mapping for directory boots (default range 65536). Required when the directory is not root-owned in a way the host can safely expose.
  • --linux= — direct-boot a kernel or UKI; for directory images without --linux=, vmspawn searches BLS entries under /boot (XBOOTLDR) and /efi (ESP).
  • --forward-journal= — host-side file or directory; guest journal is received via systemd-journal-remote semantics.
  • enforcing=0 — extra kernel cmdline token (SMBIOS), useful so SELinux does not block first boot of a lab image.

Inspect the forwarded journal later:

journalctl --file=./vm.journal -b
# or, if you forwarded to a directory:
# journalctl --directory=./vm-journals
Enter fullscreen mode Exit fullscreen mode

Lab 3 — Resource caps via scope properties

When vmspawn is not run with --keep-unit, it registers a scope under machine.slice by default:

systemd-vmspawn \
  --image=./disk.raw \
  --machine=capped \
  --slice=machine.slice \
  --property=MemoryMax=2G \
  --property=CPUQuota=200% \
  --network-user-mode
Enter fullscreen mode Exit fullscreen mode

--property= accepts the same assignments as systemctl set-property. Use it for MemoryMax=, CPUQuota=, TasksMax=, and friends so a runaway guest cannot eat the host.

Lab 4 — Credentials into the guest

vmspawn mirrors unit credentials:

printf 's3cret-db-password\n' > ./db.pass
chmod 600 ./db.pass

systemd-vmspawn \
  --image=./disk.raw \
  --machine=credlab \
  --load-credential=dbpass:./db.pass \
  --set-credential=app.env:APP_ENV=lab \
  --network-user-mode
Enter fullscreen mode Exit fullscreen mode

Inside a systemd guest, those show up through the normal credentials directory ($CREDENTIALS_DIRECTORY / ImportCredential= / LoadCredential= in units). Binary values in --set-credential= use C-style escapes (\n, \x00); shells may unescape once, so double-escaping is sometimes required.

For host-side encryption of credential files before load, pair with systemd-creds encrypt and unit LoadCredentialEncrypted= — that is the service-manager path; vmspawn itself takes plaintext load/set forms as documented.

Lab 5 — SSH over vsock with systemd-ssh-proxy

The man page ends with this workflow:

CID=3735928559   # pick an unused CID in 3..0xFFFF_FFFE

systemd-vmspawn \
  --directory=./system \
  --private-users="$(grep "^$(whoami):" /etc/subuid | cut -d: -f2)" \
  --linux=./system.efi \
  --vsock-cid="$CID" \
  --machine=sshlab \
  enforcing=0
Enter fullscreen mode Exit fullscreen mode

In another terminal (while the VM runs):

# Ephemeral key path pattern from systemd-vmspawn(1)
ls /run/user/"$UID"/systemd/vmspawn/

ssh -o StrictHostKeyChecking=no \
  -i /run/user/"$UID"/systemd/vmspawn/machine-*-sshlab-ed25519 \
  "root@vsock/${CID}"
Enter fullscreen mode Exit fullscreen mode

Notes from the docs:

  • By default vmspawn generates an ephemeral SSH key so it can talk D-Bus into the guest (--pass-ssh-key=yes default). Keys live only for that invocation under /run/user/$UID/systemd/vmspawn/.
  • --ssh-key-type=ed25519 is default; rsa exists for ancient guest sshd.
  • On Debian/Ubuntu, vsock needs membership in group kvm.
  • ssh root@vsock/$CID needs a client that understands the systemd vsock proxy path (systemd-ssh-proxy integration).

Disable key generation only if you provide another way in: --pass-ssh-key=no.

Bind mounts, extra disks, bind-user 

# Host path → same path in guest
systemd-vmspawn --image=./disk.raw \
  --bind=/var/cache/build \
  --bind-ro=/usr/src/linux-headers

# Host:guest path pair (escape colons with \:)
systemd-vmspawn --image=./disk.raw \
  --bind=/home/you/proj:/opt/proj

# Additional data disk
systemd-vmspawn --image=./disk.raw \
  --extra-drive=raw:./data.raw \
  --extra-drive=qcow2:./bulk.qcow2
Enter fullscreen mode Exit fullscreen mode

--bind-user=alice (systemd 259+) is stronger than a plain bind:

  1. Host home is exposed under /run/vmhost/home/ via virtiofs with UID translation.
  2. Transient user/group records are injected as userdb.transient.* credentials so nss-systemd in the guest can resolve the account.

Caveats from the man page (read these before using it on untrusted guests):

  • Guest needs systemd 258+ with nss-systemd in nsswitch.conf.
  • The propagated record includes the UNIX password hash — use a strong hash (yescrypt / $y$) on the host.
  • Mapping is transient; leftover files owned by a recycled guest UID can become someone else's later.

machined registration 

# As root, registration defaults on; as user, defaults off (Debian man page)
sudo systemd-vmspawn --image=./disk.raw --machine=reg1 --register=yes

machinectl list
machinectl status reg1
machinectl shell reg1
Enter fullscreen mode Exit fullscreen mode

Recent systemd also adds --system / --user to pick which manager / machined instance to talk to (v260+).

Ephemeral smoke-test pattern

Golden image stays clean; every run is disposable:

systemd-vmspawn \
  --image=./golden.raw \
  --ephemeral \
  --grow-image=30G \
  --cpus=2 \
  --ram=2G \
  --network-user-mode \
  --machine=smoke-$$ \
  --set-credential=run.id:smoke-$(date -u +%Y%m%dT%H%M%SZ)
Enter fullscreen mode Exit fullscreen mode

When the process exits, the snapshot is gone. Do not attach long-lived --extra-drive= in this mode (unsupported with --ephemeral).

systemd unit wrapper (optional)

For a lab VM you want under systemd supervision:

# /etc/systemd/system/vmspawn-lab@.service
[Unit]
Description=vmspawn lab VM %i
After=network-online.target
Wants=network-online.target

[Service]
# Type=notify works well because vmspawn notifies readiness
# after the guest is ready (--notify-ready=true by default)
Type=notify
ExecStart=/usr/bin/systemd-vmspawn \
  --image=/var/lib/machines/%i.raw \
  --machine=%i \
  --cpus=2 \
  --ram=2G \
  --network-tap \
  --register=yes \
  --property=MemoryMax=3G
KillMode=mixed
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo systemctl start vmspawn-lab@webtest
sudo systemctl status vmspawn-lab@webtest
machinectl status webtest
Enter fullscreen mode Exit fullscreen mode

Exit status

From systemd-vmspawn(1):

  • On tool/setup failure, the errno value is propagated.
  • If the guest supplies EXIT_STATUS, that is returned.
  • Otherwise success.

Boundaries — pick the right tool

Need Prefer
Same-kernel OS tree, fast iteration systemd-nspawn / machinectl
Host /usr add-on without a VM systemd-sysext / portable services
Real guest kernel, firmware, TPM, Secure Boot systemd-vmspawn
Multi-node virt farm, live migration, fancy storage pools libvirt / oVirt / Proxmox
Single app packaging Podman / Docker / Quadlet

Confidential computing (--coco=sev-snp|tdx) exists on recent systemd builds but is marked experimental in the man page — treat it as a research path, not a homelab default.

Troubleshooting checklist

  1. No KVM — ls -l /dev/kvm; nest virt; or --kvm=no for a slow lab.
  2. vsock / SSH fails on Debian — user in kvm group; re-login.
  3. TAP has no address — networkd active; 80-vm-vt.network present; check nftables.
  4. Directory boot permission errors — set --private-users= from /etc/subuid.
  5. Guest never "ready" — guest init must sd_notify READY=1, or pass --notify-ready=no.
  6. TPM-bound LUKS unlock dies every boot — do not use --tpm-state=off / ephemeral auto-off.
  7. Firmware missing — install OVMF/AAVMF packages; --firmware=list.

Wrap-up

systemd-vmspawn closes the gap between "nspawn is too weak" and "I guess I will memorize forty QEMU flags."

Practical defaults for day-to-day labs:

systemd-vmspawn \
  --image=./disk.raw \
  --machine=lab \
  --cpus=2 \
  --ram=2G \
  --network-user-mode \
  --tpm=yes \
  --register=yes
Enter fullscreen mode Exit fullscreen mode

From there, add ephemeral snapshots for CI-style smoke tests, TAP when the guest must sit on a real L2 segment, credentials instead of baking secrets into the image, and vsock+SSH when you want a shell without wiring a second NIC.

References

Top comments (1)

Collapse
 
dhruv_malaviya_cdcc71e595 profile image
Dhruv Malaviya •

That comparison table is the clearest version of the isolation ladder I've seen in one place, and "nspawn-shaped ergonomics, full VM underneath" is a genuinely good design instinct.

The row you don't have is the one where someone else owns the host. That's Krova Cloud : each Cube is a full VM with its own guest kernel, provisioned by one API call with vCPU, RAM and disk rather than a flag list or libvirt XML. No public IP by default, and every port mapping takes an IP allowlist behind a stateful default-deny firewall.

Your kernel point is the one that matters and it's worth restating: because the kernel comes from the host at boot, reboot inside a Cube doesn't change it and nothing warns you. Getting onto a refreshed kernel needs a cold restart. Same class of surprise as your QEMU flag pile, different layer.

Where vmspawn wins outright, and I won't pretend otherwise: TPM, Secure Boot firmware, and guests that must look like bare metal. We don't offer any of those.

Is TPM passthrough the main reason you moved off nspawn, or was it just needing a real kernel?