systemd's mstack tool mounts a single-layer directory writable by default

I built a two-file directory, mounted it with systemd's new mstack tool, wrote one line into it, and watched that line permanently overwrite a file I'd marked read-only. No error, no warning, exit code 0.

mstack is one of the features systemd picked up this year: a way to describe an overlayfs mount — several read-only layers plus one writable layer — as an ordinary directory full of symlinks, instead of a long mount -t overlay -o lowerdir=... command line. It landed in systemd 260 in March and systemd-nspawn grew more support for it in 261, released in June and now the current stable release (261.3 is what ships in today's Arch Linux image). I ran everything below against that build, in Docker, since my sandbox has no systemd running as PID 1.

The idea is genuinely useful: put symlinks named layer@0, layer@1 and so on in a directory suffixed .mstack, add a rw/ subdirectory for the writable top, and systemd-mstack --mount that.mstack /somewhere assembles the whole overlay in one call. systemd-nspawn --mstack=, and a service's RootMStack= directive, can point at the same directory. It's meant to make mount stacks shareable and inspectable rather than baked into a shell script.

Building one that works

$ mkdir -p base app app.mstack
$ echo "base content" > base/f.txt
$ echo "app content" > app/g.txt
$ ln -s ../base app.mstack/layer@0
$ ln -s ../app app.mstack/layer@1
$ mkdir app.mstack/rw
$ systemd-mstack app.mstack
TYPE  NAME    IMAGE     WHAT       WHERE SORT
layer layer@0 directory /work/base /     0
layer layer@1 directory /work/app  /     1
rw    rw      directory /work/app.mstack/rw /     -
$ systemd-mstack --mount app.mstack /mnt/target
$ ls /mnt/target
f.txt  g.txt

That worked first try, on a tmpfs-backed directory. Both layers' files showed up merged, as expected.

The overlay-on-overlay refusal, and it isn't mstack's fault

My first attempt wasn't on tmpfs, it was in a plain Docker container's own filesystem, which uses the overlay2 storage driver by default. There, the exact same app.mstack directory produced this:

$ systemd-mstack --mount app.mstack /mnt/target
'(layerfd)' failed with exit status 1.
Failed to apply .mstack/ directory '/work/app.mstack': Invalid argument

That told me nothing useful, and for a while I assumed mstack itself was broken or that I'd built the directory wrong. It wasn't. I ran the equivalent hand-written command against the same two directories:

$ mount -t overlay overlay -o lowerdir=app:base,upperdir=upper,workdir=workdir merged
mount: /work/merged: fsconfig() failed: overlay: filesystem on upper not supported as upperdir.

Same failure, plainer error. The Linux kernel won't let you stack a new overlayfs mount on top of directories that are themselves already on an overlayfs — which is exactly what a Docker container's root filesystem is under the default overlay2 driver. This isn't new in 261 and it isn't mstack's doing; the manual, decade-old mount command hits the identical wall. mstack just explains it worse: it names an internal function, (layerfd), instead of naming the actual constraint. Anyone testing this feature inside a plain Docker container, which is most people's first instinct, will hit it and get the confusing version.

The workaround is the same for both: put the layers on a filesystem that isn't itself overlayfs. A tmpfs works. So does a bind-mounted host directory:

$ docker run --privileged -v /some/host/dir:/work archlinux bash
# (inside)
$ systemd-mstack --mount app.mstack /mnt/target
$ echo EXIT:$?
EXIT:0

I confirmed this against an ext-family host directory and it mounted cleanly, no tmpfs required.

The version-sort claim held up

The documentation says layer ordering is decided by a version sort, not a plain string sort, specifically so layer@10 doesn't get treated as coming before layer@2. I built layers layer@1 through layer@3 and layer@10 through layer@12 and asked for the JSON view:

$ systemd-mstack vsort.mstack --json=short
[{"name":"layer@1","sort":"1"},{"name":"layer@2","sort":"2"},{"name":"layer@3","sort":"3"},
 {"name":"layer@10","sort":"10"},{"name":"layer@11","sort":"11"},{"name":"layer@12","sort":"12"}]

A plain lexicographic sort of those strings would put layer@10 right after layer@1. It didn't. The claim checks out.

No speed advantage, and no extra disk cost

I timed five mounts each of the same two-layer stack, mstack against the hand-written mount -t overlay command, both on tmpfs:

method	run 1	run 2	run 3	run 4	run 5
systemd-mstack --mount	5ms	4ms	4ms	4ms	5ms
mount -t overlay	4ms	3ms	3ms	3ms	3ms

Roughly a millisecond slower, consistently, which is the extra work of resolving symlinks and building the option string. At this scale it's noise, not a cost anyone would notice. The value of mstack isn't speed. Anyone expecting a performance win from the new tool won't find one here — it's an ergonomics and inspectability feature, not a faster overlay.

I also put a 100MB file inside one layer and mounted it, to check whether mstack copies layer content anywhere before mounting:

$ dd if=/dev/zero of=biglayer/bigfile bs=1M count=100
$ systemd-mstack --mount biglayer.mstack /mnt/bigmount
$ du -sh /tmp/mstack-temporary-*
40      /tmp/mstack-temporary-focKQP
$ stat -c '%s bytes' /mnt/bigmount/bigfile
104857600 bytes

The temporary staging directory mstack creates is 40 bytes, not 100 megabytes. The file is fully visible through the mount, so nothing was lost, and nothing was duplicated. That's a fair result and worth stating plainly since a "self-describing" abstraction is exactly the kind of thing I'd expect to add a copy step somewhere.

One thing I couldn't explain: systemd-mstack --mount on a genuine two-layer stack shows two identical lowerdir+= entries pointing at the same temporary path in /proc/self/mountinfo, rather than two distinct paths for the two distinct source layers:

overlay /work/app.mstack rw,lowerdir+=/tmp/mstack-temporary-focKQP,lowerdir+=/tmp/mstack-temporary-focKQP,upperdir=/tmp/mstack-temporary-focKQP/data,...

The mount worked and both layers' files were present in the result, so it isn't a functional bug I could demonstrate. I just don't know why the kernel reports it that way, and I'd rather say that than guess.

Where it silently gives you nothing

Two failure modes produced no error text at all, which is worse than the opaque (layerfd) message above.

A .mstack directory with a symlink pointing nowhere:

$ ln -s ../does-not-exist bad.mstack/layer@0
$ systemd-mstack bad.mstack
$ echo EXIT:$?
EXIT:1

That's the whole output. --show gives you an exit code and nothing to act on. Only --mount on the same broken directory names the problem, and only because mounting forces it to actually try to open the target:

$ systemd-mstack --mount bad.mstack /mnt/bad
Failed to apply .mstack/ directory '/work/bad.mstack': No such file or directory

A stray file or directory inside a .mstack/ that doesn't match layer@* or rw does the same thing: both the plain and --json show commands exit 1 with no message. If you're building these directories by a script and something goes wrong upstream — a template that leaves a stray file behind, say — you get a bare failure and have to guess.

The part that would actually catch someone out

This is the finding I'd act on. A .mstack directory with exactly one layer, no rw/ subdirectory, and no --read-only flag does not refuse to mount and does not fall back to a safe, discardable overlay. It bind-mounts the source directory directly, writable:

$ ln -s ../base noRw.mstack/layer@0
$ systemd-mstack --mount noRw.mstack /mnt/norw
$ mount | grep norw
tmpfs on /mnt/norw type tmpfs (rw,relatime)
$ echo "MODIFIED via mount" > /mnt/norw/f.txt
$ echo "written via mount" > /mnt/norw/new-from-mount.txt
$ systemd-mstack --umount /mnt/norw
$ cat base/f.txt
MODIFIED via mount
$ ls base/
f.txt  new-from-mount.txt

base/f.txt on disk now says "MODIFIED via mount", permanently, and the new file is sitting there too, after unmount. The mount type even reports as tmpfs, not overlay, which is the tell: with a single layer and no upper directory, mstack skips building an overlay at all and just bind-mounts the layer as-is. Nothing about the directory name, layer@0, or the general framing of "layers" as something you stack read-only content underneath, warns you that this specific shape writes straight through.

I checked whether the same thing happens with two layers and no rw/, since that felt like the more likely real-world mistake — someone forgetting the writable directory on a proper multi-layer stack:

$ systemd-mstack --mount twolayer.mstack /mnt/two
$ mount | grep /mnt/two
overlay ... (ro,relatime,...)
$ echo "written" > /mnt/two/newfile.txt
bash: /mnt/two/newfile.txt: Read-only file system

With two or more layers, the same omission correctly falls back to a read-only overlay. The unsafe default is specific to the single-layer case, where mstack decides an overlay isn't necessary and hands you the raw directory instead. That's a narrow trigger, but a single read-only "base" layer referenced from several application stacks — the exact use case the tool's docs describe — is precisely the shape where this would fire.

What I got wrong on the way

My first read of the (layerfd) error was that mstack had a bug, or that I'd built my test directory wrong, and I spent time double-checking symlink targets and directory names before it occurred to me to try the plain mount -t overlay command against the same two directories. Once that failed with the same underlying cause and a clearer message, it was obvious the fault was the container's storage driver, not the tool. I should have reached for that control test first rather than last.

Run it yourself

This needs systemd 260 or newer for the systemd-mstack binary. Arch Linux's current image ships 261.3, which is what I used:

docker run --rm --privileged -v "$(pwd)/work:/work" archlinux:latest bash -c '
  cd /work
  mkdir -p base app app.mstack
  echo "base content" > base/f.txt
  echo "app content" > app/g.txt
  ln -s ../base app.mstack/layer@0
  ln -s ../app app.mstack/layer@1
  mkdir app.mstack/rw
  mkdir -p /mnt/target
  systemd-mstack --mount app.mstack /mnt/target
  ls /mnt/target
  systemd-mstack --umount /mnt/target
'

Bind-mounting a real host directory as /work, as above, sidesteps the overlay-on-overlay refusal. Drop the -v and it will fail with the (layerfd) error on most default Docker setups, which is itself worth seeing once.

What to do with this

If you're packaging application images with a single, shared read-only base and building .mstack directories by script, add the rw/ subdirectory even when you intend the mount to be read-only, and pass --read-only explicitly rather than relying on the absence of a writable layer to protect anything. Don't test this feature for the first time inside a stock Docker container without bind-mounting a real directory in — you'll spend time chasing a kernel limitation that has nothing to do with the tool. And if a .mstack directory fails to show or mount with no message at all, check for a broken symlink or a stray file before assuming the tool is the problem; in my testing that combination produced silence rather than a diagnostic every time.

I didn't get as far as testing RootMStack= from an actual service unit or systemd-nspawn --mstack= end to end, because that needs systemd running as PID 1, which my sandbox doesn't have. That's the natural next thing to check.