DEV Community: Mustafa ERBAY

systemd-nspawn: Containers Without Images or Registries

Mustafa ERBAY — Mon, 07 Sep 2026 08:13:46 +0000

The word "container" has collapsed into one meaning: pull an image, unpack the layers, tell the daemon, run it. But that isn't the only way to use the isolation the kernel offers — and one alternative is probably already packaged for your server.

systemd-nspawn takes a directory and runs a full operating system inside it. No image format, no registry, no background daemon. In its own documentation's words it resembles chroot but is far more powerful: alongside the file system hierarchy it virtualises the process tree, the various IPC subsystems, and the host and domain names.

This article is about where that tool sits: what it isolates, what it doesn't, and the one condition for using it safely.

What does it isolate?

The documentation's list is clear. From inside the container, access to kernel interfaces such as /sys/, /proc/sys/ and /sys/fs/selinux/ is read-only; the host's network interfaces and the system clock can't be changed from within; device nodes may not be created; the host can't be rebooted and kernel modules can't be loaded.

So most "accident" scenarios are closed. A build script can't change a sysctl by mistake, an installer can't shift the host's clock.

The practical distinction I draw from that: in its default form this list of restrictions prevents accidents. Confining malicious code is a separate matter, and the documentation is remarkably direct about it.

The one condition: user namespaces

The decisive sentence in the docs: if user namespaces are not used, this sandbox can easily be circumvented from within the container. The conclusion is equally clear — untrusted code must always be run in a user namespace.

The key is the --private-users= option, which has several modes:

pick: turns on user namespacing and chooses the UID/GID range automatically. It reads a range from the root directory's owner, and if another container is using it, picks a new unused range between host UIDs 524288 and 1878982656. The cost isn't just time: this mode also triggers ownership adjustment, so the tree's file ownership on the host shifts permanently. Factor that in if you'll move the tree with tar/rsync or edit it from the host; that's why the documentation makes ownership behaviour configurable separately (--private-users-ownership=).
identity: an identity mapping for the first 65536 UIDs/GIDs. It gives no UID isolation, only capability isolation — and the docs say plainly that it "is not secure and must not be used to run untrusted code".
managed: delegates UID range allocation to systemd-nsresourced; this is the default when invoked unprivileged.
no: no user namespace at all.

Upstream's own recommendation is explicit: in most cases --private-users=managed — or pick when you're privileged — is the recommended option, because user namespacing is advised for security and massively enhances container security. The same warning applies to no: not secure, and not to be used for untrusted code.

So before saying "it runs in a container, so I'm safe", check which mode you're in.

No images, just directories

This is the big mental shift. In Docker you pull an image; with nspawn you prepare an operating system tree. The docs name the tools too: dnf, debootstrap or pacman set up a filesystem hierarchy suitable for the job.

The location is standardised as well: /var/lib/machines/ is the suggested directory for OS container images installed on the system. When you pass --machine=, a tree is searched for in a couple of locations, most importantly there.

There's a small but annoying check: when booting a container, nspawn verifies that /usr/lib/os-release or /etc/os-release exists in the tree. If you're only opening a shell the check doesn't apply; if you're booting a very old distribution tree you may have to add the file by hand.

The most useful option while experimenting is --ephemeral: the container runs on a temporary snapshot of its file system, which is removed when it terminates. The docs add two notes: taking that snapshot is far more efficient on file systems with native subvolume snapshots or reflinks (btrfs or new xfs), and the switch leaves hostname, machine ID and other identifying settings unmodified. The second matters in CI: launch parallel runs from the same tree and you get two containers with the same machine ID.

The first container: four commands

The shortest path to making this concrete. On a Debian-based host, set up a Debian tree and step into it:

apt install systemd-container debootstrap
debootstrap --include=systemd,dbus stable /var/lib/machines/demo
systemd-nspawn --directory=/var/lib/machines/demo --private-users=pick

The third command drops you into a shell inside the container. To boot the full OS with its init you add --boot — which is already the default when you run it as a service. On a fresh debootstrap tree the root password is locked, so --boot leaves you at a login prompt you can't get past; set the password first:

systemd-nspawn --directory=/var/lib/machines/demo passwd root

To run it as a service, machinectl steps in:

machinectl start demo         # runs as systemd-nspawn@demo.service
machinectl enable --now demo  # use this if it should also start at boot
machinectl shell demo         # open a shell inside
machinectl poweroff demo      # shut it down cleanly

start is one-shot; if you want the container up after a reboot you need enable, which the documentation describes as having the same effect as systemctl enable.

The container is now an ordinary systemd service, and resource control happens the ordinary way:

systemctl set-property systemd-nspawn@demo.service MemoryMax=2G CPUQuota=200%

You don't learn a separate "container resource management" layer. But know the split: those settings belong to the unit; the .nspawn file doesn't express cgroup limits, where the equivalents are limited to Limit* (rlimits), CPUAffinity and OOMScoreAdjust.

Networking: shared by default

Networking in nspawn depends on how you start it. From the command line, if you say nothing, the container shares the host's network stack — practical for a quick test, insufficient for isolation. Started as a service, the template unit applies --network-veth, so the container is born in its own network namespace. Don't go looking for the host network in a container you started with machinectl start.

--private-network gives the container its own network namespace; --network-veth creates a virtual Ethernet link between host and container, with the container-side interface named host0. The latter implies the former.

Here's the built-in convenience: systemd-networkd ships two files by default — one matching the host-side ve-* interfaces, enabling automatic address provisioning over DHCP and automatic IP routing onto the host's external interfaces, and one matching the container-side host0 interface, enabling client-side address assignment via DHCP. So with systemd-networkd, networking usually "just works"; without it, you set up bridging and NAT yourself.

In unprivileged mode the options narrow: only --private-network and --network-veth are supported. More importantly, the first item on that same list limits this article's framing directly — in unprivileged mode only disk-image-based containers are supported; directory-based ones work only if owned by the "foreign" UID range.

Running it as a service

If you'd rather not start the container by hand, the infrastructure is there: machinectl provides commands to run containers as system services through the systemd-nspawn@.service template unit. Each container instance runs as its own service instance, with the container name as the instance identifier.

There are behavioural differences that catch people out, and they touch both of this article's main themes. Here's the command line the template unit runs:

ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest           --network-veth -U --settings=override --machine=%i

Note three things: --boot starts the init process inside the container (not the default from the command line), -U already turns on user namespacing (so on the service path the template handles --private-users for you), and --network-veth puts the container in its own network namespace. Thanks to --link-journal=try-guest, the container's logs are readable from the host with journalctl -M <name>.

"It worked on the command line but behaved differently as a service" usually traces back to these three.

You don't have to bury settings in the unit file either: a settings file with the .nspawn suffix may sit alongside each container, and it overrides the template unit's defaults.

It also runs OCI bundles

Version footing matters here: some of these options are recent. The managed mode and the maturing of unprivileged nspawn arrived with systemd 258; --mstack= for stacking image layers and importctl pull-oci with 260. On Debian 12 (systemd 252) you'll find none of them.

A lesser-known option is --oci-bundle=, which takes the path to an OCI runtime bundle as specified in the OCI Runtime Specification. In that case no .nspawn file is loaded; the root directory and various settings are read from the OCI runtime JSON data, with command-line data taking precedence.

In current systemd the connection is closer still: OCI images pulled with importctl pull-oci can be stored as a stack and run with nspawn. So treating nspawn as an island closed off from the container ecosystem is wrong; the accurate statement is that the systemd-nspawn binary doesn't do image distribution — that job lives in a separate systemd component (importctl).

The state on my server

$ dpkg -l systemd-container | tail -1
un  systemd-container <none> <none> (no description available)

$ ls /var/lib/machines
ls: cannot access '/var/lib/machines': No such file or directory

$ cat /proc/sys/user/max_user_namespaces
386072

The un on the first line means the package isn't installed: on Ubuntu, systemd-nspawn and machinectl ship in a separate package, and this machine doesn't have it. So "systemd is everywhere, therefore nspawn is everywhere" is wrong.

The third line shows user namespaces are plentiful — but that isn't the only place to look. The same machine also has this:

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

Ubuntu 24.04 restricts unprivileged processes from creating user namespaces via AppArmor. It doesn't stop you using --private-users=pick as root, but it's the wall you'll hit if you try unprivileged nspawn. On Debian the equivalent is kernel.unprivileged_userns_clone.

Instead of Docker, or alongside it?

The comparison needs the right axis. Docker/Podman brought two things: distribution (image format, registry, layer sharing, versioning) and security defaults (a seccomp profile, dropped capabilities, rootless plus user namespaces in Podman). nspawn does none of the first, and its default on the second is weaker — you switch security on yourself with --private-users.

What nspawn is good at is running a full operating system. If you want an environment with systemd inside, several services, "like a small VM but not a VM", it's the natural choice. Typical uses: building and testing on different distribution releases, exercising systemd units under a real init, keeping a legacy application alive with its own distribution tree.

If you're running one process per application — today's microservice pattern — there's no reason to reach for nspawn. If you want a system container and you're already in the systemd world, it solves that without installing another daemon.

You can build a similar setup with the approach in my article on rootless containers under systemd with Podman quadlets; the difference is that quadlets bind OCI images to systemd, while nspawn doesn't use the concept of an image at all.

Checklist

Verify the package is installed; the presence of systemd doesn't imply the presence of nspawn.
If you'll run untrusted code, don't start without --private-users=pick; the identity mode provides no security and the docs say so explicitly.
Prefer --ephemeral for experiments and CI; account for the snapshot cost if your filesystem isn't btrfs or new xfs.
If you run it as a service, know the --boot difference; command-line behaviour and service behaviour aren't the same.
Keep settings in a .nspawn file rather than editing the unit.
Choose the network model up front: in unprivileged mode only private network and veth are supported.
Don't forget the os-release file when building the OS tree; nspawn checks for it.

A container isn't one thing

What interests me most about nspawn is how narrow the word "container" has become. There is no container object on the kernel side; there are namespaces, cgroups and capabilities. Every tool built on top composes those parts differently.

Docker built its composition around distribution and won. nspawn never enters the distribution business and solves only the running side — which is why you hear about it less, and why it stays simpler for some jobs.

So the question for your own setup: is your reason for containers image distribution, running a full operating system, or a tight security boundary? For the first, Docker/Podman; for the second, nspawn; for the third, a tool whose defaults are set in your favour — or nspawn with user namespaces on.

Official Sources

The Clock You Can't Stop: PostgreSQL XID Wraparound

Mustafa ERBAY — Mon, 07 Sep 2026 08:05:51 +0000

There's a clock counting down in the background of PostgreSQL, and most teams notice it for the first time when it has nearly run out.

The clock is the transaction ID — XID for short. Every write takes an XID, and the counter is 32-bit, so it wraps after four billion transactions. In the circular scheme the documentation describes, each XID has roughly two billion "older" and two billion "newer" neighbours; when the counter wraps, old transactions suddenly appear to be in the future, which means catastrophic data loss.

That's why PostgreSQL carries a safeguard: it freezes rows. VACUUM marks sufficiently old row versions as frozen; frozen rows appear to be in the past to all normal transactions no matter what the counter does, and stay valid until deleted.

This article is about that clock: how to read it, what happens at each threshold, and what not to do when the alarm goes off.

What the queries return on a trouble-free cluster

Real output from a PostgreSQL 16 instance on my own server:

$ psql -At -F'|' -c "select name, setting from pg_settings where name in
    ('server_version','autovacuum_freeze_max_age','vacuum_freeze_min_age',
     'vacuum_freeze_table_age','vacuum_failsafe_age',
     'autovacuum_multixact_freeze_max_age') order by name;"
autovacuum_freeze_max_age|200000000
autovacuum_multixact_freeze_max_age|400000000
server_version|16.13
vacuum_failsafe_age|1600000000
vacuum_freeze_min_age|50000000
vacuum_freeze_table_age|150000000

$ psql -At -F'|' -c "select datname, age(datfrozenxid) from pg_database order by 2 desc;"
postgres|4259
burcu_mutfak|4259
template1|4259
template0|4259

This output comes from a trouble-free cluster: an age of 4,259 is about two hundred-thousandths of the forced-intervention threshold at 200 million. To be honest, this table proves the cluster is young and quiet rather than proving how well freezing works — all four databases sitting at the same value is the giveaway. What matters more is knowing what these two queries say. age(datfrozenxid) gives the distance between the oldest unfrozen XID in that database and the current XID. As the number grows, the clock advances.

For a per-table view the documentation provides the query:

SELECT c.oid::regclass AS table_name,
       greatest(age(c.relfrozenxid), age(t.relfrozenxid)) AS age
FROM pg_class c
LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm')
ORDER BY 2 DESC;

A common misconception needs correcting here: datfrozenxid is, in the documentation's words, just the minimum of the per-table relfrozenxid values in that database. So the database age equals the age of its oldest table; "the database looks fine but one table is behind" can't happen. The per-table query's value is different: the database-level number tells you something has fallen behind, while only this list tells you which.

The greatest(...) in the query is no accident either: a table's TOAST table can be the one that's actually behind, so the larger of the two is taken. One more note: datfrozenxid is only recomputed when a vacuum updates it, so the number you see isn't live.

The thresholds: four separate lines

The parameter names look alike, so they get confused; in fact the four look at four different moments.

vacuum_freeze_min_age (default 50 million) is the minimum age a row version needs before it can be frozen. A lower value means more freezing work, a higher one less work but later protection.

vacuum_freeze_table_age (default 150 million) is the threshold where a normal VACUUM switches to aggressive mode — visiting every page that might contain unfrozen XIDs, not just those that might contain dead tuples. The documentation notes its effective maximum is 0.95 × autovacuum_freeze_max_age.

autovacuum_freeze_max_age (default 200 million) is where forced anti-wraparound autovacuum kicks in. It runs even if autovacuum is disabled; at that point PostgreSQL doesn't consult your preference.

vacuum_failsafe_age (default 1.6 billion) is the strategy of last resort: at that age any cost-based delay stops applying, index cleanup is skipped, and the buffer access strategy is disabled so vacuum can use all of shared buffers. If you're here, you're in recovery. There's a linked behaviour too: raise autovacuum_freeze_max_age and the failsafe is silently adjusted to at least 105% of it.

Multixacts have their own counter as well: autovacuum_multixact_freeze_max_age, 400 million in my instance. On systems making heavy use of SELECT ... FOR SHARE or foreign key locks this counter can fill before the XID one — and hardly anyone watches it.

What the server says when the alarm rings

Once you enter the warning phase, the server starts speaking on every transaction. But you need to know how late that warning is: in the source, the warning limit is set 40 million transactions before the wraparound point — that is, when age reaches 2.1 billion. The hard limit follows just 3 million transactions later. This isn't an early warning, it's the last exit sign. (In the PostgreSQL 19 development branch that window was widened to 100 million; still late.)

WARNING:  database "mydb" must be vacuumed within 39985967 transactions
HINT:  To avoid XID assignment failures, execute a database-wide VACUUM in that database.

At the hard limit, work stops:

ERROR:  database is not accepting commands that assign new transaction IDs
to avoid wraparound data loss in database "mydb"

The documentation's wording is precise: transactions already in progress can continue, but only read-only transactions can be started; operations that modify records or truncate relations fail. VACUUM still runs normally — that's your way out. And don't wait on that vacuum blindly: the pg_stat_progress_vacuum view shows which phase it's in and how far it has got.

These messages also vary by version; the ones above are from the current documentation. PostgreSQL 16 words the error slightly differently, so if you're writing log matching, check your own version's text.

The logic in the source code describes the same three tiers: past xidVacLimit autovacuum starts being forced, past xidWarnLimit warnings are issued, and past xidStopLimit executing transactions is refused outside single-user mode.

Why does age grow? Usually vacuum isn't the culprit

The most common mistake here is diving into autovacuum settings when the alarm rings. Vacuum is usually ready to work; something is blocking it. The documentation lists these blockers among the recovery steps, and in practice you read the list backwards:

Unfinished prepared transactions: a row in pg_prepared_xacts with a large age means nothing beyond that XID can be considered freezable.
Long-running transactions: large age(backend_xid) or age(backend_xmin) values in pg_stat_activity. A forgotten BEGIN, a psql session left open.
Stale replication slots: large age(xmin) or age(catalog_xmin) in pg_replication_slots. A replica that was torn down may have left a slot holding things back.
Feedback from replicas: with hot_standby_feedback on, rows the standby needs are held on the primary; pg_stat_replication.backend_xmin shows how far behind that is.

Until those three are cleared, age doesn't drop however much vacuum runs. Tuning autovacuum itself is another matter; the thresholds I covered in the PostgreSQL VACUUM and bloat article apply to the bloat side, while on the wraparound side the priority is removing blockers.

Insert-only tables: the classic trap

One classic source of wraparound trouble is tables that are never updated. Event logs, time series, audit trails — tables that constantly receive rows but are almost never updated or deleted.

Classic autovacuum ignored those tables, because its trigger was the dead tuple count and there were no dead tuples. Yet the freezing need was there: every inserted row carries an XID. autovacuum_vacuum_insert_threshold, added in PostgreSQL 13, closed that gap; the default is 1000 inserted tuples, and -1 disables it entirely.

On clusters upgraded from older versions, or setups that turned this off as "unnecessary I/O", the trap is still there: inserted rows are never frozen, age grows quietly, and when the alarm rings the table that's behind is usually one of these. Remember that partitioned time series carry a relfrozenxid per partition — partitions show up as separate rows in the table list.

PostgreSQL 18 changed this picture somewhat: with vacuum_max_eager_freeze_failure_rate (default 0.03), a normal vacuum will try to proactively freeze all-visible but not-all-frozen pages. So on 18 and later the risk of append-only tables quietly falling behind is reduced; on older versions it's unchanged.

Lock conflicts: what stops vacuum

Another subtle mechanism lives in locks. Under normal conditions autovacuum doesn't block other commands: if a process requests a lock conflicting with the SHARE UPDATE EXCLUSIVE lock autovacuum holds, the autovacuum is interrupted and gets out of the way.

The subtlety: that behaviour doesn't apply to anti-wraparound vacuum. The documentation is explicit — if the autovacuum is running to prevent wraparound (its query name in pg_stat_activity ends with "(to prevent wraparound)"), it is not automatically interrupted. The docs also warn that regularly running commands which take locks conflicting with SHARE UPDATE EXCLUSIVE — ANALYZE, for instance — can effectively prevent autovacuum from doing its job.

The operational translation runs both ways. On one hand, if your scheduled maintenance keeps interrupting vacuum, table age grows quietly. On the other, once things reach the anti-wraparound stage vacuum no longer yields; an ALTER TABLE starts waiting and the team panics that "the database is locked up". Two ends of the same story.

What not to do

Two reflexes during recovery make things worse.

First, VACUUM FULL. The documentation warns plainly: it requires an XID and will therefore fail — except in super-user mode, where it instead consumes an XID and thus increases the wraparound risk. Second, VACUUM FREEZE: it does more work than needed and burns time you don't have. The right command is plain VACUUM, run as a superuser so system catalogues are processed too.

A third, older reflex: dropping into single-user mode. The documentation says this is no longer necessary in typical scenarios, should be avoided since it takes the system down, and is riskier — because that mode disables the wraparound safeguards designed to prevent data loss.

The cost on the storage side

Raising autovacuum_freeze_max_age is tempting: vacuum runs less often, I/O drops. The price is growth in pg_xact and pg_commit_ts. The documentation gives concrete sizes: at the maximum (2 billion), pg_xact is about 500 MB and pg_commit_ts about 20 GB; at the default 200 million, about 50 MB and 2 GB respectively.

So this parameter isn't a "performance setting" but a deliberate trade between disk and vacuum work. And the mandatory rule doesn't change: every table in every database must be vacuumed at least once every two billion transactions.

Monitoring: a one-line insurance policy

The nice thing about this topic is how cheap the monitoring is. One query and one threshold suffice:

SELECT max(age(datfrozenxid)) FROM pg_database;

But put the threshold in the right place. Alerting on a percentage of autovacuum_freeze_max_age is misleading: with the defaults, reaching 150 million (75% of that value) is normal — aggressive vacuum triggers exactly there. On a busy cluster, moving inside that band is a sign of healthy operation.

Three better signals: (1) is the max(age(datfrozenxid)) trend monotonically rising — i.e. does vacuum never bring the age down; (2) is there a vacuum ending in "(to prevent wraparound)" in pg_stat_activity; (3) is the age approaching vacuum_failsafe_age (1.6 billion by default). If you want absolute thresholds, put the warning around 1 billion and the critical below the failsafe.

As a checklist:

Alert on max(age(datfrozenxid)); define the threshold as a percentage of autovacuum_freeze_max_age.
Watch the multixact side with a query: select datname, mxid_age(datminmxid) from pg_database order by 2 desc; — separate counter, separate threshold (autovacuum_multixact_freeze_max_age).
On a managed service (RDS, Aurora, Cloud SQL) there's no superuser and no single-user mode; you manage parameters through the provider's parameter group and put the provider's wraparound metric on your dashboard.
Run a monthly blocker sweep: pg_prepared_xacts, long transactions, unused replication slots.
Check autovacuum_vacuum_insert_threshold on insert-only tables; if it's disabled, age grows silently.
Look for vacuums ending in "(to prevent wraparound)" in pg_stat_activity: that's emergency vacuum, not routine maintenance, and it makes lock requesters wait.
Put the top five oldest tables on your dashboard; trouble usually starts in one table, not in the database average.
Write "don't use VACUUM FULL or VACUUM FREEZE" into the recovery runbook; those are exactly what comes to mind in a crisis.
Check the age before an upgrade: pg_upgrade preserves freeze information, but the post-upgrade analyze and the aggressive vacuum it triggers can stretch the maintenance window you planned.

Counters fill quietly

XID wraparound reminds me of the sneakiest part of running databases: some problems don't arrive as a slowdown, they arrive one day as a wall.

CPU and memory metrics warn you gradually; a counter filling up advances linearly and gives nothing away until the last moment. Then writes are refused and all you can do is wait for vacuum to finish.

So the question for your own setup: which of your counters advance linearly, and how many of them do you have alerts on? In PostgreSQL, XID is the best known — but not the only one.

Official Sources

Lazy Pulling: The Part of the Image Nobody Reads

Mustafa ERBAY — Mon, 07 Sep 2026 07:43:45 +0000

When a container comes up, most of the time goes into one job: downloading the image. The application itself starts in milliseconds, but gigabytes of layers have to be pulled and unpacked first. On a workload that scales, this decides the answer to "when does the new pod take traffic".

The annoying part is that most of the downloaded data is never read. There's a study behind that intuition, and it's the starting point for the lazy-pulling projects: as the SOCI snapshotter's documentation relays, Harter et al's 2016 FAST '16 work found that on a representative suite of images, image download accounts for 76% of container startup time, while on average only 6.4% of the fetched data is actually needed for the container to start doing useful work. Carry those numbers into 2026 with care: the images of that era aren't today's multi-gigabyte model images, and networks got faster. The direction holds; the magnitude depends on your image.

Lazy pulling builds on that observation: start the container without downloading the whole image, and fetch files in the background as they're read. The idea isn't new, but today it has several mature implementations: eStargz, zstd:chunked, SOCI and nydus. All of them work with standard registries; this article looks at the first three.

What I want to convey here is as much about which guarantee you keep in which stack as about the mechanism itself. Because "lazy pulling" isn't one thing: two implementations of the same idea come with two different security positions, and the difference hides in the name of a configuration option that's easy to skim past.

What happens on my own server

First, the ordinary case:

$ docker info | grep -E "Server Version|Storage Driver"
 Server Version: 29.4.3
 Storage Driver: overlayfs

Right now this machine pulls images the classic way: all layers come down, get unpacked, and then the container starts. But the overlayfs detail in that output matters — that's not the old overlay2 graph driver, it's the containerd snapshotter. The machine is already running on the containerd image store.

That difference retires a widely repeated claim: "Docker can't do lazy pulling" is no longer true. The stargz snapshotter's install documentation has a dedicated section for Docker (Moby); after a change that landed in Docker v24, writing this into daemon.json and running the snapshotter service is enough:

{
  "features": { "containerd-snapshotter": true },
  "storage-driver": "stargz"
}

So the side doing lazy pulling is still the snapshotter beneath the runtime; this isn't an image format change, it's a runtime configuration job. It's just that you can now do that configuration in Docker too.

eStargz: prioritised files and a TOC

What the stargz snapshotter does is fetch the necessary chunks of the image on demand: the container runs while the download continues in the background.

The format that makes this possible is eStargz. In the project's own words it's a lazily-pullable image format, and two of its properties matter. First, compatibility: it's compatible with OCI/Docker images, so it can be pushed to standard registries and still runs on eStargz-agnostic runtimes, Docker included. Second, prioritisation: the snapshotter prefetches and caches the files likely to be accessed while the container runs.

Its third property — the most important one for this article — is content verification. In the eStargz documentation's words, the goal is to ensure the downloaded metadata and the contents of all files are the expected ones, based on the calculated digests. Since the layer is pulled lazily, verification happens piece by piece: on mounting a layer the snapshotter fetches the TOC from the registry, recalculates its digest and compares it against the TOC digest in the verified manifest, then re-verifies each fetched chunk against the digest in its entry. So the equation "lazy pulling means no verification" is wrong.

The requirements are documented too: containerd newer than v1.4.2, a stargz snapshotter daemon on the nodes, configuration changes on the containerd side, and a tool like BuildKit v0.10+ to convert the image. On CRI-O and Podman the stargz snapshotter can't be used directly; the documentation points to a separate additional layer store (Stargz Store) for those.

zstd:chunked: skippable frames and range requests

The second approach comes from the Podman/containers-storage side, and its mechanism approaches the problem differently.

The classic layer format is tar+gzip; the standard was later augmented with tar+zstd. zstd:chunked is a variant of that zstd layer: it uses zstd's skippable frames to embed extra metadata inside the layer. The core of that metadata is a table of contents — the SHA-256 of each file plus the offsets of its chunks. On top of that, chunks are compressed separately.

The result: a client can dynamically fetch only the content it doesn't already have, using HTTP range requests. It doesn't have to download the whole layer.

One implementation detail matters here: each layer keeps metadata called chunked-manifest-cache, and when a new image is pulled the other layers on the system are scanned through it. If a file with a matching digest is found it isn't downloaded again — depending on configuration it's shared via a hard link or a reflink, and if the filesystem doesn't support that, a full physical copy is made. Where configured, there's also a best-effort attempt to enable fs-verity on the file.

So zstd:chunked doesn't just mean "download less", it means "don't download what you already have". Given how many libraries different images share, the saving moves from layer granularity down to file granularity.

SOCI: lazy loading without conversion

There's a third approach whose design choice differs from the others. AWS's SOCI ("Seekable OCI") snapshotter aims to lazily load standard OCI images without requiring a build-time conversion step.

The rationale is operational: existing lazy-loading snapshotters rely on a conversion step that produces a new image artifact, which is a problem for teams that can't change their CI/CD pipeline or don't want to keep the same image in two formats. And — critically for security — the conversion step invalidates signatures created against the original OCI image.

SOCI's answer is to leave the image alone and place a separate index artifact (the "SOCI index") next to it in the registry, queried at container launch through the OCI reference types mechanism.

A note: the project's own documentation says this applies to the first version, that adding or removing an index on a widely deployed image can cause performance changes in production scenarios, and that the second version went back to a build-time conversion step. So "lazy loading without conversion" produced its own price in practice.

The price: which guarantee, in which stack?

Now the main point — and here you must separate two stacks, because the same words describe different things.

On the containerd + stargz snapshotter path, "off" isn't a security setting but an installation state: without the snapshotter installed and configured, there's no lazy pulling. Content verification, as described above, is built into the format.

On the Podman/containers-storage path, it's a configuration matter. The enable_partial_images option enables partial pulls, and its default in the code is false:

{&res.enablePartialImages, "enable_partial_images", false},

A little documentation archaeology is needed here: the project's zstd:chunked design document says that "at the time of this writing, support for this is enabled by default in the code", and the document is dated August 2024. That sentence was true then; the default was switched on in February 2024 and switched back off in November 2024. So the documents don't contradict each other — one is stale. On your own system, check the behaviour rather than the commented-out line in storage.conf.

The genuinely interesting constraint lives in the option whose name says it all: insecure_allow_unpredictable_image_contents. The documentation's first sentence is that it "should almost never be set". What it does is allow partial pulls without guaranteeing that partial and non-partial pulls produce the same image contents. The side effect that rarely gets quoted is more concrete: with it enabled, image IDs become unpredictable — usually not equal to the traditional value matching the config digest.

So when is the flag needed? The source code answers clearly: zstd:chunked layers without tar-split data fall back to the ordinary download path because consistency with non-partial pulls can't be guaranteed — unless the flag is set. eStargz layers likewise use the traditional path on the Podman side without it. Today's tar-split-carrying zstd:chunked images are pulled partially with the guarantee intact; the flag is for the older format and for eStargz.

My takeaway: what's sacrificed here isn't content verification but predictability — the guarantee that partial and full pulls produce the same storage representation and the same image ID. It looks like a performance setting, but the consequences land on the supply chain side.

If you handle image security with a CI gate like the one in my article on container image scanning with Trivy, you have to ask which identity that gate scanned: is the image ID you scanned the one you run? With signatures the question is sharper still, because format conversion invalidates the signature.

On the Podman side there's an official way out of this dilemma too: the convert_images option converts images to zstd:chunked while pulling. The documentation notes it's off by default because it's an expensive operation — but you get to choose between "forcing the old format with a flag" and "converting locally".

Where does it actually pay off?

The size of the gain depends on a single ratio: how much of the image is really read? The smaller that ratio, the bigger the difference lazy pulling makes.

The places where the ratio is typically small are predictable: huge machine-learning images (CUDA layers, model weights), application images carrying a full language runtime, "just in case" debug images stuffed with tools. By contrast, a carefully built 20 MB distroless image holding a Go binary has little to gain — everything downloaded is read anyway.

On the scenario side, three stand out: workloads that scale to zero (cold start latency lands directly on the user), CI runners (the same enormous image on every job), and Kubernetes clusters with many nodes (the same image pulled hundreds of times).

When you do the maths, remember: lazy pulling doesn't remove the download, it spreads it over time. When the application takes its first request, missing files may still be arriving; on a slow network the latency of those first seconds comes not from the download but from on-demand reads. Measure to the moment "it served the first request at normal latency", not "the container started".

Growth on disk is a separate matter; I wrote about how layers behave on write in the OverlayFS copy_up article. Lazy pulling changes the read side; the write-side mechanics stay the same.

Checklist before you try it

Identify your stack: containerd (including Docker v24+ in containerd-snapshotter mode) with a stargz snapshotter is one path, Podman/containers-storage another. On CRI-O and Podman you need a separate additional layer store (Stargz Store) rather than the stargz snapshotter.
enable_partial_images is off by default today; verify that from behaviour, not from the documents.
Before switching on insecure_allow_unpredictable_image_contents, answer this: does every image that could ever reach this machine come from a build system you trust not to attack image integrity?
Move image conversion into CI; an eStargz image built with BuildKit v0.10+ also runs on classic runtimes, so the migration can be gradual.
Confirm your registry supports HTTP range requests; all three approaches rest on them.
Watch the snapshotter cache: in an on-demand read setup, cache or daemon problems can surface as file read errors while the container still looks healthy. Test your readiness probe with a real file read.
On the storage side, check use_hard_links and reflink support; that's where file-level sharing pays.

Pointing instead of copying

Lazy pulling reminded me what container images actually are: copying a filesystem snapshot over the network. The copy model is simple and durable, but as you scale, the assumption "move everything" gets expensive.

The alternative model is to point rather than copy: share a content-addressable table and fetch a chunk when it's needed. The same idea already lives in git, in registries, and in content-addressable storage; lazy pulling carries it into the runtime.

So the question for your own setup: how much of the image you pull most often actually gets executed? If the answer is "less than half", these mechanisms solve your problem — as long as you hand over the guarantee knowingly.

Official Sources

Page Reclaim Changed Quietly: MGLRU

Mustafa ERBAY — Mon, 07 Sep 2026 07:36:17 +0000

When memory starts filling up on your server, the kernel makes a decision: which page gets thrown away, which one stays? For years that decision was made with two lists — "active" and "inactive" — and most of us never argued with it. It worked well enough.

That algorithm has changed. And on your server it's most likely already on.

Here's the state on mine:

$ cat /sys/kernel/mm/lru_gen/enabled
0x0007

$ grep CONFIG_LRU_GEN /boot/config-$(uname -r)
CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y
# CONFIG_LRU_GEN_STATS is not set
CONFIG_LRU_GEN_WALKS_MMU=y

0x0007 means the multi-generational LRU (MGLRU) is active with all of its components. I didn't make that decision; the distribution did, it arrived with a kernel package, and no release note put it on my agenda.

Don't generalise that into "it's on everywhere", though: Amazon Linux 2023's official documentation states that CONFIG_LRU_GEN is compiled in but not enabled by default, and that you enable it through the same sysfs file. So the same application may be running on two cloud servers with different reclaim algorithms. Read it rather than assume it.

If it's off, there's no kernel command line parameter to make it permanent either; you need a tmpfiles.d rule or a unit that writes to sysfs at boot.

The most important changes in systems software are usually the quiet ones. Knowing MGLRU means updating your assumptions about how your server behaves under memory pressure.

The old model and the new one

In classic LRU, pages live in two lists: active and inactive. A page that gets accessed is promoted to active, an unused one falls to inactive, and under pressure pages are dropped from the tail of the inactive list. Simple and legible.

MGLRU does the same job with generations. Pages sit in numbered generations ordered by access recency; younger generations hold hot pages, older ones hold cold pages. The kernel moves pages between generations through "aging" and drops them from the oldest generation through "eviction".

The documentation's opening line sums up why this matters: page reclaim decides the kernel's caching policy and its ability to overcommit memory, and it directly impacts kswapd CPU usage and RAM efficiency.

Three bits in the `enabled` file

The kill switch, enabled, is a bitmask that turns three components on and off separately.

0x0001 is the main switch: multi-gen LRU itself. 0x0002 clears the accessed bit — the one the MMU sets — in leaf page table entries in large batches; the documentation warns this can theoretically worsen lock contention (mmap_lock), and that disabling it costs a minor performance degradation for workloads that contiguously map hot pages. 0x0004 does the same clearing for non-leaf entries; the docs note this behaviour wasn't verified on x86 varieties other than Intel and AMD, and call the cost of disabling it negligible.

Writing y or n applies to all components at once. The 0x0007 in my output means all three are on.

The fourth line in my output (CONFIG_LRU_GEN_WALKS_MMU=y) explains why those two bits exist: it isn't a user setting but a dependency enabled automatically on architectures where the hardware sets the accessed bit itself. These are the knobs that manage the cost of walking page tables.

One detail: if the hardware doesn't support a component, writing has no effect — and valid values are accepted even when the main switch is off. So don't write to the file and assume; read it back.

Thrashing prevention: `min_ttl_ms`

MGLRU has a little-known knob aimed directly at the "the system freezes" complaint.

Write N to min_ttl_ms and the working set of the last N milliseconds is protected from eviction. If that working set can't be kept in memory, the OOM killer fires. In the documentation's own words this works as an adjustable pressure relief valve which, when open, terminates applications that are hopefully not being used.

The docs even give numbers: since the average human-detectable lag is ~100 ms, N=1000 usually eliminates intolerable janks; larger values like N=3000 make janks less noticeable at the risk of premature OOM kills. The default 0 means disabled.

It's 0 on my server too — and that's the right place for it. This knob was added with desktop and laptop users in mind, the ones who don't have a userspace policy like oomd. On the server side, the better way to manage memory pressure is a pressure-based policy like the one in my systemd-oomd cgroup v2 memory pressure runbook. MGLRU's valve doesn't replace that policy.

Working set estimation

MGLRU's most valuable part on a server is the debugfs interface most people never open.

Reading /sys/kernel/debug/lru_gen returns a histogram of the number of pages accessed over different time intervals, per memcg and node. The format is documented: the leading number is the generation number, the second column is the age in milliseconds, and the next two are the estimated anon and file pages in that generation. The lowest-numbered generation holds the coldest pages and the highest-numbered one the hottest, because the age ordering runs the other way. The histograms are noncumulative.

Let me admit the first trap I fell into: looking at the memcg 1 / line at the top of the file and drawing conclusions about the machine. The root cgroup comes back nearly empty, because on a systemd system almost every page is charged to a child cgroup. The meaningful table is below, from real services:

# grep -A2 "actions.runner\|containerd" /sys/kernel/debug/lru_gen
memcg    53 /system.slice/actions.runner.merbay-erp-kopru.kopru-vps3.service
          1   23283281          0      378128
          3   23283281      17418       65909
memcg    57 /system.slice/containerd.service
          1   23283243          0       29931
          3   23283243      73949       12960

These values are in pages, so multiply by 4 KB. The runner service holds ~378k file pages in its cold generation — roughly 1.5 GB of page cache — while its hot generation holds ~65k file and ~17k anon pages. On the containerd side the table inverts: ~74k anon pages in the hot generation, about 290 MB of genuinely working memory.

The second trap is subtler and visible in my own output: the age values within each memcg are identical. The kernel source explains why — when an lruvec is first initialised, all generation timestamps are set to the same moment. So aging hasn't advanced at all on this machine; the bins aren't "different time intervals", they only show the current distribution. A real working set measurement needs aging to be triggered first.

You can also trigger aging by hand, which is the cure for the gap above: the command + memcg_id node_id max_gen_nr [can_swap [force_scan]] creates a new generation. A scheduler can run it at intervals and rank its servers by how many cold pages that interval reveals. If you're measuring, the flow is: create a generation, wait, read the histogram — a single read doesn't give you a working set.

The number of generations is fixed too: the kernel keeps at most four, and the two youngest count as "not fully aged". The documentation maps those two onto the classic LRU's active list, so there's a direct equivalent for anyone who knows the old model.

The same interface can do proactive reclaim. This command evicts generations less than or equal to the given generation number:

- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]

The details matter: min_gen_nr must be less than max_gen_nr-1, because the two youngest generations aren't fully aged and therefore can't be evicted. Here swappiness overrides the value in /proc/sys/vm/swappiness, with a valid range of 0-200 plus max, which is used exclusively for reclaiming anonymous memory. nr_to_reclaim caps how many pages get evicted.

This is the official way to say "clear cold pages while there's no memory pressure" — tailor-made for a scheduler preparing a server before placing a new job, and, as the docs stress, with limited impact on the jobs already running.

A note: this interface sits under "experimental features" and needs root access to debugfs. If you're going to wire production automation to it, do so accepting that behaviour may change across kernel upgrades.

If all you want is proactive reclaim, there's already a stable, non-experimental interface: cgroup v2's memory.reclaim. It triggers reclaim in the target cgroup and accepts a swappiness nested key:

echo "1G" > /sys/fs/cgroup/system.slice/backup.service/memory.reclaim

Note the two caveats from the docs: the kernel can over- or under-reclaim (under-reclaim returns -EAGAIN), and reclaim triggered this way is not meant to indicate memory pressure — so the networking layer's socket memory balancing isn't exercised. For cold-page cleanup in production, start here; MGLRU's debugfs interface is for measuring the working set.

Working set per container

Note the first line of that output: memcg 1 /. The histogram is per cgroup, meaning every container on the machine gets its own table. That makes it possible to separate two questions people routinely conflate in capacity planning.

First question: "how much memory has this container allocated?" memory.current answers it, and it usually looks larger than it should, because page cache counts there too. Second question: "how many pages did this container actually touch in the last N milliseconds?" Only the generation histogram answers that.

When we set limits we look at the first number and try to guess the second; MGLRU makes the second directly measurable. On a dense container host, revisiting memory.max values against this table is far cheaper than the habit of "give it twice as much, just in case".

One caveat: the table is a snapshot and aging runs continuously. Don't decide from a single read; look during both busy and quiet hours.

Where MGLRU stands in 2026

Ending this with "it settled in quietly, case closed" would be wrong, because the upstream discussion is open.

MGLRU was merged in 6.1, but progress stalled and it still isn't enabled on many systems. In the 2026 memory-management discussions, some developers looked for ways to improve it while another argued for removing it entirely — the objection being less technical than about maintenance: that the feature wasn't owned and that the assigned maintainers weren't contributing. In the same period the LSFMM+BPF summit devoted three separate sessions to it, covering both better integration and problems seen on Android.

The operational translation: classic LRU wasn't removed, MGLRU didn't become mandatory, both implementations sit side by side, and distributions have landed on different sides. This isn't a "migration to the new standard" story; it's still a choice.

So what did I gain?

The honest answer: nothing measurable changed on my own server, because there's no memory pressure there. While writing this, the machine was using 11 GB of its 96 GB and the averages in /proc/pressure/memory were zero.

MGLRU's difference shows up under pressure. The generation-based model aims to make scanning cheaper and to protect hot pages better; on paper the gain lands in the CPU kswapd burns and in the amount of work you can carry on the same RAM.

It isn't free, though: aging walks page tables, and the two bits in the enabled mask exist precisely to manage that cost. The criticism that it's expensive on low-end hardware comes from the same place.

But be careful here: I can't tell you "it got X% faster", because I didn't measure it. If you want to, the right method is to put your workload under pressure and compare kswapd CPU and application latency while flipping enabled between y and n. On one machine, with one workload.

If you use compressed swap, the two work together: MGLRU decides which page gets evicted, and zram or zswap decides where the evicted page goes. One makes the selection, the other handles the storage.

Checklist

cat /sys/kernel/mm/lru_gen/enabled → is it on, and which components are active?
grep CONFIG_LRU_GEN /boot/config-$(uname -r) → what's your distribution's default, is CONFIG_LRU_GEN_ENABLED set?
When you hit trouble, turn the main switch off and compare: echo n > /sys/kernel/mm/lru_gen/enabled. Reversible, but not free: during the switch the kernel moves pages across every cgroup between two list layouts. On a large-memory machine with hundreds of cgroups, do it in a maintenance window rather than under pressure.
On desktop/laptop setups with jank complaints, try min_ttl_ms starting at 1000; on servers, set up a pressure-based policy first.
If you're curious about working sets, look at the debugfs histogram — but remember the "experimental" label before wiring it into production automation.
Additional statistics live in /sys/kernel/debug/lru_gen_full; CONFIG_LRU_GEN_STATS only keeps historical stats from evicted generations there, at a memory cost. On a kernel like mine, where it's off, that history isn't available.
When measuring, look at pgscan_kswapd, pgsteal_kswapd and workingset_refault_* in /proc/vmstat — but account for these metrics behaving differently under MGLRU than under classic LRU; your dashboards may not be calibrated for it.

Defaults change quietly

More than a technical topic, MGLRU reminded me of a habit: the things we trust most in infrastructure are the things we question least. Page reclaim sat inside the "the kernel handles it" box for years; one day the algorithm changed, and so did the contents of the box.

I'm not saying that's bad — quite the opposite, distributions manage these transitions well. But knowing when the mental model you use to explain a server's behaviour was last updated makes a difference during diagnosis. When you say "under memory pressure it behaves like this", you should know which kernel's algorithm you're talking about.

So the question for your own setup is simple: which of your kernel's defaults changed in the past year, and how many of them did you notice?

Official Sources

Setting Up Immich: The Operational Cost of Google Photos Convenience

Mustafa ERBAY — Sun, 06 Sep 2026 09:34:52 +0000

What Is Immich and What Are Its System Requirements?

Immich is a self-hosted photo and video management platform that runs components such as PostgreSQL, Redis, and Nginx inside Docker containers. For a baseline setup, a minimum of 4 CPU cores, 8 GiB RAM, and a 100 GiB SSD is recommended. The following command verifies the kernel version on Ubuntu 22.04 LTS:

uname -r

5.15.0-1049-azure

This output confirms that the Linux kernel belongs to the 5.15 branch and smoothly supports Docker's kernel-level features (cgroups v2, overlay2). Once these prerequisites are met, running Immich's components isolated within separate containers provides significant advantages in terms of both security and scalability.

Deployment with Docker Compose (Sample Scenario)

The docker-compose.yml file below pulls official images from Docker Hub. The configuration includes PostgreSQL 15, Redis 7, and Nginx-proxy-manager 2.10.

version: "3.9"
services:
  server:
    image: ghcr.io/immich-app/immich-server:latest
    depends_on:
      - redis
      - postgres
    environment:
      - DB_HOST=postgres
      - REDIS_HOST=redis
    ports:
      - "3001:3001"
  redis:
    image: redis:7-alpine
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD=immich
      POSTGRES_DB=immich
    volumes:
      - pgdata:/var/lib/postgresql/data
volumes:
  pgdata:

Verifying container status after startup:

docker compose ps --format "table {{.Service}}\t{{.State}}"

SERVICE   STATE
server    running
redis     running
postgres  running

Seeing all services in the running state confirms that core dependencies have booted properly. The docker compose logs -f server command generates an entry confirming a successful database connection during initial boot:

2024-10-12 08:15:23.123 INFO  [server] Connected to PostgreSQL at postgres:5432

Storage and Network Cost Analysis

Immich stores media files directly on an attached disk (e.g., /mnt/immich-data). To inspect disk usage:

df -h /mnt/immich-data

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       200G   45G  145G  24% /mnt/immich-data

This output shows 24% utilization, running 45 GiB of photo/video data on a 200 GiB SSD. Network throughput can be monitored with iftop; here is a sample one-minute capture:

Total send rate: 12.3 Mb/s   Total receive rate: 9.8 Mb/s

These numbers indicate ample bandwidth for an average family's photo-streaming workload. Google Photos offers a free tier with a fixed storage quota; Immich's SSD expense, on the other hand, depends on the regional pricing model of your chosen infrastructure provider.

Performance Monitoring and Defining SLOs

Immich collects metrics using a Prometheus exporter. Running docker compose exec server curl -s http://localhost:3001/metrics | grep http_requests_total reveals the total count of HTTP requests:

http_requests_total{method="GET",handler="/api/assets"} 124578

This volume points to an average of roughly 124k requests per day, highlighting that our SLO should aim to keep 99.9% of request response times under 200 ms. A Grafana dashboard can be attached for real-time observability; for instance, auto-scaling could be triggered if CPU consumption exceeds 65%.

Rollback and Update Strategy

Upgrading Immich versions is handled by updating the image tag in your Docker Compose file. Always take a fresh database dump before updating:

docker compose exec postgres pg_dump -U postgres -d immich > immich_backup_$(date +%F).sql

This command stamps the backup file with today's date. If an issue surfaces following the upgrade, rolling back is straightforward:

docker compose down
docker compose up -d
docker compose exec -T postgres psql -U postgres -d immich < immich_backup_$(date +%F).sql

The rollback sequence restores the previous release without data loss. Command execution order is critical here: stop the database service first, bring up the targeted version, and restore the snapshot.

Cost Comparison with Google Photos

Beyond its initial 15 GB free allowance, Google Photos offers a 100 GB tier at $1.99/month. Hosting an equivalent capacity on an SSD-backed VPS running Immich—for example, a 2 vCPU, 8 GiB RAM instance—runs around $12 (Azure B2s) + $20 (200 GiB SSD) = $32/month. The trade-off is clear: Immich delivers total data privacy and full customization, whereas Google Photos involves corporate data processing and potential ad-targeting implications.

The Mermaid diagram below illustrates the architectural differences between the two approaches:

Data Backup Strategy

A disciplined backup regimen is essential for data integrity and continuity. The first line of defense is dumping the PostgreSQL database on a daily or weekly schedule using pg_dump:

docker compose exec postgres pg_dump -U postgres -d immich > /backup/immich_$(date +%F).sql

For media files, archiving them with tar and shipping them off to secondary storage is recommended:

tar czf /backup/immich_media_$(date +%F).tar.gz /mnt/immich-data

Transferring these backups to a secure remote destination (such as AWS S3, Azure Blob Storage, or another off-site cloud store) shields your data from local hardware failures. You can automate this process by adding an entry to crontab:

0 2 * * * /usr/local/bin/immich-backup.sh >> /var/log/immich/backup.log 2>&1

Testing your restore pipeline is the single most reliable way to prevent catastrophic data loss. Restoring the database backup looks like this:

docker compose exec -T postgres psql -U postgres -d immich < /backup/immich_2024-09-06.sql

Shape your backup lifecycle around your recovery time and recovery point objectives: daily backups for critical assets, weekly snapshots for less dynamic content. Pruning or archiving historical snapshots at the end of every backup cycle keeps long-term storage bills predictable.

Security and Compliance Controls

Shielding your self-hosted setup from external threats is vital for both privacy and regulatory compliance. The first step is enforcing end-to-end TLS encryption across all inbound traffic. You can issue a free Let's Encrypt certificate using certbot:

certbot certonly --webroot -w /var/www/html -d immich.example.com

Configure the nginx reverse proxy in your Docker Compose file to consume this certificate:

  nginx:
    image: nginx:alpine
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt:ro
      - /var/www/html:/usr/share/nginx/html
    ports:
      - "80:80"
      - "443:443"

In addition, isolate inter-container communication so services only touch networks on necessary ports. For example, the server container only needs access to postgres and redis:

  server:
    networks:
      - immich_net
    depends_on:
      - postgres
      - redis

For identity management, integrating OAuth2 or LDAP allows you to manage user access through a centralized provider. To protect data at rest, the underlying storage volume holding your media assets can be secured with a LUKS encryption layer:

cryptsetup luksFormat /dev/sda1
cryptsetup open /dev/sda1 immich-data

Comprehensive audit trails and system monitoring should also be configured. Tools like auditd help trace unauthorized system calls and catch modifications to sensitive configuration files.

Operational Cost Breakdown

A detailed breakdown of ongoing operational costs comes down to three pillars: compute resources, storage capacity, and bandwidth egress. In a standard VPS environment, monthly compute expenses for a 2 vCPU / 8 GiB RAM box are determined by your provider's hourly rate:

Compute Cost = (Hourly Rate) × 24 × 30

Storage expenses track directly with the volume of allocated SSD or HDD capacity:

Storage Cost = (GB per Month) × (Price per GB per Month)

Data transfer costs are billed based on outbound egress traffic:

Bandwidth Cost = (GB Outbound) × (Price per GB)

Secondary storage fees for backups will reflect the storage tier you pick. Storing a 200 GiB compressed backup snapshot each month scales linearly with that provider's per-gigabyte pricing. Tallying up each line item gives you the actual monthly operating bill.

Several tactics can help optimize these numbers: commit to reserved instances or leverage spot capacity for predictable workloads to shrink compute overhead. Auto-scaling lets your setup expand during heavy sync periods and dial back during quiet hours. Finally, placing a CDN in front of assets and enabling media compression will trim outbound network bills while boosting load times for end users. Combining these approaches keeps Immich financially viable and performant over the long haul.

Conclusion

Spinning up Immich via Docker Compose delivers a polished, open-source alternative for managing personal media libraries, though infrastructure and storage costs will easily outpace Google Photos' entry-level pricing. Establishing active monitoring, setting strict SLOs, and defining safe rollback paths are what keep operational overhead manageable over time. If your primary driver is rock-bottom price, Google Photos wins on convenience. If total data sovereignty, uncompromised privacy, and flexibility take precedence, hosting Immich on your own infrastructure is the way to go. A logical next step: wire up a production Prometheus-Grafana stack to generate real-time performance metrics and cost projections.

Official Sources

fanotify: Watching File Access — and Stopping It

Mustafa ERBAY — Sun, 06 Sep 2026 09:32:49 +0000

The classic Linux answer to "what's changing in that directory?" is inotify. It works beautifully on small trees; the problem is that watching is set up per directory. When you want to watch a storage area with hundreds of thousands of directories, you need a watch for each one — and at some point you hit the limit.

Here's the limit on my own server:

$ sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
fs.inotify.max_user_watches = 753024
fs.inotify.max_user_instances = 128

Seven hundred thousand sounds like plenty, but not for someone running a file-sharing server — and long before you reach that number, walking the tree to mark every directory becomes a job that takes minutes on its own.

fanotify asks a different question: instead of marking directories one by one, what if we watched an entire mount point or filesystem? And it does one more thing — it can not only see access but stop it.

The thesis of this article: what makes fanotify interesting isn't the scale, it's that second capability. And that capability carries a serious responsibility: if your listener doesn't answer, the process trying to open the file waits.

Two APIs, two mindsets

With inotify the model is simple: you give a path, you get events about that path. With fanotify a mark has three scopes, and the choice shapes your architecture: individual files and directories, a mount point with FAN_MARK_MOUNT, or an entire filesystem with FAN_MARK_FILESYSTEM.

The privilege side deserves precision, because most write-ups are sloppy here. fanotify isn't entirely a root affair: since 5.13 the FAN_CLASS_NOTIF class can be opened by an unprivileged user, provided one of the file-handle reporting flags is set. The comment in the kernel source explains why — unprivileged groups don't get file descriptors in events, so handle reporting is mandatory.

But here's the crux: the wide-scope marking that sells this API isn't available unprivileged. Both FAN_MARK_MOUNT and FAN_MARK_FILESYSTEM require CAP_SYS_ADMIN; an unprivileged group can only place individual inode marks, sees no pid or file descriptor in events, and can't use the unlimited-queue or unlimited-marks flags. So "one mark covers the whole disk" isn't free — it comes with root.

That scope difference is the difference between "I'll watch 10 directories" and "I'll see what happens on this disk". The second is theoretically possible with inotify but means walking the tree and opening hundreds of thousands of watches; with fanotify it's a single call.

The resource limits became tunable in 5.13. The values on my server:

$ sysctl fs.fanotify.max_queued_events fs.fanotify.max_user_groups fs.fanotify.max_user_marks
fs.fanotify.max_queued_events = 16384
fs.fanotify.max_user_groups = 128
fs.fanotify.max_user_marks = 800683

These three cap the number of queued events, the number of groups per user, and the number of marks per user respectively. Before 5.13 they were hardcoded constants: 16384 events, 128 groups per user, and 8192 marks per group.

You may have noticed something: fanotify's mark limit (800683) is only 6% above inotify's watch limit (753024). The gain isn't "more marks you can place"; the gain is that a single mark covers an entire tree.

The real difference: the power to permit

When you initialise an fanotify group you choose a "notification class", and that choice determines what you can do.

FAN_CLASS_NOTIF is the default: you're told that a file was accessed, but you can't decide before the access happens. That's the class for accounting and auditing.

FAN_CLASS_CONTENT and FAN_CLASS_PRE_CONTENT unlock permission events. The man page even names the use cases: the first is for listeners that need to access files once they already contain their final content — malware detection programs are the example given; the second is for listeners that may need to write data before the final data can be accessed — hierarchical storage managers are the example. The two permission classes require CAP_SYS_ADMIN; FAN_CLASS_NOTIF can be opened unprivileged under the conditions described above.

With a permission class you receive FAN_OPEN_PERM, FAN_ACCESS_PERM and FAN_OPEN_EXEC_PERM (open with intent to execute) events, and for each one you must write a response: FAN_ALLOW grants, FAN_DENY refuses. A denied call receives EPERM. Since Linux 6.13, groups initialised with FAN_CLASS_PRE_CONTENT can deny with a different error using FAN_DENY_ERRNO() — you can make the application see EIO, for instance. It isn't an arbitrary error code though; the man page lists the permitted set: EPERM, EIO, EBUSY, ETXTBSY, EAGAIN, ENOSPC, EDQUOT.

The last two boxes are what anyone considering fanotify in production needs to digest first — and this is where intuition fails.

If your listener is alive but not responding (a hung thread, an unread queue), processes trying to reach the file wait; those waiting tasks are killable, so SIGKILL gets you out. But if the listener dies, the picture inverts: the man page is explicit — upon close, outstanding permission events are set to allowed. The kernel source does the same thing; as the group is released, every pending permission event is finished with FAN_ALLOW.

So the real risk isn't a wedged system, it's your security gate opening silently. When a listener doing malware scanning crashes, everything looks normal; there's simply nobody at the door any more. There's an interesting asymmetry too: when memory can't be allocated the kernel denies the permission event, failing closed in that case.

Linux 6.18 shed a little light on this blind spot: with fs.fanotify.watchdog_timeout, if a permission event goes unanswered for the configured period the kernel logs a warning naming the PID that failed to respond to the queue. It doesn't resolve the event, but it does make "the doorkeeper fell asleep" visible.

Lost events and an overflowing queue

The most dangerous state for a monitoring system is silently missing events while believing it sees everything. fanotify is honest here: when the queue limit is exceeded a FAN_Q_OVERFLOW event is generated. There's one exception, and it concerns exactly the permission class: overflow events aren't queued for permission events — there, access is denied outright. The FAN_UNLIMITED_QUEUE flag removes the limit — and again requires CAP_SYS_ADMIN.

One more detail: events can be merged. Consecutive events for the same filesystem object originating from the same process may be merged into a single event. The one exception is permission events — two of those are never merged into one queue entry.

The analytical consequence is clear: fanotify doesn't give you a "how many times was it read" counter, it gives you "it was read". If you're counting, account for merging.

Choosing between descriptor and handle

In classic usage each event carries an open file descriptor, and closing it is the reading application's responsibility. In a high-volume system that's the easiest path to descriptor leaks.

The alternative is initialising the group with flags like FAN_REPORT_FID or FAN_REPORT_DIR_FID. But know this first: the trade-off only exists in the notification class. The kernel rejects fid mode together with any class other than FAN_CLASS_NOTIF with EINVAL. In a permission class there is no choice — every event carries a descriptor and you must close it. For anyone building a permission gate, descriptor-leak risk isn't a preference, it's a given. Objects are then identified by file handles instead of descriptors, and a separate information record arrives with the event. These flags stack; with FAN_REPORT_TARGET_FID and FAN_REPORT_PIDFD together, one event can carry two fid records and a pidfd record. The man page's warning matters: with such a stacked configuration there is no guarantee about the ordering of information records.

If you want to turn a handle into a path, open_by_handle_at() comes into play — the answer to "which file was it" isn't free, it's an extra call. In exchange you don't open and close a descriptor for every event; at high volume that trade usually favours handles.

In practice: use a descriptor if you're going to read the file's contents, and a handle if you only need to know which object it was.

The things it doesn't see

The man page's "limitations and caveats" section is short but it shapes production decisions.

fanotify reports only events that a user-space program triggers through the filesystem API, so it doesn't catch remote events on network filesystems. A change made by another client on a directory you share over NFS never reaches your listener.

Accesses and modifications happening through mmap(), msync() and munmap() are not reported either. If you're watching an application that works with memory-mapped files, the picture you see is incomplete.

Events for directories are created only if the directory itself is opened, read and closed; adding, removing or changing children of a marked directory doesn't create such an event. To see changes in directory contents you have to request separate event types like FAN_CREATE and FAN_DELETE. There's a scope trap here as well: these handle-identified events can't be supplied in the mask together with FAN_MARK_MOUNT — the attempt returns EINVAL. So the plan of "mark the mount point and watch what gets created on the file server" doesn't work directly.

There's also a security note that bears directly on the permission class: the kernel doesn't check whether the receiving process is authorised to read or write the file before passing it a descriptor. When CAP_SYS_ADMIN is granted to programs run by unprivileged users, that's a risk.

Locking yourself out

There's a trap every listener author in a permission class eventually hits, and it's easy to miss in the man page: the descriptor delivered with the event has FMODE_NONOTIFY set, so access through that descriptor generates no new events.

But if your listener reopens the file by path — while writing a log, copying to quarantine, or resolving a handle with open_by_handle_at() — it triggers its own permission event and blocks itself. That's the classic fatal bug in fanotify-based scanners. The rule is simple: while making the decision, use only the descriptor you were given, and keep your own open() calls away from the path you're gating.

A second detail: marks aren't retroactive. Access through descriptors opened before your listener started is invisible to you. Service ordering is therefore a security matter.

The state on my server

Two kernel options decide, and both are on for me:

$ grep CONFIG_FANOTIFY /boot/config-$(uname -r)
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y

The first enables the API, the second the permission handling. On a kernel with the second one off, trying to open a FAN_CLASS_CONTENT group goes nowhere — don't start a permission-based design before verifying this on your distribution's kernel.

What changed in recent releases

fanotify isn't a static API; several recent additions touch this article's topics directly.

FAN_MARK_EVICTABLE (5.19) belongs to the scale side: it doesn't pin marked inodes in memory, lowering the memory cost on wide trees. FAN_MARK_IGNORE (6.0) is the modern replacement for the old ignore mask and the right way to narrow an event flood. FAN_FS_ERROR (5.16) surfaces filesystem errors and FAN_RENAME (5.17) renames as their own events.

6.14 advanced on two fronts: mount namespace watching (FAN_MARK_MNTNS plus mount attach/detach events) and genuine pre-content events (FAN_PRE_ACCESS) — meaning the hierarchical storage manager scenario is no longer just a theoretical example in the man page.

When fanotify, and when not?

The question for your own setup isn't "which API is more powerful" but "what am I going to do".

If you're watching a handful of configuration files to reload a service, inotify is enough; there's no need to pay fanotify's CAP_SYS_ADMIN price. If you're on a file server working out access patterns, answering "which share is actually used", or producing an audit trail, fanotify's notification class is the right tool: one mark covers an entire mount point.

If you want to block access — malware scanning, a data classification gate, "this file must not be opened before it's restored from the archive" — the permission class is the only way. But then your listener is an infrastructure component: restarts, timeouts, crash scenarios and the question "what should happen when the listener isn't there" all become part of your design.

If an audit trail is your actual goal, don't decide without comparing fanotify to auditd: auditd works at the system call level and gives richer user/process context, while fanotify works at the filesystem object level and comes with the ability to block. The rule structure in my runbook on monitoring privileged commands with Linux auditd answers "who ran what"; fanotify is strong on "which file was touched". Nor are they entirely separate worlds: open the fanotify group with FAN_ENABLE_AUDIT and set the FAN_AUDIT flag on responses, and your permission decisions land directly in the audit subsystem. You'll be shipping whatever you collect somewhere, which is a design job of its own.

One more warning: file access records are inherently sensitive data. Which user opened which document is, in most organisations, more critical than the log itself. Decide on retention and access rights before you start collecting.

Checklist before you build

Verify kernel support: CONFIG_FANOTIFY, and CONFIG_FANOTIFY_ACCESS_PERMISSIONS if you need permissions.
Pick the scope: file/directory, FAN_MARK_MOUNT or FAN_MARK_FILESYSTEM. The wrong scope means either far too many events or missing coverage.
Size fs.fanotify.max_queued_events for your workload and always handle FAN_Q_OVERFLOW — a listener that misses overflow has incomplete data.
If you use descriptors, put the close on every branch of the code; if you don't, move to the FAN_REPORT_FID family.
In a permission class, test timeouts and the "listener died" scenario; run your first production attempt on a narrow mount point, not an entire filesystem.
Document the blind spots — mmap, network filesystems, directory child events — so nobody has to ask later why something wasn't seen.

The difference between watching and deciding

What strikes me most about fanotify is that both capabilities live in the same API. Watching is passive: get it wrong and you collect incomplete data. Deciding is active: get it wrong and you stop the system.

Because they sit behind the same interface, the two get conflated easily — "we're watching anyway, let's block too" sounds so innocent. Yet the moment you start blocking, your listener becomes a component as critical as your database.

So the question for your own setup: do you really want to decide, or only to see? If it's the latter, stay in the notification class.

Official Sources

Sharing IO: io.max, io.weight and io.latency

Mustafa ERBAY — Sun, 06 Sep 2026 09:18:47 +0000

When one process is choking the disk on a machine, the first reflex is always the same: "let's set a limit". Then someone writes an io.max into a cgroup directory, the backup job is capped at 50 MB/s, and everyone relaxes. A few weeks later the backup window starts overrunning — while the disk sits idle most of the time.

The reason for that story is that cgroup v2 doesn't have a single way to share IO. The kernel documentation lists four: absolute limits (blk-throttle), weight-based proportional distribution (iocost, or BFQ's own cgroup support when the BFQ scheduler is in use), latency-based protection, and IO priority assignment. This article covers the first three, because that trio drives the decision; the fourth (io.prio.class) is worth keeping in your pocket for cheap moves like putting background jobs into the idle class.

Three mechanisms, three questions: io.max says "this group must never exceed this rate", io.weight says "when the disk is contended, split the share in this proportion", and io.latency says "keep this group's latency under this target, slowing others down if necessary".

My argument: most teams set up the first one when what they need is one of the other two. A hard cap doesn't protect the system — it only punishes a group and throws idle capacity away.

First, see what you actually have

Let's start with my own server: Ubuntu 24.04, a single SSD root disk (sda, major:minor 8:0) and cgroup v2.

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc

$ cat /sys/fs/cgroup/system.slice/io.weight
default 100

$ cat /sys/fs/cgroup/system.slice/io.stat
7:5 rbytes=20272128 wbytes=0 rios=758 wios=0 dbytes=0 dios=0
7:4 rbytes=54236160 wbytes=0 rios=1842 wios=0 dbytes=0 dios=0
7:3 rbytes=755712 wbytes=0 rios=107 wios=0 dbytes=0 dios=0
7:2 rbytes=20448256 wbytes=0 rios=693 wios=0 dbytes=0 dios=0
7:1 rbytes=1116160 wbytes=0 rios=59 wios=0 dbytes=0 dios=0
7:0 rbytes=8969216 wbytes=0 rios=896 wios=0 dbytes=0 dios=0
8:0 rbytes=835511135232 wbytes=1563887470592 rios=43331841 wios=133872437 dbytes=699905486848 dios=352412

$ cat /sys/fs/cgroup/system.slice/io.pressure
some avg10=0.08 avg60=0.05 avg300=0.10 total=7877797953
full avg10=0.00 avg60=0.00 avg300=0.04 total=6108319475

The first thing to learn from that output isn't policy, it's reading discipline: the first six lines are 7:x — loop devices, the mount points of snap packages. The real disk is at the bottom, 8:0. Taking a device number from the first line of io.stat is the easiest way to write a rule against the wrong device. systemd's documentation says the same thing separately: these settings should refer to the originating physical device, not to partitions or loopback devices.

The io.cost.qos and io.cost.model files come back empty — I can't show their content in this article because there is nothing to show.

What the picture says: the IO controller is enabled, accounting works, pressure is measured — but there is no policy at all. io.weight sits at the default 100.

The gap has more than one cause. Per the kernel documentation, io.cost.qos and io.cost.model exist only on the root cgroup, and the line for a device is populated on the first write to one of those files. But that write isn't always yours: since systemd 254, udev's iocost tool fills in io.cost.* at boot on block devices for which it finds a matching solution in hwdb. My virtual disk has no match — udevadm info /dev/sda | grep -i IOCOST comes back empty — which is why the files are empty. Don't be surprised if yours are populated; it may have configured itself.

Three mechanisms, three questions

io.max: the hard ceiling

(The io.low you'll see in older guides is gone; that blk-throttle experiment was removed from the kernel.)

io.max is a nested file keyed by device number, taking rbps, wbps, riops and wiops. IOs are delayed once the limit is reached, and temporary bursts are allowed. For example, capping device 8:0 at 2 MB/s reads and 120 write IOPS:

echo "8:0 rbps=2097152 wiops=120" > /sys/fs/cgroup/backup.slice/io.max

The critical property is stated plainly in systemd's own documentation: this limit is not work-conserving, and the processes are not allowed to use more even if the device has idle capacity. So at 3 a.m., with the disk completely idle, your backup still crawls at 50 MB/s.

There's also persistence: a value you write into a cgroup file by hand can be overwritten by systemd on the next daemon-reload or when the unit restarts. The durable route is systemd's own directives — IOReadBandwidthMax=, IOWriteBandwidthMax=, IOReadIOPSMax=, IOWriteIOPSMax=:

systemctl set-property backup.slice IOWriteBandwidthMax="/dev/sda 50M"

That makes io.max a safety valve rather than a performance tool: it's right where predictability matters more than speed, when you want to say "this workload must never exceed this, whatever happens". Per-tenant ceilings in a rented environment, for instance.

io.weight: fairness, but only with a cost model

What's usually wanted is this: as long as the disk isn't contended everyone runs free, and when it is, the share is split by weights. That's io.weight, and the engine behind it is iocost.

There's a setup step here that gets missed: for weights to mean anything you need one of two things. Either the device's scheduler is BFQ — in which case BFQ's own cgroup support handles the distribution — or iocost is enabled. On my server the scheduler is none:

$ cat /sys/block/sda/queue/scheduler
[none] mq-deadline

So there's no BFQ here; if I want weights, I have to enable iocost. The controller is disabled by default and enabled by setting enable to 1.

# turn on weight-based control for device 8:0
echo "8:0 enable=1 ctrl=auto" > /sys/fs/cgroup/io.cost.qos

ctrl=auto means the parameters are managed by the kernel. And when rpct and wpct are left at zero, the controller uses the device's internal saturation state to adjust the overall IO rate between the min and max percentages. If you need device-specific coefficients, tools/cgroup/iocost_coef_gen.py in the kernel tree exists for that. When you need better control quality you can specify latency QoS parameters by hand — the documentation's example reads enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=..., i.e. scaling against a target like "the 95th percentile of reads shouldn't exceed 75 ms".

The systemd equivalent is IOWeight=: it takes a value between 1 and 10000 and controls the io.weight attribute directly (which defaults to 100). The share is proportional: a unit's share is its weight divided by the sum of weights within the same slice. So IOWeight=500 means five times only if every neighbour sits at the default 100; raise the neighbours to 500 too and nothing changes.

io.latency: protection, not punishment

The third one is the least known: the most direct tool when you want to protect a single service. io.latency defines a target latency and is work conserving — as long as everybody meets their target the controller does nothing. The syntax resembles the others:

echo "8:0 target=25000" > /sys/fs/cgroup/database.slice/io.latency   # 25 ms

Once a group starts missing its target, its peers get throttled. Two details here are the key to setting it up correctly.

First: throttling is applied only at the peer level. In the documentation's example, A, B and C under the root influence each other; D and F under A influence each other; groups in different branches never touch. Translated to systemd: two services under system.slice are peers, system.slice and user.slice are peers at the root, and two services in different slices don't affect each other at all. Put the target at the wrong level and nothing happens — and nothing errors either.

Second, and less known: the throttled peers don't need targets of their own. The comment in the kernel source is explicit — groups throttled as collateral have min_lat_nsec == 0, i.e. groups where io.latency was never set. In practice: you protect one service, and its target-less neighbours back off.

The throttling takes two forms: queue depth throttling (the number of outstanding IOs a group may have, clamped down all the way to one IO at a time) and artificial delay induction. The second is interesting, because IO types that can't be throttled — swapping and metadata IO — are allowed to proceed normally but are "charged" to the originating group.

For picking a target the guidance is clear: don't set a value lower than the latency your device supports. On rotational devices use the avg_lat value in io.stat as a basis and set it 10-15% higher; on non-rotational devices no average latency is reported, so set the target from your device and verify with the missed/total fields. But run this line first, or you'll never see those fields:

echo 1 > /sys/module/blk_cgroup/parameters/blkcg_debug_stats

Those debug stats are disabled by default; when latency fields are missing from io.stat, the usual reason isn't a broken setting but a disabled measurement.

The systemd equivalent is IODeviceLatencyTargetSec=, which takes a device and a timespan, e.g. /dev/sda 25ms.

A footnote: the introductory sentence in the kernel documentation says it throttles peers with a lower target, while its own "How IO Latency Throttling Works" section says higher. The source code supports the second; if you trip over that contradiction while reading, that's why.

The layer that's easy to skip: writeback

Everything above assumes IO can actually be charged to the right cgroup. With buffered writes that assumption collapses easily.

Page cache is dirtied through buffered writes and shared mmaps, then written asynchronously by the writeback mechanism. The documentation treats this layer separately: the io controller, together with the memory controller, implements control of page cache writeback IOs — the memory controller defines the memory domain for which the dirty ratio is maintained, and the io controller defines the IO domain that writes those pages out. Both system-wide and per-cgroup dirty memory states are examined and the more restrictive of the two is enforced.

And the critical sentence: cgroup writeback requires explicit support from the underlying filesystem. The documentation lists the supported ones — ext2, ext4, btrfs, f2fs and xfs.

In practice that means: if your filesystem isn't on that list, or the memory controller isn't enabled on the relevant hierarchy, buffered writes reach the disk with unclear ownership and your IO policy is largely ineffective on the write side. A good share of "I set a limit and the disk still fills up" complaints come from exactly here.

Which one, when?

Three questions, in order:

Is this workload's ceiling defined by a contract? A per-tenant quota, IOPS sold to a customer, a "backups must never exceed this" policy — if yes, io.max. Set it knowing it can't use idle capacity.

Is your problem the split under contention? Several services sharing a disk and the sentence "the database matters more than the log shipper" — if yes, io.weight (IOWeight= on the systemd side). But turn iocost on first, or the weights are decoration.

Is there a specific latency you want to protect? A user-facing service, a p99 target, "this service stays under 25 ms whatever the background jobs do" — if yes, io.latency.

Let's be precise about combinations: io.weight and io.latency are two answers to the same question, and stacking them is pointless. Combining io.max with either of them, on the other hand, is perfectly sensible — a contractual ceiling plus behaviour under contention. Still: measure first, then start with one mechanism.

On container hosts, think about this trio alongside CPU and memory limits; the diagnostic flow in my article about spotting and limiting a resource-hog container on a VPS is the starting point. If storage latency under the virtualisation layer is in play, the questions in my piece on storage IO latency battles in legacy infrastructure apply too: the problem may not be cgroup policy but a shared queue underneath.

Measurement: where to look

io.stat → bytes and operations read/written per group; this is where you see which device is really in use.
io.pressure → pressure, but read it correctly. The averages (avg10/60/300) decay; an avg300 read at noon says nothing about the backup window at midnight. The right method is to take the delta of total across the window you care about. And for IO the full line is a sharper signal than some: full counts the time when every task in the system was waiting.
If you set io.latency, the missed/total fields in io.stat → is the target actually being met?
After the change, the workload's own metric → did the restriction protect what you wanted, or just move the problem?

As a checklist: get the device numbers ($MAJ:$MIN) right and confirm with lsblk that the device is the disk you meant; enable iocost if you're going to use weights; check the filesystem's cgroup writeback support for write-heavy workloads; and justify every policy with a pressure measurement first.

Setting a limit isn't solving the problem

Putting these three files side by side, what I see is the general state of resource management: the hard limit is the easiest tool to understand and the least useful. It's understandable because you give a number and the system obeys. It's not useful because the system's real problem isn't average throughput, it's behaviour under contention.

If there's no contention you need no policy at all — as on my server. If there is, the question isn't "how many MB/s" but "who comes before whom, and what am I protecting?" The moment you know that answer, which file to write into is already decided.

Official Sources

nftables flowtable: Speed Paid for in Visibility

Mustafa ERBAY — Sun, 06 Sep 2026 08:56:44 +0000

When you use a Linux box as a router, every packet takes the same journey: connection tracking, rule chain, NAT, rule chain again, egress. For a single connection that journey is negligible. On a box forwarding hundreds of thousands of packets per second, most of the CPU bill accumulates exactly there — and the interesting part is that the decision about the vast majority of those packets has already been made.

Netfilter's flowtable infrastructure is built on that observation. The first packets of a flow go through the normal path and get the decision; after that the flow is written into a table, and packets matching it reach the egress without ever seeing the classic path.

The real price of flowtable, paid in exchange for the CPU it saves, is visibility. The speed you gain is paid for by losing the answer to "which rule saw this packet?" Set it up knowing that and it's an excellent tool; set it up without knowing and months later you'll be chasing a counter that doesn't move or a rule that never fires.

What exactly the fast path skips

The kernel documentation says it in one sentence: a packet that finds a matching entry in the flowtable is transmitted straight to the output netdevice, so packets bypass the classic IP forwarding path — and then the critical note in parentheses: the visible effect is that you do not see these packets from any of the Netfilter hooks coming after ingress.

If there's no match, the packet continues on the classic path. So flowtable isn't an all-or-nothing switch, it's a cache.

An update is needed here, because this document hasn't been meaningfully revised since 2021. The text describes the transmit path via neigh_xmit(); current kernels consolidated that path and call dev_queue_xmit(). The difference isn't academic: on the dev_queue_xmit() path, nftables' netdev egress hook and the tc/qdisc layer are still in play. So what you lose on the software fast path isn't "all visibility" but the rules in the forward and postrouting hooks, and their counters. Your traffic shaping (HTB, fq_codel, CAKE) keeps working; tc ingress and XDP run before the flowtable anyway, so they're unaffected too.

With hardware offload it really is "you see nothing": the packet never reaches the CPU, so no rule, no qdisc and no egress hook is involved. The meaningful line is software versus hardware.

The lookup uses this n-tuple: layer 2 encapsulation (VLAN and PPPoE), layer 3 source and destination, layer 4 source and destination ports, and the input interface. That last one matters for setups with several conntrack zones.

Setup: one table, one rule

The documentation's example is as plain as it gets — you define a flowtable and add a single rule to the forward chain:

table inet x {
    flowtable f {
        hook ingress priority 0; devices = { eth0, eth1 };
    }
    chain y {
        type filter hook forward priority 0; policy accept;
        ip protocol tcp flow add @f
        counter packets 0 bytes 0
    }
}

The flow add @f statement means you choose which flows take the fast path. The documentation is explicit about this: packets follow the classic path unless the user explicitly instructs otherwise. Flowtable doesn't kick in by default; it is enabled by policy.

If you need it, you can create several flowtables for resource partitioning. The flowtable priority defines the order in which hooks run in the pipeline. If you already have an nftables ingress chain, the documentation's rule is explicit: keep the flowtable priority smaller than that chain's, so the flowtable runs first.

What falls off the fast path

Not every packet qualifies, and the rules make sense:

Fragmented traffic goes to the classic path, because with the transport header missing a flowtable lookup isn't possible.
TCP RST and FIN packets are described by the documentation as passed up, but that has gone stale: since the change landed in 6.14 these packets mark the flow as CLOSING and stay on the fast path; the flow is torn down only when a new SYN arrives on a closing connection.
Packets exceeding the MTU take the classic path too, so a packet-too-big ICMP error can be reported to the sender.

The list needs additions that aren't in the documentation but sit in the source, because "why aren't some of my flows accelerated" is usually answered here: flows with a conntrack helper attached — FTP, SIP, TFTP and friends — are never offloaded; packets carrying IPsec, IPv4 packets with options, and connections marked with sequence adjustment or NAT clash stay out too. On the TCP side the flow must be established. And there's a configuration constraint: flow add is only valid in the forward chain.

On the NAT side: the flowtable stores the NAT configuration, and packets are mangled according to the NAT policy specified from the classic path. The TTL is decremented before neigh_xmit() is called. So the fast path doesn't skip the router's duties, only the decision chain.

To see which flows took the fast path, look at the connection tracking table; offloaded flows carry the [OFFLOAD] tag:

# conntrack -L
tcp  6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 ... [OFFLOAD] mark=0 use=2

Visibility: what exactly do you lose?

In the example above the rule ends with a counter. The documentation states plainly that once the flow is offloaded, that counter no longer gets updated for packets forwarded through the bypass.

This holds for everything rule-based: counters, log statements, filter rules further down, marking. You wrote the rule, but the packet never visits it.

Don't assume you lose all accounting, though. You can add a counter statement to the flowtable itself, in which case it synchronises packet and byte counters with the existing connection tracking entry:

table inet x {
    flowtable f {
        hook ingress priority 0; devices = { eth0, eth1 }; counter
    }
}

That support has been available since kernel 5.7. On its own, though, it isn't enough: the counters are written into the conntrack entry, so conntrack accounting has to be on too — and it's off by default.

sysctl -w net.netfilter.nf_conntrack_acct=1   # make it permanent in /etc/sysctl.d/

One caveat: the accounting extension is attached when the conntrack entry is created, so enabling this later affects new connections only. Do both and flow-level accounting stays with you; what you lose is rule-level detail. Clarify that distinction before deployment: if your monitoring reports "how many packets did each rule see", that report starts lying quietly the moment flowtable comes online.

There's a second cost next to visibility, and it gets even less attention: when a flow is offloaded, the kernel disables TCP window validation for that conntrack entry. The comment in the source states the reason plainly — conntrack will no longer see all the packets, so sequence and window checking becomes meaningless. In other words, state validation is relaxed for the flows you put on the fast path; whether that's acceptable depends on how trusted the flow is.

The same distinction applies to security. If you built detailed egress control like the one in my article on an nftables egress policy layer, you have to narrow your flow add condition knowing that those rules won't inspect the offloaded flows. Opening the fast path only to already-decided, trusted traffic is mandatory here.

How long does a flow stay on the fast path?

The behaviour that confuses people most after setup: a flow shows up on the fast path, disappears, then comes back. The reason is timeouts, and they have their own sysctls:

$ sysctl net.netfilter.nf_flowtable_tcp_timeout net.netfilter.nf_flowtable_udp_timeout
net.netfilter.nf_flowtable_tcp_timeout = 30
net.netfilter.nf_flowtable_udp_timeout = 30

In the kernel documentation's words these control the offload timeout: TCP and UDP connections may be offloaded from conntrack to the flow table, and once aged, the connection is returned to conntrack.

The important detail: this is an idle timer. Every packet taking the fast path refreshes the timeout, so a flow with traffic doesn't drop off every 30 seconds. What drops off are flows that genuinely carry no packet for 30 seconds — sparse heartbeat channels, idle sessions. When those come back, your rule counters tick by a packet or two per flow; that's why you see small increments where you expected "the counter never moves".

That's why you shouldn't judge the fast path from a single snapshot; watch the number of [OFFLOAD]-tagged flows in conntrack -L over time.

Hardware offload and bridges

If your network device supports hardware offload, adding flags offload; to the flowtable definition is enough. A workqueue adds flows to the hardware; the documentation warns that a few packets may still run over the software fast path until the workqueue gets a chance. Flows that made it to hardware appear with the [HW_OFFLOAD] tag in conntrack. The infrastructure also supports DSA (Distributed Switch Architecture).

At layer 2 things got easier in 5.13: the flowtable discovers the real netdevice behind VLAN and PPPoE devices, parses those headers, and uses the ethertype and VLAN/PPPoE session id for lookups. So you don't need to add the VLAN and PPPoE devices to your flowtable definition; the real device is sufficient.

For bridges the rule is the opposite, and the difference is easy to miss: if you want a fast path between your bridge ports and the IP forwarding path, you have to add the bridge ports (as represented by the real netdevice) to your flowtable definition. With VLAN/PPPoE you don't need to add them; with a bridge you must. The infrastructure then discovers the topology behind the bridge device and establishes a fast path between bridge ports and the gateway device; bridge VLAN filtering (PVID, untagged) is supported as well. For people building home or branch routers, that's usually where the real gain is.

The price of being a cache: stale entries

The documentation's "limitations" section is short but important: the flowtable behaves like a cache and entries might get stale if either the destination MAC address or the egress netdevice used for transmission changes. Two situations are called out: running the flowtable in software mode while combining bridge and IP forwarding, and having hardware offload enabled.

In practice that means care in setups with moving topology: a router switching between redundant uplinks, virtual MACs carried by VRRP, bridge ports relearned after a cable change. In such an environment, test the failover scenario deliberately before enabling the fast path: the real question is what happens the moment the link changes.

Do you need it on your box?

Looking at my own VPS, the answer is clearly no:

$ nft --version
nftables v1.0.9 (Old Doc Yak #3)

$ ls /lib/modules/$(uname -r)/kernel/net/netfilter/ | grep flow
nf_flow_table.ko.zst
nf_flow_table_inet.ko.zst
nft_flow_offload.ko.zst

$ lsmod | grep -c nf_flow
0

$ cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
745
262144

The modules exist, they aren't loaded, and they don't need to be: this machine is an application server, forwarded traffic is negligible, and the conntrack table hovers around 745 entries out of a 262k capacity. The CPU flowtable would save here disappears into measurement noise.

Where it does add value is clear: edge boxes forwarding at high packet rates, home and branch routers, VPN gateways, heavy east-west traffic on container hosts. What they share is millions of packets from the same flow passing through the same decision over and over.

Where it doesn't is equally clear: traffic you want to inspect deeply, environments doing detailed rule-level accounting, and setups whose topology keeps changing. On the IDS/IPS side the concrete trap is NFQUEUE: if you run Suricata or Snort inline in the forward chain, offloaded packets never enter that queue — silently. Your inspection rule is still there, your engine is running, but part of the traffic no longer passes through it.

Let's also correct a conntrack expectation: offloading doesn't reduce conntrack entries. The entry stays, it just gets the [OFFLOAD] tag. Flowtable is not a cure for a filling conntrack table.

If you want to measure the gain, conntrack -L | grep -c OFFLOAD alone won't do; compare forwarded packet rate and softirq CPU (mpstat -P ALL, /proc/net/softnet_stat) before and after. I didn't run that measurement on my own box for this article — there's no load to measure; do it before you enable it.

Checklist before you enable it

Keep the flow add condition narrow: only decided, trusted flows (specific interface pairs and protocols, say).
Add counter to the flowtable definition so you keep flow-level accounting.
Review monitoring dashboards that rely on rule counters or logs; offloaded flows won't appear there.
Check for the [OFFLOAD] and [HW_OFFLOAD] tags in conntrack -L: are the flows you expected really on the fast path?
Test failover, VLAN changes and bridge topology scenarios; the stale-entry risk lives there.
With hardware offload, verify your NIC driver support and account for the first packets still taking the software path.

In my article about moving from iptables to nftables we talked about the readability of a rule set; flowtable changes how much of that rule set actually runs. The two belong in the same conversation.

Speed always comes from the same place

What flowtable does is computing's oldest trick: instead of making the same decision over and over, store the result. And like every stored decision, a gap can open between it and reality — an entry going stale when a MAC address changes, a counter that stops moving, a packet that never reaches the queue.

So the question for your own setup: how many times is the same decision made on this machine? If the answer is millions, flowtable has a place; if it's a few thousand, what you get for the lost visibility is speed you won't be able to measure.

Official Sources

Compressed Swap: zram or zswap?

Mustafa ERBAY — Sun, 06 Sep 2026 08:48:59 +0000

No corner of server discussions surrenders to slogans faster than swap. On one side, "turn swap off, RAM is cheap"; on the other, people blaming swap after every freeze. Both make the same mistake: they treat swap as a single thing.

In a modern kernel, compressed swap exists as two separate mechanisms, and because their names look alike they get confused constantly. zram is a compressed block device; you put swap on it and it never touches the disk. zswap is a compressed cache in front of your existing swap area; it catches pages before they go to disk.

The thesis of this article is simple: choosing between them isn't a matter of taste, it's an architectural decision determined by the machine's disk situation. And neither of them creates memory — they trade CPU for disk.

Neither is present on my server, and the reason is interesting

Let's start with my server running Ubuntu 24.04, because the picture is typical:

$ modinfo zram
modinfo: ERROR: Module zram not found.

$ apt-cache policy linux-modules-extra-$(uname -r)
linux-modules-extra-6.8.0-138-generic:
  Installed: (none)
  Candidate: 6.8.0-138.138

$ grep -E "CONFIG_ZSWAP|CONFIG_ZSWAP_DEFAULT_ON" /boot/config-$(uname -r)
CONFIG_ZSWAP=y
# CONFIG_ZSWAP_DEFAULT_ON is not set

$ cat /sys/module/zswap/parameters/enabled
N

There are two different kinds of "absent" here. zram really is absent: on Ubuntu the module ships in a separate package (linux-modules-extra) which isn't installed on this machine. zswap, on the other hand, is compiled into the kernel but off by default — one line turns it on.

That distinction matters in practice. Someone saying "let me try zram" on this server starts with a package installation; someone saying "let me try zswap" starts with an echo. That's two very different preparation costs on the same machine.

They don't do the same job

Broadly speaking, both say "compress the swap page", but architecturally they sit in different places.

zram, in the kernel documentation's words, creates compressed RAM-based block devices: pages written to these disks are compressed and stored in memory itself. So /dev/zram0 is an ordinary block device as far as you're concerned; you run mkswap on it, say swapon, and your system's swap area now lives inside RAM in compressed form. It has no relationship with the disk — unless you want one.

zswap isn't a device but a cache. As the documentation puts it, it's a lightweight compressed cache that takes pages in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based pool. When the pool reaches its size limit, it evicts pages on an LRU basis to the backing swap device. So zswap needs a swap area on disk to work; it's a buffer in front of that area.

The operational consequence: zram makes swap possible on machines with no disk, or whose disk you'd rather not wear out. zswap makes the swap you already have less painful.

zram: setting up the device and measuring it

The zram flow is short. Load the module, size the device, pick an algorithm, put swap on it:

modprobe zram num_devices=1
echo zstd > /sys/block/zram0/comp_algorithm
echo 4G   > /sys/block/zram0/disksize
mkswap /dev/zram0 && swapon --priority 100 /dev/zram0

To add a device at runtime, reading /sys/class/zram-control/hot_add returns a new device id, and hot_remove takes one away. You see the available algorithms by reading comp_algorithm — the selected one appears in brackets.

For sizing, the documentation has its own rule and it's eminently sensible: since a 2:1 compression ratio is expected, there's little point creating a zram larger than twice the size of memory. If you want a hard guarantee anyway, mem_limit caps how much memory the device may use.

The real work starts after setup: measuring. The mm_stat file gives you the truth in a single line: orig_data_size is the uncompressed data size, compr_data_size its compressed form, mem_used_total the memory actually allocated including fragmentation, same_pages the identical pages that need no storage at all, and huge_pages the incompressible ones.

That last field is the number anyone running zram in production needs to watch. An incompressible page means zram is spending memory while giving you nothing. On a workload carrying already-compressed data — encrypted files, media, pre-compressed database pages — huge_pages climbs fast and the "I'm saving memory" story quietly reverses. To see the real ratio, divide orig_data_size by mem_used_total: the memory you pay for is mem_used_total. Comparing it against compr_data_size measures something else — the allocator's space efficiency.

Now zram's most critical and least discussed property: when it fills up, there is nowhere to fall back to. A full zswap pool evicts pages to the real swap behind it; zram has no such tier — a full device is just a block device at capacity, and the system walks straight towards OOM.

The antidote is backing_dev. You define a backing device and use writeback to push idle or incompressible pages to a real disk. Two setup constraints will make you fail silently: the backing device must be set before disksize, and only partitions are supported at the moment.

echo /dev/sda5 > /sys/block/zram0/backing_dev   # this first
echo 4G        > /sys/block/zram0/disksize      # then this
echo 1         > /sys/block/zram0/writeback_limit_enable
echo 4096      > /sys/block/zram0/writeback_limit

That third line matters: writeback_limit_enable defaults to 0, and while it is 0 the writeback_limit value you wrote is meaningless. You'd think you set a budget when you didn't.

There's one more capability: recompression. In a multi-compression configuration you can define one primary and up to three secondary algorithms, then use recompress to re-compress specific page types with a stronger algorithm.

Don't skip the second half of the sizing rule either: the documentation notes that an unused zram costs about 0.1% of the disk size, so a huge zram is wasteful.

On the system side, rather than wiring this up by hand, systemd's zram-generator is cleaner: it creates the device and sets up swap at boot, with the configuration living in a single file.

zswap: a buffer you enable with one echo

The zswap side is less setup, more tuning. To turn it on at runtime:

echo 1 > /sys/module/zswap/parameters/enabled

To make it permanent you add zswap.enabled=1 to the kernel command line; if your distribution ships with CONFIG_ZSWAP_DEFAULT_ON it's already active.

All the knobs live under /sys/module/zswap/parameters/, and here's how they look on my server:

$ grep -r . /sys/module/zswap/parameters/ | sed 's|.*parameters/||'
same_filled_pages_enabled:Y
enabled:N
shrinker_enabled:Y
max_pool_percent:20
compressor:lzo
non_same_filled_pages_enabled:Y
zpool:zbud
exclusive_loads:N

That output is from Ubuntu 24.04 / kernel 6.8, and part of it is already history: upstream removed the zbud and z3fold allocators along with the zpool parameter — the current zswap documentation says it in one line, zsmalloc manages the compressed pool. exclusive_loads and same_filled_pages_enabled are gone from today's kernel source too. So some of these knobs may simply not exist on your machine; instead of memorising the list, read grep -r . /sys/module/zswap/parameters/.

The three that persist are these. max_pool_percent is the maximum percentage of memory the compressed pool may occupy — default 20, a fifth of your RAM. compressor is the compression algorithm; this kernel's default is LZO. accept_threshold_percent is the hysteresis threshold for when to start accepting pages again after the pool fills.

There's also shrinker_enabled, which turns on the shrinker that proactively writes cold pages out to swap. On my kernel it ships enabled — but since zswap itself is off, it has no effect.

Know from the outset that turning it off isn't symmetric with turning it on: the documentation says that when zswap is disabled at runtime it stops storing pages, but does not immediately write out or fault back the pages already in the compressed pool. To actually drain the pool you have to run swapoff on the swap device.

Rethinking swappiness

Setting up compressed swap and leaving vm.swappiness alone is a job half done. That knob answers the kernel's question of "how expensive is swap I/O relative to filesystem paging?", and the default of 60 is an estimate from the world of spinning disks.

The kernel documentation is surprisingly explicit here: the value ranges from 0 to 200, at 100 the VM treats both costs as equal, and for in-memory swap like zram or zswap, values beyond 100 can be considered. The docs even give the arithmetic: if random I/O against the swap device is on average 2x faster than filesystem I/O, swappiness should be 133 (x + 2x = 200).

Here are the values on my own server:

$ free -m | head -2
               total        used        free      shared  buff/cache   available
Mem:           96556       17525       69760        1595       11839       79031

$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file   8G   0B   -2

$ sysctl vm.swappiness vm.page-cluster
vm.swappiness = 10
vm.page-cluster = 3

swappiness is 10, meaning the system treats writing to swap as almost a last resort compared with dropping page cache. That's defensible for a swapfile on disk; but if that swap moved to zram, the same 10 would mean deliberately tying your own hands.

The second line gets less attention and deserves as much: vm.page-cluster is a logarithmic value controlling how many consecutive pages are read from swap in one attempt, and it defaults to 3 — eight pages. That behaviour makes sense for spinning disks, where fetching neighbouring pages costs little extra. In compressed swap each page is decompressed individually, and seven of those eight may have been decompressed for nothing. Setting it to zero disables swap readahead entirely, which is the common recommendation for zram setups.

Changing the mechanism without changing the cost assumptions is the most common mistake in compressed swap setups — and that assumption isn't one knob, it's at least two.

Which one for which machine?

The first question in the decision tree: is there a swap area on this machine that you want to use?

If there isn't and you don't want one — a laptop whose SSD lifetime you're protecting, a diskless edge device, a VM on a tight disk quota — the answer is zram. Swap lives entirely in RAM and the disk never enters the picture.

If there is, and your actual complaint is the machine grinding to a halt whenever it swaps, the answer is zswap. Some pages stay in the RAM pool without ever reaching disk, disk traffic drops, and your existing swap area stands behind it as insurance.

Using both at once is theoretically possible, but I wouldn't recommend it — and not only because you pay CPU twice. When pages evicted from the zswap pool land in zram, cold pages stay locked in fast RAM and the LRU logic inverts. You end up with a setup doing the opposite of what you wanted, which was keeping the active working set in RAM.

So why is neither present on my 96 GB server? Because swap there is already just insurance; the machine sits at 17 GB in use and the 8 GB file swap in swapon --show reports zero usage. Compressed swap helps systems that actually fall into swap. On a system that never does, every mechanism you add raises more questions than it answers.

And there's the obvious truth nobody says out loud: neither mechanism creates memory. Compressed pages still live in RAM, they just take up less room. Under genuine memory pressure zram buys you time, it doesn't rescue you — and you buy that time with CPU cycles.

Containers, OOM and hibernation

Three more details, all of which show up during the first crisis rather than the first setup.

First, cgroup accounting — and here the two differ sharply. In both setups swap usage counts against memory.swap.max, so the expectation that "I added compressed swap, my containers can breathe now" doesn't materialise unless you loosen the limits yourself.

The real difference is in the memory held by the compressed pages. For zswap, cgroup v2 has dedicated knobs: memory.zswap.current shows that group's zswap consumption, memory.zswap.max is a hard limit, and memory.zswap.writeback set to 0 disables writing to swap devices for that group (the setting is hierarchical — disable it above and it stays disabled below). zram has no such accounting: the RAM the device holds for compressed pages is spent at host level without any record of which container it was held for. On a container-dense machine that alone is an argument for zswap if you care about isolation and pressure signals.

Second, OOM behaviour. Compressed swap delays memory exhaustion, it doesn't cancel it — and while it delays, the system burns CPU. If you want to catch memory pressure early and evict in a controlled way, the answer isn't compression but a pressure-based policy; the approach in my systemd-oomd cgroup v2 memory pressure runbook is complementary here. Compressed swap buys you time; policy decides what you do with it.

Third, hibernation. The hibernation image has to be written to a persistent device, and a zram device living in RAM vanishes the moment power goes. If you set up zram on a laptop and still expect hibernate to work, you need a separate swap area on disk.

These three are different faces of what I lived through in my first OOM and swap storm: in memory management, turning a single knob rarely settles anything.

Don't decide without measuring

The sneakiest thing about compressed swap is that whether it's working isn't obvious. After setup, the places to look:

awk '{print $1/$3}' /sys/block/zram0/mm_stat → the real compression ratio (orig_data_size / mem_used_total). The field order in that file is orig_data_size compr_data_size mem_used_total mem_limit mem_used_max same_pages pages_compacted huge_pages huge_pages_since. Track incompressible workloads with huge_pages_since, which counts since setup, rather than the instantaneous huge_pages.
grep -r . /sys/kernel/debug/zswap/ → this is where zswap is measured; the documentation notes a debugfs interface providing statistics about pool size, the number of pages stored, and counters for why pages were rejected.
swapon --show → which swap area is actually being used, and at what priority.
cat /proc/pressure/memory → did memory pressure really drop, or did it just move?
the si/so columns of vmstat 1 → the volume of page in/out.

As a checklist: for zram, confirm the module package is installed; don't make disksize larger than twice your RAM; reconsider swappiness for compressed swap (persuading the kernel to be more willing can make sense when swap doesn't touch disk); for zswap, tune max_pool_percent to your workload's real compression ratio; and on both sides, do a reboot drill before making the change permanent.

Rethinking swap

Putting these two mechanisms side by side, what strikes me isn't their technical difference but that both are answers to the same question: when memory runs short, what gets sacrificed?

Classic swap sacrifices the disk — latency explodes but the system stays up. zram and zswap sacrifice CPU instead, and in return keep latency far lower. Which is right depends on which resource is abundant in your system.

So the question for your own setup is this: is your bottleneck really memory, or the disk latency that appears the moment memory gets tight? If it's the second, the fix may not be buying more RAM — sometimes an echo 1 is enough.

Official Sources

Landlock: When a Program Jails Itself

Mustafa ERBAY — Sat, 05 Sep 2026 18:39:15 +0000

When we want to limit what a program can do, our habit is to look in the same direction: at the administrator. We write an AppArmor profile, compile an SELinux policy, add ProtectSystem= to a systemd unit. All of it works. But all of it shares one weakness — the party that sets the limit and the party that writes the restricted code are different, so the two drift apart over time. The application starts writing to a new directory and the profile stays behind; or the profile is far too wide and nobody notices.

Landlock inverts that relationship. The process places the limit on itself: at startup it says "I will only read these directories and connect to this port", and from that moment on it cannot do more. No root privileges required, no policy file, and no way to undo it.

The thesis of this article: Landlock's real novelty isn't being "one more sandbox", it's changing who owns the policy. That change has a price — version negotiation is now the application's job, and the list of things it cannot restrict is too long to ignore.

Already installed on my server

Landlock has been in the kernel since 5.13 and most current distributions ship it enabled — but not all of them. On my server running Ubuntu 24.04, the active security modules look like this:

$ cat /sys/kernel/security/lsm
lockdown,capability,landlock,yama,apparmor

AppArmor and Landlock run side by side on the same machine; neither replaces the other. The restrictions stack, and each layer can separately deny the access that falls to it.

If you don't see landlock in that list, you're not done yet. There are two separate switches: the module being compiled in (CONFIG_SECURITY_LANDLOCK=y) and being present in the list enabled at boot (CONFIG_LSM). If the second is missing — and on some distribution kernels the list ships as lockdown,yama,integrity,apparmor — you need to add lsm=landlock,... to the kernel command line. You can check from the kernel log, since Landlock announces itself when it comes up:

zgrep -h "^CONFIG_LSM=" "/boot/config-$(uname -r)" /proc/config.gz 2>/dev/null
dmesg | grep landlock || journalctl -kb -g landlock   # "landlock: Up and running"

But which Landlock? Because there isn't a single "Landlock", there's a numbered series of capabilities. The way to ask for the version is, interestingly, a system call:

$ python3 -c "
import ctypes
libc = ctypes.CDLL(None, use_errno=True)
abi = libc.syscall(444, None, ctypes.c_size_t(0), ctypes.c_uint32(1))
print('ABI:', abi, '| errno:', ctypes.get_errno())"
ABI: 4 | errno: 0

(444 is the landlock_create_ruleset syscall number on x86_64 and arm64; on failure the return value is -1 and errno tells the two cases apart.)

The landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION) call returns the highest supported ABI version. On my kernel the answer is 4. We'll get to what that means shortly; for now note this: the answer can be anything between 1 and 11, and the difference directly determines what your sandbox can and cannot restrict.

The mechanism: three steps and a one-way door

The flow is short. First you define a ruleset declaring which kinds of access it handles — filesystem rights, and since ABI 4, TCP port rights. Then you add the hierarchies you allow: "this directory is readable", "this one is writable". In the final step, the process restricts itself with that ruleset.

The third step has two conditions. First: for unprivileged processes, the no_new_privs attribute must be set. Processes carrying CAP_SYS_ADMIN can skip it, but the documentation advises against skipping — a sandboxed process could still execute a SUID binary, and that binary would run with elevated privileges inside a Landlock domain it doesn't expect, turning it into a confused deputy.

Second, and more importantly: this door is one-way. The documentation is explicit — once a thread is landlocked there is no way to remove its policy, only to add more restrictions. Every new thread born via clone(2) inherits the restriction from its parent, and it survives execve. So if a shell landlocks itself and then invokes a compiler, the compiler is born into the same jail.

The nicest practical consequence: the sandbox no longer lives in a file shipped with the distribution, it lives in the application's own code. It can't be forgotten, it doesn't fall behind on updates, and there's no "why is the profile disabled" conversation.

There is one boundary: at most 16 stacked rulesets, and once that limit is reached landlock_restrict_self() returns E2BIG. That's why the documentation recommends building the ruleset carefully, once in a thread's lifetime — especially for shells and container managers that launch other applications.

The code itself: three system calls

To make this concrete, the skeleton from the kernel documentation's tutorial is enough. First, the ruleset is defined by explicitly listing the access types to be handled. Landlock denies those types by default; an access type you don't declare isn't checked at all:

struct landlock_ruleset_attr ruleset_attr = {
    .handled_access_fs =
        LANDLOCK_ACCESS_FS_EXECUTE | LANDLOCK_ACCESS_FS_WRITE_FILE |
        LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR |
        LANDLOCK_ACCESS_FS_REMOVE_DIR | LANDLOCK_ACCESS_FS_REMOVE_FILE |
        LANDLOCK_ACCESS_FS_MAKE_DIR | LANDLOCK_ACCESS_FS_MAKE_REG |
        LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_TRUNCATE,
    .handled_access_net =
        LANDLOCK_ACCESS_NET_BIND_TCP | LANDLOCK_ACCESS_NET_CONNECT_TCP,
};

That attribute struct becomes a ruleset descriptor:

int ruleset_fd = landlock_create_ruleset(&ruleset_attr, sizeof(ruleset_attr), 0);

Then the allowed hierarchy is added. The striking detail is that the directory is opened with O_PATH and handed over as a file descriptor — the rule is defined over an opened object, not a path string:

struct landlock_path_beneath_attr path_beneath = {
    .allowed_access = LANDLOCK_ACCESS_FS_EXECUTE |
                      LANDLOCK_ACCESS_FS_READ_FILE |
                      LANDLOCK_ACCESS_FS_READ_DIR,
};
path_beneath.allowed_access &= ruleset_attr.handled_access_fs;
path_beneath.parent_fd = open("/usr", O_PATH | O_CLOEXEC);
landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &path_beneath, 0);

The final step closes the door:

prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
landlock_restrict_self(ruleset_fd, 0);

The masking line is no accident: allowed rights must be a subset of handled rights. When you trim handled_access_fs according to the ABI, that mask narrows the rule automatically. The documentation applies the same trimming to landlock_restrict_self() flags — logging flags are removed below ABI 7, the multithreaded enforcement flag below ABI 8.

The filesystem rights list being this granular is deliberate too. Reading and listing a directory, creating a file and creating a directory, truncating and writing all have separate rights. That level of detail looks tiring at first, but it's the only way to keep a restriction genuinely narrow.

ABI negotiation is now your job

This is Landlock's most misunderstood aspect. Your code doesn't know which kernel it will run on, and Landlock deliberately builds backward compatibility on "explicitly declared rights" rather than "silently getting stricter". Making the handled access rights explicit creates a clear contract between kernel and user space, so a system update can't tighten the sandbox unexpectedly and break the application.

The price is that the application itself has to climb the version ladder. ABI 2 brought control over renaming and linking across directories (LANDLOCK_ACCESS_FS_REFER), ABI 3 file truncation (LANDLOCK_ACCESS_FS_TRUNCATE), ABI 4 the TCP side (LANDLOCK_ACCESS_NET_BIND_TCP, LANDLOCK_ACCESS_NET_CONNECT_TCP), ABI 5 ioctl control on device files, ABI 6 scoping restrictions for abstract unix sockets and signals, ABI 7 the logging flags, ABI 8 multithreaded enforcement, ABI 9 restrictions on connecting to pathname unix sockets (LANDLOCK_ACCESS_FS_RESOLVE_UNIX), ABI 10 the UDP rights plus per-rule log suppression, and ABI 11 the flag that folds no_new_privs into the enforcement call itself.

Now back to my server: ABI 4. A sandbox I write there can restrict the filesystem and TCP connections, and can't touch UDP at all.

But the real boundary isn't UDP, and it deserves to be said plainly: Landlock sees the network at port level, not at destination level. The rule structure (struct landlock_net_port_attr) carries exactly one field — the port. So when you say "you may connect to TCP 443", the process may connect to port 443 on every server in the world. If you want to close the exfiltration path, Landlock alone won't do it; destination-based egress control is still a job for nftables, an egress proxy, or a network namespace. The missing UDP is only a sub-heading of that picture — and ABI 10 doesn't solve it at the address level either, it merely makes UDP controllable by port.

The most painful rung of the ladder is threads. Up to ABI 8 — and even at ABI 8 when the relevant flag isn't used — landlock_restrict_self() enforces the policy only for the calling thread and its children; sibling and parent threads stay outside. The documentation's warning is exactly this: the call is equivalent to "I jailed the process" only if the process is single-threaded. Forget it in a program written with Go, Java or any pooled runtime and you'll have jailed one thread while believing you jailed yourself. The multithreaded enforcement flag (LANDLOCK_RESTRICT_SELF_TSYNC) arrived with ABI 8; on older kernels the only way out is to apply the restriction at the very start of the program, before any extra thread is born.

The recommended path here is "best effort": detect the ABI version at runtime, use only the supported subset of rights, and protect users as much as possible whatever kernel they run. The error returned when the query fails also guides you: ENOSYS means the kernel doesn't support Landlock at all, EOPNOTSUPP means it has been disabled.

The engineering decision is yours: what will you do on a missing ABI? Silently continuing unrestricted is what most desktop tools choose. On the server side I think the right move is at least to print a warning at startup saying which protections aren't active.

How will you see a denied access?

When writing a sandbox, the real time sink isn't setting up the policy, it's answering "why doesn't this work". This side of Landlock matured late and in two steps. First ABI 7: the LANDLOCK_RESTRICT_SELF_LOG_* flags that govern whether denied accesses produce audit records. Then ABI 10: the LANDLOCK_ADD_RULE_QUIET flag together with the ruleset's quiet_access_fs/quiet_access_net fields, which selectively suppress logs for denied accesses on specific objects. Since suppression only applies to the layer that denies the access, a sandboxed program can't use it to hide its own violations. Landlock tracepoints are unaffected by these flags either way, so they remain a separate observability channel for debugging.

None of that exists on my ABI 4. So on an older kernel, debugging falls back to the classic method: hunt EACCES with strace and loosen the restriction step by step. Measuring which paths the application actually touches before switching the sandbox on in production isn't just a good habit on older kernels, it's a requirement.

One more caveat from the documentation: there is a LANDLOCK_CREATE_RULESET_ERRATA query for checking which bugs are fixed at runtime, but the docs say the vast majority of applications should not look at it — it complicates the code and can reduce protection when misused. In security APIs, "more configuration knowledge" doesn't always mean better security.

What it cannot restrict

Landlock is not a container and doesn't try to be. The "current limitations" section of the documentation lists them honestly; these are the ones that will hurt in production.

A sandboxed thread cannot modify filesystem topology: mount(2) and pivot_root(2) are denied. But chroot(2) is not refused — changing the root is still possible.

Objects that don't come from a user-visible filesystem (pipes, sockets) can still be reached through /proc/<pid>/fd/* yet cannot be explicitly restricted; the same goes for special kernel filesystems such as nsfs. What steps in for those is the ptrace restrictions: access to such sensitive /proc files is automatically limited according to domain hierarchies.

The item that traps most people is already-open file descriptors. Landlock evaluates access at open time; a descriptor opened before the jail stays in your hands and can even be passed between processes, carrying its Landlock properties with it. LANDLOCK_ACCESS_FS_IOCTL_DEV says so explicitly: it applies only to newly opened device files, and pre-existing descriptors like stdin, stdout and stderr are unaffected.

That behaviour is both a trap and a tool. A trap, because while you say "I restricted everything" an old descriptor may remain in your hands. A tool, because opening the file you need before landlocking and keeping the descriptor is one way to preserve access to it. Just know the limits: what you keep is that one open file, every later openat in the same directory is checked again, the rights that stick to a descriptor are limited to LANDLOCK_ACCESS_FS_TRUNCATE and LANDLOCK_ACCESS_FS_IOCTL_DEV, and when a log file is rotated your descriptor keeps writing to the old inode — you'll be logging to the wrong file without noticing.

There's also a list of operations that simply aren't covered: calls like chdir, stat, flock, chmod, chown, setxattr, utime, fcntl and access fall outside Landlock's scope. So if you think "it can't do anything in this directory", quite a lot of metadata work remains possible.

Where it helps and where it doesn't

What accelerated Landlock's rise over the past year is, I think, coding agents. You want a tool that runs shell commands on your machine, but you don't want it reading ~/.ssh. On the filesystem side Landlock gives you exactly that; on the network side, remember the port restriction above — you can say "it only reads these directories", you can't say "it only connects to this server". A container is heavy and full of friction here; Landlock works inside the process itself, without asking for extra privileges. The go-landlock library and command-line wrappers like landrun exist because of that need.

On the server side, put Landlock on top of admin-side hardening, not in place of it. Restrictions in the unit file are still your first line of defence; the ProtectSystem= family I wrote about in hardening systemd services with the service sandbox is the only practical route for third-party services whose code you can't change. Landlock enters where you do own the code: the application knows its own needs best.

To see the two working together, a glance at the LSM list on my server is enough: AppArmor and Landlock sit right next to each other.

There's also a trap for services running under systemd. In a unit that says SystemCallFilter=@system-service, Landlock's system calls get caught by the filter, because they live in systemd's @sandbox group and @system-service doesn't include it. If a service trying to jail itself fails unexpectedly, check the unit's syscall filter first — the fix is SystemCallFilter=@system-service @sandbox. The same logic applies to seccomp: Landlock and seccomp aren't alternatives, a typical sandbox uses both, and a narrow seccomp filter can accidentally block Landlock itself.

So where doesn't it help? It doesn't if you can't change the code — then you either write a wrapper or return to admin-side tooling. If you want filtering at the system call level you're holding the wrong tool; that's seccomp's job. And if you want an isolated root filesystem you need containers; Landlock doesn't give you a separate world, it gives you a narrower view of the same one.

Before you start

First check that it's enabled (/sys/kernel/security/lsm, dmesg | grep landlock), then measure the ABI version. The lowest version in your fleet is your real level of protection.
Declare rights explicitly; an access type you don't handle isn't restricted.
Set no_new_privs — even when you have the privileges to skip it.
Build the ruleset once; don't spend the 16-layer limit on other sandboxes.
If you have a multithreaded runtime, apply the restriction at the very start: below ABI 8, enforcement only covers the calling thread and its children.
Open the descriptors you need before the jail; path-based access closes afterwards — but think through file rotation.
Network control is port-level: if you need destination control, plan for nftables or an egress proxy; UDP isn't covered at all before ABI 10.
Restrictions can't be undone: on failure your only option is restarting the process, so design for it.

Who writes the policy?

Looking at Landlock, what caught my attention more than the technical detail was the answer to a question: who should place a restriction? The classic answer was the administrator, because security was operations' job. Landlock's proposed answer is different — let the restriction be placed by the party that knows best what the program does, which is the code.

That means responsibility shifts to the developer, and frankly not every team is ready for it. But the reward is large: the sandbox becomes a feature of the application rather than an add-on of the deployment. It travels with versions, goes through code review, and is protected by tests.

The question for your own setup follows from that: for how many of your services does the policy live in their own repository, and for how many does it live in a file on a server? If the second group is large, drift is accumulating there — and the first party to notice drift is usually an attacker.

Official Sources

MPTCP: Enabled in the Kernel, Used by Nobody

Mustafa ERBAY — Sat, 05 Sep 2026 18:08:49 +0000

The first time you hear about Multipath TCP it sounds too good: let a single TCP connection flow over several paths, don't drop the session when a link dies, and if you have two links, use both. Anyone with fibre and an LTE backup has had that thought. And the feature isn't in some distant future — it is in your kernel right now, enabled.

That is exactly what I found when I checked my own server. And right after it, the thesis of this article: being enabled does not mean being used. In MPTCP, enabled=1 isn't a starting line, it's just the first door standing ajar. Behind it are three more doors that must open in order, and in practice the place people get stuck is almost never the kernel.

Enabled on my server; never used

Here's the state on my server running Ubuntu 24.04 and kernel 6.8:

$ sysctl net.mptcp
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.checksum_enabled = 0
net.mptcp.close_timeout = 60
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

$ ip mptcp endpoint show
$ ip mptcp limits show
add_addr_accepted 0 subflows 2

$ nstat -az | grep -c MPTcpExt
63
$ nstat -az | grep MPTcpExtMPCapableSYNTX
MPTcpExtMPCapableSYNTX          0                  0.0

The kernel supports MPTCP, it's on, it stands ready to keep 63 separate counters — and every one of them is zero. Not a single MPTCP connection attempt has ever left this machine. On most distributions the picture is identical; this table is the rule itself.

Why zero? Because the kernel being enabled is only one of four conditions.

The first door: the application itself

MPTCP is not a transparent layer that slots in where TCP was. The application has to ask for it when it opens the socket; in kernel documentation terms, that means creating the socket with IPPROTO_MPTCP (value 262):

int sd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Even the errors give you the diagnosis: kernels older than 5.6 return EINVAL, a kernel built without MPTCP gives EPROTONOSUPPORT, and if MPTCP was turned off via net.mptcp.enabled you get ENOPROTOOPT. An application can also ask TCP_IS_MPTCP whether the connection really ended up as MPTCP.

The operational meaning is inconvenient: if the service you run doesn't carry that line in its source, flipping the sysctl changes nothing. The kernel documentation lists several techniques for forcing an application into MPTCP — LD_PRELOAD (mptcpize), eBPF (mptcpify), SystemTAP, GODEBUG on the Go side — and these four are the ones that matter in practice.

The first is systemd's socket activation. In .socket units, SocketProtocol= takes udplite, sctp or mptcp, and the last one opens the socket with IPPROTO_MPTCP. Mind the version trap: this support arrived in systemd 257, and my server on 255 doesn't have it. Why collecting systemctl --version across the fleet is the first step of a runbook is something we also discussed in the previous article, about soft-reboot; the same discipline applies here.

The second is mptcpize, which ships with the mptcpd package. In its man page's own words, it is a program that enables MPTCP on existing legacy services: mptcpize run prog runs the target program forcing MPTCP socket usage instead of TCP, while mptcpize enable unit updates the systemd unit file so the service runs under that launcher.

The third is changing the application. The option that sounds like the most work is actually the cleanest one when the service is yours: a one-line protocol change.

The fourth partly refutes my own thesis: Go. The language added MPTCP support in 1.21, but only used it if the application explicitly asked. Go 1.24 changed the default — GODEBUG=multipathtcp is now 2, meaning MPTCP is enabled by default on listeners, and you need 0 to get the old behaviour back. So a server built with Go 1.24 or later may be listening with MPTCP without anyone doing anything. "Nobody uses it" is increasingly true only of the client side.

The second door: path manager, endpoints and limits

Say the socket was opened with MPTCP. That alone still doesn't give you multiple paths; for additional subflows to be created, the path manager has to know what to do.

There are two kinds: the in-kernel manager applies the same rules to all connections and is configured with ip mptcp endpoint, while the userspace manager (mptcpd) lets you write per-connection rules. One note on how current this is: the net.mptcp.pm_type sysctl that selects between them is deprecated as of 6.15 and replaced by net.mptcp.path_manager. My 6.8 only has the old one — the name you read in the docs may not exist on your kernel.

The endpoint flags are the heart of it. signal means the address will be announced to the peer with an ADD_ADDR option; in the man page's words, a server would typically be responsible for this. subflow means that, if the limits allow, an additional subflow will be created using this address as the source after the connection is established; a client would typically do this. backup marks a subflow as backup: the remote peer only sends data on it when all non-backup subflows are unavailable. fullmesh tries to create a subflow towards every address the peer announces.

A laminar flag was added to that list recently: the patch landed on 26 September 2025 and shipped in 6.18. Its job is to create subflows towards the addresses the peer announces without leaving source address selection to the routing tables, which makes configuration easier when you don't know the announced address in advance; fullmesh takes precedence over it. The help output of iproute2 6.1 on my server still lists signal | subflow | backup | fullmesh, so pay attention to which version's documentation you're reading.

Then come the limits, and here's the real surprise:

$ ip mptcp limits show
add_addr_accepted 0 subflows 2

add_addr_accepted 0 answers the question "how many addresses will I accept if the peer announces some?", and the default is zero. So even if your server announces addresses properly, nothing happens on the client side. Most people who say "I enabled MPTCP but it doesn't work" are standing exactly here; there is nothing wrong with the kernel, the policy is empty.

One more detail: passing dev with the interface name when adding an endpoint matters. The man page warns about it explicitly — without that binding, you may need extra IP rules and routes for packets from that source address to reach the right interface. On a multi-link machine, skipping this means the second subflow tries to leave over the first link, and you end up with one path wearing two names.

Concretely, the simplest setup between a server with two addresses and a client on a single uplink looks like this:

# server: announce the second address to the peer
ip mptcp endpoint add 203.0.113.10 dev eth1 signal
ip mptcp limits set add_addr_accepted 1 subflows 2

# client: allow an extra subflow towards the announced address
ip mptcp endpoint add 192.0.2.20 dev wwan0 subflow
ip mptcp limits set add_addr_accepted 1 subflows 2

# verification on both sides
ip mptcp endpoint show
ip mptcp limits show

The asymmetry is deliberate: the announcing side uses signal, the side that creates the subflow uses subflow. Forget to lift the limits off zero on either side and every command returns successfully while nothing happens.

What actually happens on the wire

To see why these doors are so brittle, look at the handshake; RFC 8684 describes the mechanism plainly.

The first connection opens with the MP_CAPABLE option: the client states its intent in the SYN, the server sends its key in the SYN/ACK, and the client sends both keys together in the third packet. Those keys are never sent in the clear again; each end hashes its key with SHA-256 to produce a 32-bit token, and the connection is identified by that token from then on.

Second and subsequent paths join with MP_JOIN. The new subflow's SYN carries the token that answers "which connection am I joining", along with a random nonce; the peer replies with an HMAC derived from the exchanged keys. So an additional path is authenticated with the keys from the initial handshake, which is what stops an off-path attacker from adding a subflow to your session. Don't expect more than that: the keys travel in the clear during the initial handshake, and the goal the RFC states is for MPTCP's security to be "no worse" than today's TCP. Against someone who can observe the path, your guarantee is the same as plain TCP's.

Address announcements travel in ADD_ADDR, and those are signed with an HMAC too. The use case the RFC highlights belongs squarely to our world: when a NAT prevents setup in one direction, an address is announced so the other side can connect instead.

The mechanism that stitches data across paths is the DSS option, which maps subflow-level sequence numbers onto connection-level ones. It has an optional checksum whose purpose is interesting: detecting whether a middlebox unaware of MPTCP has adjusted the payload. On Linux it is off by default (net.mptcp.checksum_enabled = 0), a trade made in favour of performance. The RFC is blunt here: without checksumming, corrupt data may be delivered to the application if a middlebox alters segment boundaries or content, so checksumming is recommended unless you know the path contains no such devices. If you're speaking MPTCP over an internet path you don't control, don't skip that line.

And the fallback rule: if the SYN carries MP_CAPABLE but the SYN/ACK doesn't, the peer is treated as incapable; if the option is missing from the third packet, the session must fall back to plain single-path TCP. That strictness isn't a shortcoming — it is a deliberate choice for compatibility with middleboxes that drop TCP options.

The third door: the peer and the boxes in between

MPTCP requires end-to-end agreement. If the far end or any middlebox in between doesn't support it, the documentation says the connection is "downgraded" to plain TCP and continues on a single path. The application doesn't even notice — which is good design, but if you don't measure it you'll be fooling yourself.

The kernel is defensive about this — as long as it's new enough. net.mptcp.blackhole_timeout (3600 seconds by default) disables MPTCP on active sockets for a while when a firewall blackhole is detected, and the period grows exponentially if the problem recurs; net.mptcp.syn_retrans_before_tcp_fallback says how many SYN attempts carrying MPTCP options are made before falling back to TCP, defaulting to 2.

Now look again at my sysctl net.mptcp output above: neither of them is there. Blackhole detection arrived in 6.12 and the SYN retransmission knob in 6.14, so my 6.8 has neither reflex; path_manager is 6.15's work. The first thing to run when reading an MPTCP article is uname -r — the documentation doesn't always describe your kernel.

Knowing the reflex matters because on 6.12 and later, if your test once traversed a bad path, MPTCP staying off in later attempts may not be a configuration mistake. The place to look then is the counters: MPTcpExtBlackhole and MPTcpExtMPCapableSYNTXDisabled exist exactly for this. Fiddling with settings before measuring means chasing a bug that isn't there.

On the firewall side, expectations need correcting too. An additional subflow is a new TCP connection as far as the network is concerned: a different source address, usually a different interface. If your rule set only knows the address the first connection came from, the second path dies before it is born. net.mptcp.allow_join_initial_addr_port defaults to 1 and lets the peer send join requests to the address and port used by the initial subflow, which makes the narrowest scenario — no extra ports opened — possible. If the announcement goes unacknowledged, net.mptcp.add_addr_timeout steps in: the value (120 seconds by default) is the maximum retransmission timeout for ADD_ADDR, while the actual interval is derived from the connection's estimated round-trip time; set it to zero and it is never retried.

There's a sneakier obstacle on the server side, and it usually shows up on the first attempt: because MP_JOIN arrives with a new four-tuple, an L4 load balancer or ECMP in front may well send it to a different backend. That machine doesn't hold the token, so the join is refused and the MPTcpExtMPJoinNoTokenFound counter climbs. If you're deploying MPTCP behind a load balancer, you have to guarantee that every subflow of a session lands on the same backend.

What to expect and what not to

On the bandwidth side, expectations need to be set correctly. The kernel documentation lists aggregation as an explicit use case: using multiple paths at the same time for higher throughput, for instance combining a fixed and a mobile network to send files faster. So saying "it doesn't add up" would be wrong. But addition isn't a guarantee either: the gain depends on the latency and loss profile of the paths, on the receive buffer, and on head-of-line blocking. Combine two wildly dissimilar links and the total looks less and less like the arithmetic sum — which is also why the same list keeps "best network selection" as a separate item.

The resilience side, by contrast, is far more concrete. A subflow created with the backup flag waits quietly as long as the primary path works, and carries the session without a break when the primary goes down. How fast that switch happens is governed by net.mptcp.stale_loss_cnt: the number of retransmission intervals before a subflow is declared stale, defaulting to 4, with lower values making active-backup switching faster according to the documentation.

A little-known detail on this side is net.mptcp.close_timeout (60 seconds by default), the "make-after-break" timeout in the documentation's own words. As long as the application makes no close or shutdown syscall, the socket keeps its state for that long after the last subflow is removed, and only then moves to TCP_CLOSE. So a connection can sit path-less for a while. For it to carry on, the peer has to hold state too and a successful MP_JOIN has to arrive from the new address; if you can establish the second path before the break, you've already done better.

So if you buy MPTCP as a "bandwidth combiner" you're likely to be disappointed; look at it as "redundancy that doesn't drop the session" and it's the right tool.

Don't talk about it without measuring

The sneakiest thing about MPTCP is that everything looks normal when it isn't working. So learn the verification commands before you touch the configuration:

# list MPTCP sockets
ss -M

# watch subflow creation live
ip mptcp monitor

# client side: did I try, did the peer answer, did I fall back?
nstat -az | grep -E "MPCapableSYNTX|MPCapableSYNACKRX|MPCapableFallbackSYNACK|MPJoinSynAckRx"

# server side: did it reach me, did it complete, did it fall back, did a join arrive?
nstat -az | grep -E "MPCapableSYNRX|MPCapableACKRX|MPCapableFallbackACK|MPJoinSynRx|MPJoinNoTokenFound"

Note that the counters are directional; getting this wrong is the most common way to misread them. The naming in the kernel source is explicit: MPCapableFallbackACK counts the server-side fallback and MPCapableFallbackSYNACK the client-side one. Likewise MPJoinSynRx increments on the side that receives MP_JOIN; on the client that creates the subflow, the counter to watch is MPJoinSynAckRx.

That yields three separate diagnoses. On a client, if MPCapableSYNTX is zero the application never attempts MPTCP — the first door. If there are attempts but MPCapableFallbackSYNACK is climbing, the peer or the path is refusing — the third door. If the connection is established as MPTCP yet MPJoinSynAckRx stays at zero, no additional subflow was ever born — the second door, meaning your endpoints and limits. On a server the same three questions are asked with MPCapableSYNRX, MPCapableFallbackACK and MPJoinSynRx; since my machine is a server, the meaningful zero there is MPCapableSYNRX.

Without that split, what you're doing isn't tuning, it's guessing.

You're not condemned to blindness inside the application either. The kernel exposes four socket options at the SOL_MPTCP level (value 284): MPTCP_INFO for connection-level information, MPTCP_TCPINFO for an array of per-subflow tcp_info, MPTCP_SUBFLOW_ADDRS for the subflow addresses, and MPTCP_FULL_INFO for subflow info plus the tcp_info array plus the connection-level information together. If you write your own service, exporting "how many paths am I running on" as a metric is a few lines of work — and it's the most honest way to see on a dashboard whether one of your links has quietly died.

If your policy needs to differ per connection, that's the sign the in-kernel manager is too narrow; that's when you move to the userspace manager, mptcpd. Unnecessary complexity for most setups, but the right place if you're going to write a rule like "sessions from this customer must not use the second link".

Is this really your problem?

MPTCP's biggest constraint is organisational: you need to control both ends. If you want multi-link resilience between your own client and your own server, it's the right tool. But if you want to split traffic heading for arbitrary internet services, you're left with plain TCP unless the far end speaks it.

So the first question to ask is whether the layer of your problem really is the transport layer. Sharing traffic leaving a branch office across two links, or picking a path based on link quality, are solved in the routing layer, not the transport layer; the approach in my article about path selection with SLA probes in SD-WAN is that domain's answer. MPTCP is about a single session surviving over two paths — a different problem entirely.

If you've decided to try it, an ordered checklist helps:

Kernel and tools: sysctl net.mptcp.enabled and the flag list in ip mptcp help.
Application: IPPROTO_MPTCP in the source, SocketProtocol=mptcp on systemd 257+, or mptcpize.
Policy: signal endpoints on the server, subflow on the client; lift subflows and add_addr_accepted off zero with ip mptcp limits set.
Routing: a dev binding on every endpoint, and source-based routes where needed.
Verification: the output of ss -M and the MPCapable* counters — see both the attempt and the fallback.
Firewalls: on 6.12+ kernels, if blackhole detection has kicked in, wait out blackhole_timeout or repeat the test over a clean path.
Persistence: ip mptcp endpoint and limits are runtime settings and vanish on reboot. Don't call it done until they're wired into networkd/netplan configuration or a unit that runs at boot.

Being enabled is not being used

The real lesson MPTCP left me with isn't about the protocol. In modern systems, a feature being "supported" no longer means it is working; the support chain runs from the application to the kernel, on to the far end and through every firewall in between, and any link in that chain can quietly revert to the old behaviour.

That silence shouldn't be underestimated either. The system doesn't raise an error; it simply picks the less capable path and tells you nothing. I read that not as a flaw but as a transfer of responsibility: showing that a feature actually works is now the job of whoever measures. The 63 zeroed counters on my server are the proof — the kernel has been doing its part for years; the missing party is me.

Official Sources

Keep the Kernel Running: systemd soft-reboot

Mustafa ERBAY — Sat, 05 Sep 2026 16:45:14 +0000

Rebooting is the bluntest instrument we own. A service gets stuck, a library is updated, we say "something happened but I couldn't find out what" — and we always press the same button. That button charges the same fee regardless of how big the problem was: every process dies, the kernel stops, the hardware resets, the firmware performs its own little ceremony, the boot loader shows up, the kernel comes back. On a desktop this is called "time for a coffee"; in a server room where you stare at the screen for minutes while a disk controller introduces itself, it is called an outage.

The annoying part is this: in most of the cases where we pay that price, we have no quarrel with the hardware or the kernel at all. Our quarrel is with userspace — with services, libraries, half-finished state. The soft-reboot that systemd introduced in version 254 draws exactly that line: it rebuilds userspace from scratch and never touches the kernel.

My argument in this article is that describing soft-reboot as a "fast reboot" is both wrong and dangerous. It is a different operation with different guarantees. Anyone who uses it without knowing what it gives you — and, more importantly, what it does not — will one day discover at the worst possible moment that their most trusted tool only did half the job.

What we call a "reboot" is really seven separate jobs

While the upstream documentation lists what soft-reboot does not do, it also spells out how many stages a normal reboot goes through. A userspace reboot passes through none of the following: the second phase of a regular shutdown (systemd-shutdown), the third phase (the return to the initrd context), the hardware reboot operation, firmware initialization, boot loader initialization, kernel initialization, and initrd initialization.

Looking at that list, it becomes clear how carelessly we have been using the word "reboot" for years. We call seven jobs by a single name and pay for all of them together.

So what does systemctl soft-reboot actually do? It sends SIGTERM to any processes left running — without waiting for them to exit — and follows up with SIGKILL. If the /run/nextroot/ directory exists (it may be a regular directory, a directory mount point, or a symlink to either), it switches the file system root to it. Then it re-executes the service manager off the — now possibly new — root, which enqueues a new boot transaction just like a normal boot.

PID 1 survives, but it renews itself. The kernel has no idea any of this happened.

The survivors: /run, sockets and file descriptors

What makes soft-reboot interesting is not its speed but what comes out of the transition alive. The documentation lists these one by one under "resource pass-through".

The most valuable of them is the file descriptor store. Descriptors belonging to services that stay active until the very end are passed to the next boot and placed back into the same unit's store. For this to work, the unit must declare DefaultDependencies=no (and must not tie itself to shutdown with something like Conflicts=shutdown.target); alternatively, FileDescriptorStorePreserve= keeps the store pinned even while the unit is down.

The distinction looks thin on paper, but the consequence isn't; journald is the best illustration. On my own server running Ubuntu 24.04, journald's settings look like this:

$ systemctl show systemd-journald.service \
    -p FileDescriptorStoreMax -p FileDescriptorStorePreserve -p DefaultDependencies
FileDescriptorStoreMax=4224
FileDescriptorStorePreserve=restart
DefaultDependencies=no

$ systemd-analyze fdstore systemd-journald.service
FDNAME TYPE DEVNO     INODE RDEVNO PATH               FLAGS
stored sock 0:8   405178459 -      socket:[405178459] rw
stored sock 0:8   375205204 -      socket:[375205204] rw
stored sock 0:8   371286301 -      socket:[371286301] rw

What sits in that store are the sockets applications write their logs to. When journald is restarted it takes those sockets back, so services don't stumble with "I can't write logs". But FileDescriptorStorePreserve=restart is the default value, and it makes a narrow promise: the store is only kept across a restart of the service, and released when the service is stopped altogether.

And journald is stopped precisely during a soft-reboot transition. The upstream unit file does this deliberately, with the reasoning right there in a comment:

# To avoid journald SIGKILLed during soft-reboot and corrupting journals.
Before=soft-reboot.target
Conflicts=soft-reboot.target

So on the 255 running on my server, journald releases its store on the way into the transition. Upstream treated this as a bug and fixed it in 256 by adding FileDescriptorStorePreserve=yes to the journald unit; the commit message describes the symptom precisely: a unit using the default StandardOutput=journal lost its stdout/stderr sockets when journald stopped, because the descriptors on journald's side were not preserved.

The reason I care about this detail is that soft-reboot is not magic that works on its own. Which resources come out of the transition alive depends on lines written into individual units — and upstream itself left its own logging service out in the first release. Don't enter a transition without asking the same two questions about your own critical service: does this unit stop during the transition, and is its store preserved?

The list of survivors doesn't end there. Descriptors associated with .socket units remain open and connectible if the units aren't stopped during the transition — so an incoming request isn't refused, it waits in the queue. The /run/ file system stays mounted and populated; it is the officially recommended place to pass state between two userspace reboot cycles. File system mounts can also stay mounted if they are configured to remain until the very end of shutdown.

Even processes themselves can survive, but at a cost. The recipe in the documentation goes like this: the unit uses SurviveFinalKillSignal=yes to be skipped in that final SIGTERM/SIGKILL spree, IgnoreOnIsolate=yes so it isn't stopped on isolate, DefaultDependencies=no to step out of the normal shutdown chain, After=basic.target for correct ordering at boot, plus Conflicts= and Before= lines so it does stop on real shutdowns (reboot, poweroff, rescue, emergency).

[Unit]
Description=My Surviving Service
SurviveFinalKillSignal=yes
IgnoreOnIsolate=yes
DefaultDependencies=no
After=basic.target
Conflicts=reboot.target kexec.target poweroff.target halt.target rescue.target emergency.target
Before=shutdown.target rescue.target emergency.target

Templated units add a slice detail: since a foo@test.service instance runs in a slice named system-foo.slice by default, the same three lines must be added to that slice too. And applications that publish a service over D-Bus need to re-establish the connection, because the broker is stopped and started again.

What strikes me here is that upstream refuses to market its own feature. The documentation suggests using this pass-through sparingly, and specifically recommends avoiding letting processes survive. The reasoning is solid: as long as a process lives, code updates are necessarily incomplete, and since a process pins the file system beneath it, two versions of the OS may be held in memory at once. In other words, the enthusiasm of "let me keep everything alive" shortens the outage while making the system more fragile.

What it doesn't solve: the kernel, sysctl, hardware

In the documentation's own words there are two big gaps. First: because the kernel is not reset and keeps running, the OS update remains incomplete. Second: kernel settings under /proc/sys/ (i.e. sysctl) and /sys/ are not reset.

Don't take the second one lightly. Anyone who runs sysctl -w by hand one evening and thinks "I'll reboot in the morning, it'll clear itself" will be surprised to find the same setting still in place after a soft-reboot. The fix is what the docs also point at: keep sufficiently comprehensive /etc/sysctl.d/ files — derive the system's state from files rather than from memory.

The first gap points straight at security. soft-reboot is not an answer to a kernel CVE. There you need either a real reboot or live patching; we compared those two in detail in kernel live patching and the maintenance model in enterprise Linux. The documentation names the same two mitigations: kernel live-patching and proper sysctl files.

One more small detail that can bite: because systemd-shutdown is not executed, the executables under /usr/lib/systemd/system-shutdown/ are not executed either. If your team leaves shutdown cleanup there, soft-reboot quietly skips it.

The list left behind after `apt upgrade`

The most everyday benefit of this feature isn't in exotic image-based systems but in that familiar scene with an ordinary package manager: the upgrade finishes, the libraries on disk are new, but running processes still carry the old code in memory. On Debian and Ubuntu, needrestart is what reminds you; version 3.6 is installed on my own server and its description says the job out loud — check which daemons need to be restarted after library upgrades.

Looking at that list and typing systemctl restart one by one usually works. But the moment you miss an ordering dependency, or a process that isn't on the list is still holding the old library, you're left with a half-finished upgrade — and that half-finished state usually shows up weeks later as an unrelated-looking bug. soft-reboot is a broad broom here: it starts everything again with the new code, without paying for the firmware round.

With one condition: this broom never visits the kernel. On Ubuntu, if the /var/run/reboot-required file is there, that file is asking for a real reboot, not a soft-reboot. Don't conflate them; the first is a problem of services, the second a problem of the kernel.

Containers: the kill spree doesn't discriminate

On a host running Docker, the first question that came to my mind was whether --live-restore would save me.

It won't, because we're talking about two different things. Docker's live restore keeps containers running if the daemon crashes or is restarted deliberately. Its documented limits are clear too: it is only supported across patch releases, reconnection can break if the daemon configuration changes, Swarm services are out of scope, and if the daemon stays down long enough the default 64K logging buffer can fill up and block containers from logging.

But with soft-reboot, the daemon restarting isn't even the issue. Every remaining process gets SIGTERM and then SIGKILL; the only exceptions are units that have explicitly declared themselves exempt with the recipe above. Container processes are inside that spree. So on a container host, soft-reboot means containers restart.

Worse, live restore doesn't improve things here — it makes them harsher. With it off, containers go through the daemon's own stop flow when docker.service is stopped, so the application sees the grace period docker stop grants. With it on, containers stay alive after the daemon goes down, so the spree at the end of the transition catches them as leftover processes and they take a SIGKILL without a chance at an orderly shutdown. For a container writing data, that difference matters.

The quiet trap: /run/nextroot

There is a behavior in the release notes that doesn't draw much attention but will surprise you in operations: since systemd 255, when a reboot operation is invoked, systemctl automatically performs a soft-reboot if a new root file system has been set up under /run/nextroot/.

The phrase "set up" stays vague in the man page; the actual condition lives in the source, and it has two gates. The logind side requires /run/nextroot to be a valid OS tree — in practice, an os-release must be found inside it, so an empty directory doesn't trigger the switch. From systemd 256 onwards a second gate was added on the systemctl side: the directory must also be a mount point. The comment in the code spells out why — if the new root were stored directly on the /run tmpfs, /run/nextroot could never go away, and every reboot would soft-reboot forever.

The operational translation: in a team running an image-based update flow, what actually happens when someone types reboot depends on the machine's state at that moment. An operator who believes they applied a kernel patch may carry on with a machine whose kernel is unchanged. The ordering makes that risk worse: in logind's decision the soft-reboot branch comes before the kexec branch. Even if you prepared a new kernel with kexec --load, a reboot with nextroot set up gives you a soft reboot, and that kernel is never loaded. This detail matters most on immutable or image-based setups; in my article about adding tools to a server without leaving a trace I also touched on A/B image flows with systemd-sysupdate — in that world, preparing a new root tree and switching to it is nothing unusual.

There is an escape hatch: if the SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 environment variable is set, systemctl skips the automatic switch and performs the real reboot you asked for. The same logic exists for kexec, and SYSTEMCTL_SKIP_AUTO_KEXEC=1 turns that one off. A small but painful detail: you have to pass the variable through sudo, because the sudoers default resets the environment.

sudo SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 systemctl reboot

All of this translation also goes through the logind path. systemctl reboot --force bypasses logind, so neither the automatic soft reboot nor the kexec substitution kicks in — but --force also skips the orderly shutdown of units, so treat it as an emergency exit rather than a solution.

And run the right check; ls -ld /run/nextroot produces false positives, because an existing empty directory doesn't trigger the switch:

findmnt /run/nextroot
ls /run/nextroot/usr/lib/os-release /run/nextroot/etc/os-release

Make comparing uname -r after maintenance a habit too. If you're writing automation, state your intent explicitly: systemctl soft-reboot when you want userspace, and systemctl reboot with the escape variable when you want the kernel renewed.

systemd already took care of measuring it

I'm not going to invent a number for "how much faster?" — and there's no need, because systemd records it itself. As of version 256 the moment shutdown began is passed to the next service manager, which logs the overall "grey-out" time of the operation: from the start of shutdown until the system is fully up again. The same release added a counter of how many soft reboot cycles the system has gone through, exposed over D-Bus. In 257 that counter is also handed to generator processes as the $SYSTEMD_SOFT_REBOOTS_COUNT environment variable.

I verified on my own machine that the counter depends on the version; on a server carrying systemd 255.4 the query comes back empty:

$ systemctl --version | head -1
systemd 255 (255.4-1ubuntu8.17)

$ systemctl show -p SoftRebootsCount
$

So if you plan to build telemetry around "how many soft reboots have I done", check the systemd versions across your fleet first. As I write this the stable line is at 261.2 and the first release candidate for 262 was tagged on 1 September 2026 — but the version running on your servers is most likely whatever your distribution ships.

How to decide

The question to ask about your own setup isn't "should I use soft-reboot?" but "which layer is this maintenance actually about?" The answer picks the tool.

But the first question across a fleet is simpler: do I even have this command? soft-reboot arrived with systemd 254; before that it doesn't exist at all. Debian 12 bookworm carries 252, so the command isn't there; Debian 13 trixie carries 257, and Ubuntu 24.04 carries 255 — that's the version on my own server. Collect systemctl --version across every machine before you write the plan; on a mixed fleet that single line decides whether the runbook is applicable at all.

Work that stays in userspace — library and service updates, applying a configuration change everywhere, the distribution's "restart these 14 services" list, opening a wedged userspace with a clean slate, switching to a new root tree on an image-based system — is soft-reboot's natural territory. Whenever you suspect the kernel, a driver or module, sysctl, or the hardware, a real reboot is the only honest answer.

Don't forget the step in between either: systemctl kexec. This command shuts down and reboots the system via kexec; it loads a kexec kernel if one isn't loaded yet, or fails if it can't (--force falls back to a normal reboot). That automatic loading has an easily missed prerequisite: because the enumeration follows the Boot Loader Specification, the system must be using UEFI and the boot loader entries must be configured appropriately — bootctl list shows you what it sees. On a classic BIOS/GRUB install, this command won't load a kernel for you unless you prepared one by hand with kexec --load. Loading it by hand is also what you want when you need a custom initrd or extra kernel command line options. So you have three steps available: soft-reboot never touches the kernel, kexec renews the kernel but skips the firmware and boot loader round, and a full reboot resets everything. If you suspect the hardware, nothing but the third will do the job. If only the kernel needs patching and the machine must never go down, live patching is the fourth option on the table.

Run the first attempt on a test machine born from the same image rather than in production, and watch what the log stream, socket connections and database clients do on the way back. Two preparations are mandatory: out-of-band access (your SSH session dies during the transition, and if a unit gets stuck on the way back you're left without a console) and making sure the services that must come up at boot are actually enabled — the system doesn't remember what you started by hand, it builds a normal boot transaction.

A short checklist for the first run:

Is out-of-band console access (IPMI/BMC/provider console) open and verified?
Have the services that must come up been confirmed with systemctl is-enabled?
Does /run/nextroot exist — i.e. what would actually happen if you typed reboot?
Is there a pending kernel-patch marker (/var/run/reboot-required) sitting there?
Have you written down the three post-transition checks (systemctl --failed, health endpoints, version verification)?

One more note: the man page says systemd-soft-reboot.service and related units should never be executed directly. The correct entry point is systemctl soft-reboot.

Get to know the failure modes in advance too; they all follow from the nature of the transition. A service that depends on an old file system pinned by another surviving process may meet an unexpected version at startup. A client speaking over D-Bus may have lost its connection — because the broker restarted — and may not reconnect on its own. Lock files you left in /run/ stay where they are even though their owner is dead, because /run/ isn't cleared during the transition: a gift if you want to pass state, a trap if you delegate lock cleanup to boot.

As for a rollback plan: there isn't one. If userspace doesn't come back after a soft-reboot, there is no button that says "return to the previous state"; your only path is a real reboot from the console. On a mutable root, that reboot brings you back to the upgraded packages, not to the old version. So your actual rollback mechanism isn't soft-reboot at all — it's whatever update method sits beneath it: A/B images, snapshots, or package pinning.

There is no single thing called a restart

What soft-reboot really taught me isn't a command but a distinction. For years we treated "restart" as one operation and bought the most expensive version of it every time. Yet a system's life isn't measured by a single clock: the kernel has its clock, userspace has another, and individual services run on faster ones still. Knowing which clock you need to reset in order to fix a problem is half the work.

The real gain here, in my view, isn't shaving a few minutes off an outage. The real gain is replacing the "turn it all off and on again" reflex with the question "which layer broke?" Once you start asking that, the next step arrives on its own — and sometimes the answer is to not touch the machine at all.

DEV Community: Mustafa ERBAY

systemd-nspawn: Containers Without Images or Registries

What does it isolate?

The one condition: user namespaces

No images, just directories

The first container: four commands

Networking: shared by default

Running it as a service

It also runs OCI bundles

The state on my server

Instead of Docker, or alongside it?

Checklist

A container isn't one thing

Official Sources

The Clock You Can't Stop: PostgreSQL XID Wraparound

What the queries return on a trouble-free cluster

The thresholds: four separate lines

What the server says when the alarm rings

Why does age grow? Usually vacuum isn't the culprit

Insert-only tables: the classic trap

Lock conflicts: what stops vacuum

What not to do

The cost on the storage side

Monitoring: a one-line insurance policy

Counters fill quietly

Official Sources

Lazy Pulling: The Part of the Image Nobody Reads

What happens on my own server

eStargz: prioritised files and a TOC

zstd:chunked: skippable frames and range requests

SOCI: lazy loading without conversion

The price: which guarantee, in which stack?

Where does it actually pay off?

Checklist before you try it

Pointing instead of copying

Official Sources

Page Reclaim Changed Quietly: MGLRU

The old model and the new one

Three bits in the enabled file

Thrashing prevention: min_ttl_ms

Working set estimation

Working set per container

Where MGLRU stands in 2026

So what did I gain?

Checklist

Defaults change quietly

Official Sources

Setting Up Immich: The Operational Cost of Google Photos Convenience

What Is Immich and What Are Its System Requirements?

Deployment with Docker Compose (Sample Scenario)

Storage and Network Cost Analysis

Performance Monitoring and Defining SLOs

Rollback and Update Strategy

Cost Comparison with Google Photos

Data Backup Strategy

Security and Compliance Controls

Operational Cost Breakdown

Conclusion

Official Sources

fanotify: Watching File Access — and Stopping It

Two APIs, two mindsets

The real difference: the power to permit

Lost events and an overflowing queue

Choosing between descriptor and handle

The things it doesn't see

Locking yourself out

The state on my server

What changed in recent releases

When fanotify, and when not?

Checklist before you build

The difference between watching and deciding

Official Sources

Sharing IO: io.max, io.weight and io.latency

First, see what you actually have

Three mechanisms, three questions

io.max: the hard ceiling

io.weight: fairness, but only with a cost model

io.latency: protection, not punishment

The layer that's easy to skip: writeback

Which one, when?

Three bits in the `enabled` file

Thrashing prevention: `min_ttl_ms`

The list left behind after `apt upgrade`