The 204,800 Everyone Copies Costs 295 MiB of Kernel Memory

When a file watcher dies with "No space left on device", the internet's answer is one line: set fs.inotify.max_user_watches to 204800. Syncthing's own documentation recommends that number. You paste the line, the watcher starts, the case closes.

What nobody mentions is that the number is a memory budget. The cost per watch, on the bench: around 1.5 KiB on ext4, 2.45 KiB on a container's own layer. Actually installing 204,800 watches therefore means 295 MiB of kernel memory — and drop_caches will not give it back. Raising the limit is free; what is free is the permission, not the bill.

Error messages that lie are well-trodden ground on this blog: df reported zero while root kept writing, and a single fork error message hid four separate ceilings. Both inotify limits belong to that family, so I will pass through them quickly and get to the real question: if you use what you were permitted, what do you pay?

The rig

Measurements come from Docker Desktop's Linux virtual machine on my own Mac: kernel 6.10.14-linuxkit, 14 vCPUs, MemTotal: 5036592 kB (4.8 GiB). Because the experiments write to host sysctls and call drop_caches, no customer server and not this blog's own server took part.

The containers ran like this — budget tests in an ordinary container, memory tests --privileged, since they write to drop_caches:

# budget test: ordinary container
docker run --rm -v "$LAB":/lab:ro python:3.12-slim python3 /lab/watchbomb.py watches 20000 /tmp/w
# memory test: privileged, on a real ext4 volume
docker run --rm --privileged -v inolab:/ext4lab -v "$LAB":/lab:ro python:3.12-slim \
  python3 -u /lab/bellek_ext4.py 150000

There is no library on the watcher side; ctypes calls libc directly, because I want to see which errno stopped me without asking any wrapper:

libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
fd = libc.inotify_init1(0)
wd = libc.inotify_add_watch(fd, path.encode(), IN_ATTRIB)
if wd < 0:
    err = ctypes.get_errno()      # raw errno, no interpretation

Every watch went on a separate inode. Add a second watch to the same inode and the kernel allocates nothing new, it updates the existing record (inotify_update_watch() tries inotify_update_existing_watch() first and falls through to inotify_new_watch() on -ENOENT). What drains the budget is not the number of files but the number of distinct inodes — for a tool that watches directories, that means the directory count.

The lab scripts print Turkish keys; each output block below is glossed in the line that follows it.

Two limits, two borrowed errnos

For the budget tests I lowered the limit to the classic field value, 8192. As root in a container, watching 20,000 files:

eklenen_watch=8172
durma_errno=28 (No space left on device)

(eklenen_watch = watches added; durma_errno = the errno that stopped it.) At that moment the container's disk — the VM's thin-provisioned virtual disk, so not the Mac's real free space, but what df sees:

overlay        1007G   21G  936G   3% /

The second limit is less well known: how many inotify instances (descriptors) you may open. This ran under a separate uid (1002) so the bucket would be clean:

ulimit_n=1048576
acilan_instance=8192
durma_errno=24 (Too many open files)

(acilan_instance = instances opened.) EMFILE, "too many open files" — yet the process holds 8,192 descriptors and the descriptor limit is 1,048,576. The reflex here is to raise ulimit -n; nothing changes, because the wall is fs.inotify.max_user_instances. (That 1,048,576 happening to equal the ceiling of the watch limit is a coincidence; they are separate resources.) The source is explicit about both:

if (!inc_inotify_watches(group->inotify_data.ucounts)) {
    ret = -ENOSPC;
...
group->inotify_data.ucounts = inc_ucount(current_user_ns(),
                                         current_euid(),
                                         UCOUNT_INOTIFY_INSTANCES);
if (!group->inotify_data.ucounts) {
    return ERR_PTR(-EMFILE);

The kernel's own default for the instance limit is 128; my VM reports 8192, so somebody raised it.

The bucket is not on the "real user", it is on the euid

The man page uses the same phrase for both limits: per real user ID. The source says something else — the second argument in the call above is current_euid(), the effective user ID. The watch side uses the same bucket, because inc_inotify_watches() counts against the ucounts record frozen at inotify_init time.

That gap between documentation and code does not stay on paper. The classic privilege-dropping service pattern — start as root, call inotify_init(), then setuid() down — charges its watches to the old euid's bucket, and the user you dropped to is never blamed. For the same reason, the advice to "run the watcher under its own uid" only works if the uid change happens before the inotify descriptor is opened.

In my own runs the real and effective uids were identical, so the bench does not separate the two; this claim rests on the source, not on a measurement.

What the per-user bucket means in practice, though, did get measured. While the first watcher held its 8,172 watches, a completely separate container started next to it — separate PID, mount and network namespaces, nothing running inside:

eklenen_watch=0
durma_errno=28 (No space left on device)

Zero. A spotless container, unable to watch a single file; the only thing they share is uid 0. The pattern itself is not new here — when a neighbouring container shrank my pipes eightfold the same thing happened to pipe buffers, and there no error came back at all.

A user namespace does not rescue you either: after filling uid 1000's bucket, becoming root in a fresh namespace via unshare -U -r still returned zero. inc_ucount() walks up the chain and hits the parent's ceiling; the child namespace's own ceiling is INT_MAX, so the refusal comes from the parent, not the child. One caveat: unshare -r maps the inner 0 to the same outer uid, which is why no fresh bucket appears. A rootless setup that maps into a genuinely different range via /etc/subuid (say podman --userns=auto) lands on a different kuid and does get a fresh bucket.

Who spent the budget before me

That 8,172 is the interesting part, because the limit was 8192. Counting the missing 20 across the whole VM, grouped by uid:

uid=0 pid=84337 comm=python3          izleme=8172
uid=0 pid=362   comm=dockerd          izleme=1
uid=0 pid=361   comm=rpc.idmapd       izleme=1
uid=0 pid=24277 comm=containerd-shim  izleme=2
uid=0 pid=30789 comm=containerd-shim  izleme=2
uid=0 pid=797   comm=containerd-shim  izleme=2
uid=0 pid=800   comm=containerd-shim  izleme=2
uid=0 pid=803   comm=containerd-shim  izleme=2
uid=0 pid=804   comm=containerd-shim  izleme=2
uid=0 pid=83354 comm=containerd-shim  izleme=2
uid=0 pid=84318 comm=containerd-shim  izleme=2
uid=0 pid=96409 comm=containerd-shim  izleme=2
TOPLAM_IZLEME=8192

(izleme = watches held by that descriptor; TOPLAM_IZLEME = total watches.) Nine containerd-shim processes at two each, dockerd and rpc.idmapd at one each: 20. Add my 8,172 and the total is exactly 8192. The container infrastructure eats from the same bucket. At a limit of 8192, twenty watches are noise; a capacity plan that budgets 650 watches for a 650-directory tree fails by a hair.

Here is the counting script, with two caveats:

tot=0
for f in /proc/[0-9]*/fdinfo/*; do
  [ -r "$f" ] || continue
  n=$(awk '/^inotify /{c++} END{print c+0}' "$f" 2>/dev/null)
  [ -z "$n" ] && n=0            # the process may vanish mid-loop
  [ "$n" -gt 0 ] || continue
  pid=$(echo "$f" | cut -d/ -f3)
  uid=$(awk '/^Uid:/{print $2}' "/proc/$pid/status" 2>/dev/null)
  [ -z "$uid" ] && continue
  echo "uid=$uid pid=$pid comm=$(cat /proc/$pid/comm 2>/dev/null) watches=$n"
  tot=$((tot+n))
done
echo "TOTAL_WATCHES=$tot"

First: because the budget is per user, the lines have to be summed per uid; a raw total mixes every user on the machine. Second: after a fork the same fsnotify_group shows up under two descriptors — the kernel counts it once, this loop counts it twice. On a machine running containers the loop belongs on the host; from inside a container you cannot see the neighbours you share the bucket with.

What one watch costs

This was the real question. First the watch record itself: create the files, stat them all (so inodes and dentries are already cached), read Slab, then add the watches and read again. For 150,000 watches the increase was 15.8 MiB, or 110.3 bytes per watch — and almost all of it in SUnreclaim, the class that cannot be reclaimed under memory pressure. Noise stayed at 20 kB. The kernel's own object size for that record agrees:

$ cat /sys/kernel/slab/inotify_inode_mark/object_size
80

An 80-byte object against 110 bytes on the bench; the gap is the connector structure allocated per watched inode, plus slab overhead.

In real life, though, the inodes of the files you watch are not sitting ready. The second rig creates the files, flushes the cache with drop_caches, takes a cold baseline, and only then adds the watches. On a genuine ext4 volume, 150,000 watches, three runs:

Run	Slab increase	Per watch
1	244.5 MiB	1,709.5 bytes
2	215.1 MiB	1,503.8 bytes
3	216.1 MiB	1,510.8 bytes (median)

Runs two and three land within 0.5% of each other; the first is the outlier. The number to work with is ~1.5 KiB per watch. On the container's own writable layer (overlay) the same bench gave 2,450.6 and 2,458.4 bytes across two runs — a watch there is roughly 1.6× more expensive.

Where does the difference come from? A watch holds on to the inode it watches. The kernel's own comment says so: "An inotify watch requires allocating an inotify_inode_mark structure as well as pinning the watched inode." The object sizes on this machine complete the arithmetic: ext4_inode_cache 1,168 bytes, dentry 192, inotify_inode_mark 80.

So is that memory truly held, or would pressure reclaim it, as its place in KReclaimable suggests? Calling drop_caches with the watches still in hand, across three runs:

Run	Increase	drop_caches returned	On closing the fd
1	250,412 kB	48,076 kB (19.2%)	all returned
2	220,288 kB	0 (Slab grew by 3,972 kB)	all returned
3	221,304 kB	28,564 kB (12.9%)	all returned

At best the kernel handed back a fifth; in one run it handed back nothing. The remainder stayed for as long as the watcher lived, and in all three runs the whole lot was freed the instant the descriptor closed. The optimism in the KReclaimable column does not survive contact with a live watch.

One caveat: 1.5 KiB is the upper bound, for a watcher that never touches the files. In a tool that opens and reads what it watches — a compiler's watch mode, a log shipper — those inodes would be resident anyway, and the marginal cost of a watch approaches 110 bytes.

What to do about the limit

Now the multiplication from the opening. At the median unit cost, 204,800 watches come to 295.1 MiB (333.9 MiB with the outlier run; roughly 480 MiB on a container layer). Filling the 1,048,576 my VM reports would be 1.48 GiB — 30.7% of a 4.8 GiB machine.

That 1,048,576 is a story of its own. The kernel computes it at boot from RAM: 1% of memory, divided by INOTIFY_WATCH_COST, clamped to [8192, 1048576]. The cost constant in the source is one record plus two inodes, which on this machine's object sizes is 80 + 2×624 = 1,328 bytes. Since 1% of 4.8 GiB is 51,572,736 bytes, the number the kernel would arrive at is 38,834. The reported value is twenty-seven times that, so it did not come from the arithmetic — it was written. I checked /etc/sysctl.conf, /etc/sysctl.d/ and /lib/sysctl.d/ in the VM (/run, /usr/lib and /usr/local/lib have no such directory in this image) and found no inotify line. The author stays unidentified — though a privileged container overwriting a host sysctl has happened to me on this very machine before: when I went looking for who changed fs.file-max, the trail led into a container.

My decision framework:

• Count before you raise. The loop above gives current consumption broken down by uid. A watcher that dies at 8192 rarely needs 204,800; the number it needs is usually your tree's directory count plus the few dozen the infrastructure eats.
• Do the multiplication. Target watches × ~1.5 KiB (≈2.45 KiB on a container layer) = kernel memory that will not be reclaimed while the watcher lives. Decide there whether that is acceptable against the machine's RAM.
• You cannot give a container its own budget. fs.inotify.* is not namespaced: /proc/sys is read-only inside the container, and docker run --sysctl fs.inotify.max_user_watches=... refuses outright ("sysctl ... is not allowed"). The only addresses are the node's own sysctl, or a different uid.
• A separate uid, but before the descriptor. Because the bucket is charged to the euid at inotify_init time, ordering matters in privilege-dropping services.
• If the tree is large, change the instrument. inotify is the wrong one for hundreds of thousands of directories; fanotify watches an entire mount point with a single mark, at the price of CAP_SYS_ADMIN.

What I did not prove

The measurements come from a single kernel (6.10.14-linuxkit) on a single machine. Since the size of struct inode varies with build options, both the 1.5 KiB figure and the 1,328-byte cost constant belong to this machine and should be re-read elsewhere via /sys/kernel/slab/.../object_size. The 1.48 GiB figure was extrapolated linearly from 150,000 watches; a million were never actually installed. The 110-byte figure rests on a single run. The max_queued_events limit and the IN_Q_OVERFLOW event that arrives on overflow went untested. The euid claim rests on the source; no experiment here separates real from effective uid. And one more: my first "ext4" bench was silently running on overlay because a bind mount had failed, and I was about to publish the claim that both filesystems cost the same — repeating it with a named volume is what exposed the difference.

Conclusion

inotify's limits are not badly documented; what is bad is mistaking them for a capacity measurement. The number in max_user_watches is not how many watches your machine can carry, it is a permission somebody granted — and filling that permission has a price measured in gigabytes, one the kernel will not hand back under pressure.

The habit worth taking from this: before raising a sysctl, measure its unit cost and multiply by your target. It is two lines of work, and more often than not the answer comes back "don't raise it, lower it".

Official Sources

• fs/notify/inotify/inotify_user.c (Linux, the commit behind tag v6.10) — where ENOSPC/EMFILE are returned, current_euid(), INOTIFY_WATCH_COST, and how the defaults are computed
• kernel/ucount.c (Linux, the commit behind tag v6.10) — inc_ucount() walking the namespace chain, and its rollback loop
• Syncthing FAQ — inotify limits — the 8192 limit in the field and the 204800 recommendation