When 'No space left on device' has nothing to do with disk space

You're running a log shipper or a config reloader and suddenly it starts failing with ENOSPC — "No space left on device." You check df -h, and the disk is fine. Plenty of inodes, plenty of blocks. So why is the kernel refusing to do work?

Chances are it's not disk space at all. It's the inotify watch limit.

inotify is the kernel's file change notification mechanism. Every directory you watch with tools like inotifywait, a config reloader, or a log shipper consumes a "watch." Those watches come from a finite pool, and when you exhaust it, the kernel returns ENOSPC even though the filesystem is nowhere near full. The error message is misleading, and it's easy to burn an hour chasing disk space that was never the problem.

Check the limits first

Two sysctls control this:

# Max number of watches (default is often 8192 or 65536)
sysctl fs.inotify.max_user_watches

# Max number of queued events per instance
sysctl fs.inotify.max_user_instances

The one that bites most people is max_user_watches. If you're running a bunch of containers or a monitoring agent that watches a large directory tree, 8192 goes fast.

Confirm it's actually inotify

If you're getting ENOSPC and the disk is fine, check the kernel logs:

dmesg | grep -i inotify
journalctl -k | grep -i inotify

You'll often see something like:

inotify watch limit reached

That's your smoking gun. It's not disk space, it's the watch table.

How many watches do you actually need?

Rough rule of thumb: one watch per directory you're monitoring. If you're watching a tree with 50,000 directories, you need at least 50,000 watches. You can get a rough count of what a tool is using with strace, but for planning purposes, size it against the directory count of whatever you're shipping logs from.

Fix it persistently

The one-liner that works until reboot:

sysctl -w fs.inotify.max_user_watches=524288

But if you want it to survive a reboot, put it in a sysctl drop-in file:

echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
sysctl --system

sysctl --system applies everything in /etc/sysctl.d/, so it takes effect immediately and sticks across reboots. On systemd systems this is the clean way to do it — don't edit /etc/sysctl.conf directly when a drop-in file does the job and keeps your change obvious.

The part people forget

Raising the watch limit is the easy half. The other thing to check is max_user_instances — the number of separate inotify instances a single user can create. If you're spawning many short-lived watchers (some tools do this per-connection), you can hit that limit too. Bump it in the same drop-in file if you need to.

Also remember: watches are a per-user resource, and the limit applies to your user, not the whole box. If you run monitoring as root, root has its own budget. Container runtimes often share this, so a busy host can blow through it faster than you'd expect.

The takeaway

When a monitoring tool fails with "No space left on device" and the disk is fine, don't fight the filesystem. Check fs.inotify.max_user_watches, confirm with dmesg, and raise it via a sysctl drop-in file. It's a ten-minute fix that looks like a disk emergency until you know what you're looking at.