The Watcher Raised ENOSPC for 48 Hours. df Still Showed Free Blocks.

Have you ever treated errno 28 as a disk emergency on a box that still had gigabytes free? I did that for forty-eight hours, and the volume never ran out of blocks or inodes. The test helper watched fixture directories, my laptop stayed green, and the second Linux shell died during setup. Why did every cleanup ritual make df look healthier while the watcher still crashed on start?

What I thought I was building

I wanted a small harness that restarted a worker whenever a fixture file changed on disk. That sounded like an afternoon chore, not a two-day hunt through storage myths and leftover caches. I asked a coding assistant for a recursive Linux watcher and dropped the sketch into a pytest helper. I also wanted a second Linux environment that did not inherit my laptop's generous defaults.

Hour 0–8: I cleaned the wrong resource

The remote run failed in setup, before any assertion had a chance to speak. The traceback pointed at watchdog's Observer, and the message was the classic “No space left on device.” I did what muscle memory always does when errno 28 appears, and I started deleting things.

Here is the exact sequence I wasted a workday on:

1. Ran df -h and df -i, then stared at healthy percentages like they were lying to me.
2. Wiped pytest caches, __pycache__ trees, and everything under /tmp that looked disposable.
3. Blamed an overlay filesystem, then reran the helper on a bind mount that still crashed.
4. Asked the assistant again; it doubled down on disk pressure and log rotation.
5. Added strace -e file and finally saw the failing call was not write().

# the checks that kept saying the disk was fine
df -h .
df -i .
du -sh .pytest_cache /tmp 2>/dev/null

# the call that actually failed, once I stopped tracing the wrong family
strace -f -e inotify_add_watch,inotify_init1 python watcher_lab.py

If df is green and write() never shows up, why are we still talking about disks? Because Linux reused ENOSPC for inotify exhaustion, and I let that alias steer the whole investigation.

Hour 8–24: the counters I should have read first

Linux budgets three inotify resources per user namespace, and none of them appear in df. I only opened these files after I had already emptied /tmp twice and written a fake cleanup runbook. Would you have looked here first, or would you also have trusted the errno text?

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_queued_events

# rough count of inotify instances already held by this user
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

max_user_watches is the ceiling on watched paths, not a disk quota. max_user_instances caps how many inotify handles you may open. Recursive watchers multiply both numbers because each subdirectory becomes another watch. My laptop had headroom. The constrained shell did not, especially once IDEs, test runners, and leftover observers were already alive.

The assistant's sketch scheduled one Observer on a fixture root with recursive=True. That is convenient until the tree contains hundreds of directories, or until a previous crashed run never called stop(). Do you join those threads in a pytest fixture teardown, or do you leak a watch set into the next test?

Hour 24–48: a lab that fails on purpose

I stopped arguing with production-shaped mystery boxes and wrote a lab that can hit the ceiling without pretending the disk is full. Pin nothing exotic; this is just watchdog plus a directory fan-out. Label this as a local reproduction, not a benchmark and not a claim about anyone's hosted hardware.

# watcher_lab.py — labeled lab harness, not a production service
from pathlib import Path
import os
import sys
import tempfile
import time

from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer

class Quiet(FileSystemEventHandler):
    def on_any_event(self, event):
        return

def fanout(root: Path, n: int) -> None:
    for i in range(n):
        (root / f"w{i:04d}").mkdir()

def main() -> None:
    n = int(os.environ.get("WATCH_FANOUT", "512"))
    root = Path(tempfile.mkdtemp(prefix="inotify-lab-"))
    fanout(root, n)
    observers = []
    try:
        for i in range(n):
            obs = Observer()
            obs.schedule(Quiet(), str(root / f"w{i:04d}"), recursive=False)
            obs.start()
            observers.append(obs)
        print(f"started {n} observers under {root}")
        time.sleep(2)
    except OSError as exc:
        print(f"OSError errno={exc.errno} ({os.strerror(exc.errno)}) after {len(observers)} observers", file=sys.stderr)
        raise
    finally:
        for obs in observers:
            obs.stop()
        for obs in observers:
            obs.join(timeout=2)

if __name__ == "__main__":
    main()

If you can set sysctls in a container, shrink the budget so the failure happens quickly. Some engines reject that sysctl; treat the next block as optional, and fall back to raising WATCH_FANOUT until the OSError appears.

pip install 'watchdog>=4,<6'

# optional: a tiny watch budget, when the runtime allows it
docker run --rm --sysctl fs.inotify.max_user_watches=128 \
  -v "$PWD":/work -w /work python:3.12 \
  bash -lc "pip install -q 'watchdog>=4,<6' && WATCH_FANOUT=64 python watcher_lab.py"

When it breaks correctly, you should see errno 28 from inotify_add_watch, while df -h still reports free blocks. That mismatch is the whole lesson. If the container cannot change the sysctl, read the three /proc files on that box and size the fan-out against the remaining budget instead of guessing.

Polling is the boring fix I should have reached for

For tests, I did not need kernel events. I needed a worker restart that was merely correct. PollingObserver spends CPU and adds latency, but it does not consume inotify watches, and it behaves more similarly across laptop and constrained shells.

from watchdog.observers.polling import PollingObserver

obs = PollingObserver(timeout=1)
obs.schedule(Quiet(), str(root), recursive=False)
obs.start()

Even better for pytest: skip watching entirely and call a reload function after the fixture write. If the production code under test already exposes a reload() hook, why am I teaching the kernel about my temp directories?

Decision table I now keep in the notes

Symptom	Looks like	Check next	Safer default in tests
OSError: [Errno 28] No space left on device during Observer.start	Full disk	df -h, then immediately the three inotify /proc files	PollingObserver or an explicit reload()
Helper hangs after a failed run	Deadlock	leftover Observer threads, missing stop()/join()	fixture finalizer that always stops watchers
Works on a laptop, fails on a shared Linux shell	“environment drift”	watch headroom, recursive directory count	non-recursive watch on one directory
Assistant-generated watcher looks tiny	Low risk	recursive=True on a fat tree	do not watch trees you did not create

What broke, and what I would repeat

The helper was not wrong about files changing. It was wrong about the kernel resource it spent to notice. Recursive inotify is a production-shaped shortcut that tests rarely need, and errno 28 will gaslight you until you print os.strerror next to df.

I would repeat four moves, in this order, before deleting another cache:

• Print exc.errno and os.strerror(exc.errno) in the test helper, not only the exception text.
• Cat the three inotify counters on every Linux environment I claim is “the same.”
• Count directories before I pass recursive=True, and refuse to watch trees I did not create.
• Prefer an explicit reload hook in unit tests, then polling, and only then a kernel watcher.

I would not raise max_user_watches on a shared host just to keep a pytest helper alive. That hides the leak and taxes every other user of the namespace. I also would not take a generated Observer snippet as proof that the environment can afford it.

Limitations, and who should not copy this

This field note is Linux-specific. macOS and Windows file events travel through different APIs, so an ENOSPC story from inotify will not travel with you. The Docker --sysctl path is optional and may be rejected; it is a lab knob, not a portability guarantee. I did not measure latency, throughput, or any hosted quota, and you should not treat the fan-out script as a benchmark.

Do not use recursive inotify in unit tests if you lack rights to inspect /proc and you cannot tolerate setup flakes. Do not use polling on a huge tree inside a latency-sensitive service without measuring CPU. If you need sub-second reloads on a shared CI worker, push for an explicit reload API instead of bargaining with watch ceilings you do not own.

Would I still generate the first watcher draft with a free model? Yes, but I would paste the errno table beside the prompt and ask it to fail closed. The forty-eight hours were not about file events being hard. They were about trusting a disk error that never named a disk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server option is enough to reproduce the setup.