I Debugged a Silent Collector for 48 Hours. inotify Still Owned the Old Inode.

Have you ever watched a log collector go silent while the service itself kept claiming it was healthy? I burned forty-eight hours on that mismatch, and the disk never actually stopped receiving those bytes at all. Linux had renamed the inode my watcher still owned, then created a fresh empty path with the same name. The dashboard looked dead because I had followed a name as if it were a file descriptor.

I accused the process before I accused the path

I did what any tired on-call brain does, and I accused the application of stalling first. Was the writer blocked on a full volume, a stuck pipe, or a forgotten debug sleep? I walked /proc, lsof, and strace expecting ENOSPC or a thread stuck in write(). Every write returned success, which is a rude way for a filesystem to tell you that you are looking at the wrong object.

These are the commands I actually ran, in the order that still makes sense to me:

pid=$(pgrep -n logdemo)
ls -l /proc/$pid/fd
ls -li /var/tmp/loglab/app.log
stat /var/tmp/loglab/app.log
lsof -a -p "$pid" | grep -E 'log|deleted'

The open descriptor kept growing, but the path I was tailing stayed suspiciously still after each write. Have you trained yourself to stat both sides of that relationship, or do you still trust the filename?

Rotation did the documented thing

Once the sizes diverged, I finally opened the rotate config and stopped hoping for a missing signal. Did the job use create, which swaps the path, or copytruncate, which keeps the inode and hopes nobody writes mid-copy? I had copied create 0644 into a snippet months ago because it looked tidy in somebody's tutorial.

Proposed lab snippet only; do not drop this onto a host you care about:

/var/tmp/loglab/app.log {
    size 1k
    rotate 3
    missingok
    nocompress
    create 0644
}

After a rename of app.log, any process that already called open() still holds the previous inode. A watcher that subscribed to that inode will never see bytes land on the new path. Why do we keep talking about the log file as if the string /var/log/app.log were a capability?

The laptop lied for a night

This is the stretch of the forty-eight hours that I would rather not admit in public. I reproduced the watcher on a laptop where file events are not Linux inotify, and the path appeared to recover after rotation. Why was the same Python calm at my desk and mute once I moved onto a server image? I had been testing the wrong event API, and the laptop flattered a name-based mental model. I needed a throwaway Linux box that actually implements inotify, without turning a two-day inode argument into a billing project.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the harness, then ran it on the free server option for a real Linux inotify surface. Those two availability notes are the only product claims I can honestly stand behind in this write-up. I will not invent model names, quotas, machine shapes, or how long any free tier remains in place. The first draft skipped IN_DELETE_SELF and never re-registered the watch, which is how this bug ships again.

Artifact: fail the test until the inodes split

Label the following script as a reconstruction you can run locally, not as telemetry from some unnamed outage. The assertion is the whole lesson: after rename-and-recreate, the writer fd and the path must disagree. Run it on Linux, because the split is a POSIX inode story more than a Python story.

#!/usr/bin/env python3
"""inode_lab.py — reconstruction only. Linux assumed. Not a collector."""
from __future__ import annotations

import os
import sys
import time
import subprocess
import tempfile
from pathlib import Path

WRITER = r'''
import os, sys, time
path = sys.argv[1]
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
sys.stdout.write(str(os.fstat(fd).st_ino) + "\n")
sys.stdout.flush()
n = 0
while True:
    os.write(fd, ("line-%d\n" % n).encode())
    n += 1
    time.sleep(0.05)
'''

def main() -> None:
    root = Path(tempfile.mkdtemp(prefix="loglab-"))
    path = root / "app.log"
    path.write_bytes(b"")
    proc = subprocess.Popen(
        [sys.executable, "-c", WRITER, str(path)],
        stdout=subprocess.PIPE,
        text=True,
    )
    assert proc.stdout is not None
    writer_ino = int(proc.stdout.readline().strip())
    time.sleep(0.2)
    path_before = path.stat().st_ino
    if path_before != writer_ino:
        raise SystemExit(f"setup mismatch path={path_before} fd={writer_ino}")
    rotated = root / "app.log.1"
    path.rename(rotated)
    path.write_bytes(b"")
    path_after = path.stat().st_ino
    rotated_ino = rotated.stat().st_ino
    print(f"root={root}")
    print(f"writer_fd_inode={writer_ino}")
    print(f"path_inode_before={path_before}")
    print(f"path_inode_after={path_after}")
    print(f"rotated_inode={rotated_ino}")
    if path_after == writer_ino:
        raise SystemExit("FAIL: path still shares the writer inode")
    if rotated_ino != writer_ino:
        raise SystemExit("FAIL: writer did not keep the rotated inode")
    proc.kill()
    proc.wait()
    print("PASS: rename split the path from the writer inode")

if __name__ == "__main__":
    main()

Optional inotifywait check

If you have inotify-tools installed, the adjacent check below makes the mute watcher visible without extra Python bindings. Export ROOT from the script output, then run the file watch and the directory watch in two terminals.

# Proposed lab commands. File watch is the failure mode.
inotifywait -m -e modify,move_self,delete_self "$ROOT/app.log"

# Directory watch is the recovery path I would actually keep.
inotifywait -m -e create,moved_to,moved_from,modify "$ROOT"

# Rotate the way logrotate `create` does.
mv "$ROOT/app.log" "$ROOT/app.log.1"
: > "$ROOT/app.log"
ls -li "$ROOT"

Notice that tail -f follows the descriptor, while tail -F retries the name after it disappears. I had collapsed those two flags in my head for years, and the man page is not subtle about the difference. Which flag do your incident runbooks actually recommend when a collector goes quiet after a rotation event?

A directory watcher still has to reopen the path. Proposed recovery steps, not production code:

1. Watch the parent directory for CREATE, MOVED_TO, and MOVED_FROM on the target name.
2. On those events, close the old fd, open the path again, and print both inodes on one line.
3. If DELETE_SELF or MOVE_SELF arrives on a file watch, treat the watch as dead until you add it again.
4. Fail the test when path inode and fd inode match after a rename-and-recreate cycle.

The table I wanted on hour one

I wish I had kept this table on hour one, instead of arguing with the application logs.

Symptom	What I thought	What to check	Repeatable action
File inotify goes mute after rotate	App crashed	ls -li plus /proc/<pid>/fd	Watch the directory, then reopen on create
tail -F recovers, my Python does not	Language bug	Did I use -F or -f?	-F retries the path; -f follows the fd
Works on a laptop, fails on Linux	Flaky image	Event API	Reproduce on inotify, not a path-fuzzy desktop API
Disk usage grows after rotate	Rotator failed	lsof lines marked deleted	Signal the writer to reopen, or accept copytruncate loss
Generated watcher looks complete	I can ship it	Does it re-add the watch?	Keep the inode assertion in CI

What broke, and what I would repeat

The writer never crashed, and the rotator never failed in any way the man page would recognize as failure. My collector treated a path string as a stable capability, even though inode identity is the actual capability. Once I said that out loud, the silent dashboard stopped looking like a mystery and started looking like a contract.

What I would repeat on the next quiet collector:

• Print st_ino from the path and from the open fd in the same log line, every time the collector starts.
• Watch the parent directory, then reopen when the target name is created, moved in, or replaced.
• Reproduce file events on Linux, because a desktop event API will compliment a watcher that still follows names.

I would not paste a generated event mask into a collector until this lab shows two inodes on purpose. Would you really ship a collector that cannot print the inode it thinks it currently owns?

Limitations, and who should skip this

This lab is a teaching harness, not a production collector, a security review, or a performance study of filesystems. Overlayfs, NFS, and FUSE can change what sameness even means for an inode, and I did not measure those cases. A free shared scratch box is also the wrong place for logs that contain secrets, tokens, or customer identifiers. Skip this approach if you live on Windows without a Linux VM, or if you need an SLA-backed runner for compliance.

Skip it if you cannot read every line of a generated watcher before it touches a real log directory. copytruncate avoids the inode split by copying bytes and truncating in place, but it can drop writes that arrive during the copy window. Signaling your own writer to reopen the path is cleaner when you own that process and can afford the handler. When you do not own the writer, a directory watch plus an explicit reopen is the least bad pattern I still trust.

What I would tell myself at hour zero

Start with ls -li and an lsof line for deleted files before you ask whether the application is healthy. Ask whether the fd and the path still share an inode, and only then start pulling thread dumps. Keep the decision table beside the harness, and make the inode split fail the test on purpose every time. That single st_ino print would have given me my weekend back, and I am not dropping it again.