## Linux From Zero: Inodes and the Filesystem Limit df -h Doesn't Show (for Platform Engineers)

Every platform engineer learns that df -h shows disk space. Fewer learn there's a second limit, completely independent of bytes: inodes. And when that second limit is the one you're actually hitting, df -h will lie to you with a straight face — plenty of gigabytes free, and still No space left on device.

This is the fifth piece in Linux From Zero, a track on Linux fundamentals for people building and operating Kubernetes platforms. Full track and essays on GitHub.

The problem df -h doesn't show

A filesystem reserves a fixed number of inodes when it's created. Every file, however small, consumes exactly one. A directory with millions of tiny files — web session files, dead mail queues, log rotation without a retention limit — can exhaust inodes while the disk still shows gigabytes of free space.

The classic symptom: No space left on device when creating a file, while df -h insists there's plenty of room.

What an inode actually holds

An inode doesn't store the filename. It stores metadata — owner, permissions, timestamps, size, and pointers to the data blocks on disk. The filename lives only in the directory entry, which points to an inode number. That's why renaming a file is instant (only the directory entry changes), and why a hard link is, in practice, two names pointing at the same inode.

ls -i /etc/hostname

Try it: run the command above and note the number. Create a hard link (ln /etc/hostname /tmp/hostname-link) and run ls -i on both — the number should be identical.

Seeing the inode limit

df -i /

If IUse% is near 100% while df -h shows free space, the problem is inodes, not bytes.

Try it: compare df -h / and df -i / on your machine. Then create 200,000 empty files in a scratch directory and watch df -i move while df -h barely budges. Clean up afterward.

Finding what's eating the inodes

for dir in /var/log /var/cache /tmp; do
  echo "$dir: $(find "$dir" -xdev | wc -l) files"
done

In production, the usual suspects are web application session directories, dead mail queues, and log rotation with no retention cap.

lsof +L1 — the other way to lose space you can't see

There's a second classic way for a disk to be "full with free space": a process holds open a file that's already been deleted. unlink removes the directory entry, but the inode is only freed once the last process closes its file descriptor. Meanwhile, the space doesn't show up in du, but it doesn't come back to df either.

lsof +L1

A line with NLINK 0 is the tell: zero links in the directory, but the file is still holding space because some process keeps it open. The fix isn't deleting it again — there's no entry left to delete — it's restarting or signaling the process so it closes and reopens the file (logrotate with copytruncate, or a kill -HUP).

Try it: in one terminal, hold a file open with tail -f app.log; in another, delete it (rm app.log); run lsof +L1 and watch the NLINK 0 line while the first terminal still holds it open.

The Kubernetes connection: DiskPressure

The kubelet watches node disk on two axes, not one: free bytes and free inodes. When either crosses its configured threshold (nodefs.inodesFree, roughly 5% by default), the kubelet marks the node with the DiskPressure condition and starts evicting pods to free space — lowest-priority pods first, then the ones consuming the most in emptyDir and container logs.

kubectl describe node <node-name> | grep -A2 DiskPressure

A node can show DiskPressure=True with df -h looking perfectly normal, if the cause is inode exhaustion from an app generating lots of small files under /var/lib/containerd or in the logging volume. Investigating without knowing this leads to the wrong conclusion ("there's space, it's not disk"), and the root cause never gets found.

Investigation checklist

1. Run df -h and df -i together, always. Never one without the other.
2. If a volume has many small, ephemeral files, ask whether it needs more reserved inodes, or whether the application should clean up after itself.
3. In production, monitor node_filesystem_files_free (Prometheus/node_exporter) with the same alerting rigor you already apply to free bytes.

---

This piece is part of Linux From Zero, a Portuguese-language track on Linux fundamentals for Platform Engineering. The Portuguese original is chapter 05 on GitHub.