One Tuesday in August new pods on three of our CI nodes began failing to start with no space left on device. The disk panel for those nodes said fifty eight percent used. I spent twenty minutes assuming the panel was wrong.
It was right about bytes. The nodes had run out of inodes. Every file and directory on an ext4 filesystem needs one, the total is fixed when the filesystem is created, and the default assumes files averaging sixteen kilobytes. df -i showed the root volume at a hundred percent. Starting a container means unpacking image layers and writing a few files of its own, and there was nowhere left to put them.
Our CI jobs run in containers built from images the teams maintain. The frontend build image contained a full node_modules directory, about four hundred thousand files, most of them a few hundred bytes. It was rebuilt several times a day, and every version a node pulled was unpacked in full. The kubelet's image garbage collection triggers on disk usage, at eighty five percent on our nodes, and a disk full of tiny files never gets near eighty five percent of its bytes. So nothing was collected, and one node held forty three versions of that image.
The kubelet should still have noticed free inodes running low. Years earlier we had customised two of its eviction thresholds, and setting any of them replaces the whole default set, so free inodes were not watched at all. Failed pods landed on the next node, pulled the image there and used up more inodes. By mid afternoon nine nodes of twenty were refusing work.
The node image now formats the container volume as XFS, which allocates inodes as it goes instead of fixing them up front. Inode usage sits next to disk usage on every node panel, with the same alerts. The kubelet configuration lists every eviction threshold explicitly, including free inodes. The frontend image no longer ships node_modules; dependencies come from a cache volume at job time, which took it from four hundred thousand files to under nine thousand. And image garbage collection also removes anything unused for three days, whatever the disk says.
A disk can fill up in two ways and we graphed one of them. Small files are cheap in bytes, which is exactly why a byte count will never warn you about them.
– Sergey Shinder
Top comments (0)