Our EDI archive server has four cores and mounts a NAS share over NFS at /mnt/archive, with the default hard mount. A cron job runs every minute, moves each new partner file into the archive and then sends its acknowledgement. The host's CPU alert was a Prometheus rule on node exporter's node_load5: fire when it goes above twice the number of cores.
At ten past two on a Thursday in October it paged. Load was sixty one. The engineer on call opened top, saw the CPU ninety seven percent idle and nothing using it, decided the backup had confused the number, and acknowledged it. At seven three partners rang to ask why no acknowledgements had come since two in the morning.
The Linux load average is not a measure of CPU demand. It counts tasks that are running or waiting to run, and also tasks in uninterruptible sleep, the D state, which almost always means waiting on I/O. The NAS had started a firmware update at two and stopped answering. With a hard mount the NFS client retries forever, so every process that touched /mnt/archive simply waited. Each minute cron started one more job, and each new job joined the queue in D state. Sixty of them, using no CPU at all, added up to a load of sixty. ps -eo state,cmd showed them all, with D in the first column.
The alert had noticed something real and described it as something else. Nobody looks for a hung file system when the page says CPU.
CPU now has its own alert, on utilisation from node_cpu_seconds_total, which stayed near zero all night. We enabled node exporter's processes collector, which counts processes by state, and a separate alert fires when node_processes_state with state D stays above five for ten minutes. Its description tells the reader to check mounts and disks before anything else. A systemd timer stats /mnt/archive under a ten second timeout every minute and writes the result for the textfile collector, so a dead share shows up as a dead share. The cron job now runs under flock, so a hung run cannot be joined by another.
Load had been high for five hours and it was right the whole time. We had labelled it as the one thing it was not measuring.
– Sergey Shinder
Top comments (0)