On a Sunday night in October the database behind our returns service shut itself down with a PANIC in its log: it could not write to a file in pg_wal because there was no space left on the device. Nobody was paged. Our disk alert pages at ninety seven percent full, and the dashboard for that volume, when we opened it on Monday morning, said ninety five.
The volume is two terabytes of ext4, and for weeks it had been filling slowly with a table of return photographs nobody had archived. The warning at ninety percent had raised a ticket on the Tuesday, and the ticket was planned for the following week, because at that rate the page would arrive with days to spare.
Both numbers on our dashboards came from node_exporter, and both were correct. It exports two figures for free space. node_filesystem_free_bytes counts every free block. node_filesystem_avail_bytes counts the free blocks an ordinary user is allowed to write to. The difference is the reserve ext4 keeps for root, which by default is five percent of the filesystem, set when it is created. On our volume that is about a hundred gigabytes that Postgres, running as the postgres user, can never touch.
Our alert and our panel were both built on free_bytes. When Postgres wrote the last block it was allowed, the expression still saw a hundred gigabytes free and reported ninety five percent. It could never have reached ninety seven. df, which bases its percentage on the available figure, would have said one hundred.
We had Postgres running again within twenty minutes of looking, by lowering the reserve to one percent with tune2fs -m 1, which takes effect at once on a mounted filesystem. The photographs were archived properly that afternoon.
Every disk alert and panel now uses avail_bytes, the figure our applications actually live within. Volumes written only by a service account are created with a one percent reserve, because the reserve exists to keep a system disk usable for root and does nothing for a database volume. And before a new disk alert is trusted, we fill a scratch volume as an ordinary user until writes fail, and check what the alert said at that moment.
On Linux, how much space is left has two answers. Our alert had been asking on behalf of root.
– Sergey Shinder
Top comments (1)
The free_bytes versus avail_bytes split is an easy one to miss, because both dashboards were correct and still wrong for the user that matters. Your scratch-volume test is a great habit, since it validates the alert by actually failing writes. One thing we'd consider adding is a time-to-full alert using predict_linear over avail_bytes, since a fixed percentage can't tell a slow fill from a fast one. Was the WAL on the same volume as the data?