Our partner file gateways are four Linux hosts that receive and forward EDI files. For years their memory alert was a Prometheus rule on node exporter's numbers: fire when node_memory_MemFree_bytes falls below ten percent of node_memory_MemTotal_bytes.
It fired most nights. At one in the morning a backup reads the archive of the last ninety days, sixty gigabytes, and afterwards free memory sat at two or three percent until the next day. We looked once, saw no process using much, and lowered the threshold to two percent. After that it fired only occasionally and nobody looked at all.
In October the gateway service on one host was killed by the kernel's OOM killer at twenty to four on a Tuesday afternoon, and partner files stopped being forwarded until it was restarted. The alert had not fired. MemFree had been between two and four percent for the whole of the previous month, which is exactly how it looked on the healthy hosts.
MemFree is memory the kernel is using for nothing at all. Linux does not leave memory idle when it can keep file contents in it, so after the backup almost everything not used by processes held page cache, and page cache is given back the moment a program needs memory. That is why our memory looked full every night. It is also why the leak was invisible: as the gateway grew, the kernel dropped cache to make room, and free memory stayed the same until there was no cache left to drop.
The number we wanted was MemAvailable, which the kernel has published since version 3.14 and node exporter exposes as node_memory_MemAvailable_bytes. It estimates how much memory could be given to programs without swapping, counting reclaimable cache. Graphed for the same month, it went down in a straight line for eleven days, from seventy percent to zero.
The alert now fires when MemAvailable stays below ten percent of total for fifteen minutes. A second rule uses predict_linear over six hours and opens a ticket if it says MemAvailable will reach zero within a day. The backup no longer touches either. Each service's resident memory is on the gateway dashboard too, so a growing process has its name next to the line.
We had an alert on memory nobody was using. The memory we were about to run out of was never on the graph.
– Sergey Shinder
Top comments (0)