On the first Monday of the peak season the handheld scanners in our Leeds warehouse began taking several seconds to confirm each pick. The warehouse system runs in our data centre, and the warehouse reaches it over a one gigabit circuit. The first thing the on call engineer opened was the bandwidth graph for that circuit, which showed about three hundred megabits a second. The link was ruled out in the first five minutes, and we spent the next two hours on the database.
The firewall at the warehouse end showed the link at ninety eight percent when someone finally logged in to it.
We scrape the firewall with the Prometheus SNMP exporter every sixty seconds. Its module was written by hand in 2019, when the circuit was a hundred megabits, and it walks ifInOctets and ifOutOctets from the original interface table. Those are 32 bit counters. They count bytes and wrap back to zero after 4,294,967,296. The 64 bit versions, ifHCInOctets and ifHCOutOctets, are in a different table, and our module never asked for them.
At a hundred megabits a 32 bit byte counter wraps about every six minutes, so a scrape every minute almost always saw it climb. At one gigabit it wraps every thirty four seconds. Prometheus treats a counter that goes down as a counter that restarted from zero, and counts only the new value, so every interval in which the counter wrapped lost most of what had passed through it, and an interval with two wraps lost more. With one sample a minute, the graph could never have shown more than 2^32 bytes a minute, about 573 megabits a second, whatever the circuit was doing. Ordinary days used between two and four hundred, so for two years the graph looked plausible.
The module now walks ifXTable and uses the HC counters, as the exporter's standard if_mib module already did. A recording rule compares each interface's rate with its ifHighSpeed and alerts above eighty percent, and a test in the exporter config fails if any module asks for a 32 bit octet counter on an interface faster than a hundred megabits.
The circuit was upgraded and its monitoring was not. A 32 bit counter scraped once a minute has a ceiling, and ours sat about halfway up a one gigabit link.
– Sergey Shinder
Top comments (0)