Our checkout service has a latency objective: ninety five percent of requests answered within one second. The alert for it divides the rate of the histogram bucket with le="1" by the rate of all requests, and pages when the ratio stays below 0.95. In October, for two hours on a Saturday afternoon, a slow fraud check pushed a fifth of checkouts past one second. The alert never fired. We heard about it from a customer on Monday.
Three weeks earlier we had upgraded Prometheus from 2.53 to 3.0. The upgrade went quietly, every target was up, and every dashboard drew.
Prometheus 3 normalises the le label of classic histograms when it ingests them, so that the same bucket gets the same value whatever format it was scraped in. A bucket the service exposes as le="1" is stored as le="1.0". The migration guide says so, and says that rules and dashboards which name whole number buckets will stop matching. We had read the guide for the flags that were removed and skimmed the rest.
Checkout is written in Go, and the Go client writes whole number buckets without a decimal point. Our Python services were never affected, because their client already writes le="1.0". In the alert, the selector for le="1" now matched nothing, a division with nothing on one side returns nothing, and an alert expression that returns nothing is an alert that is not firing. The rule evaluated every thirty seconds for three weeks without an error.
The rules and panels now name le="1.0". Panels that look back past the upgrade use a regular expression that matches both values, because older data in the TSDB still carries the old label. Every alert on a ratio now has a companion that fires when its numerator or denominator returns no series for a job that is up, so a selector that stops matching becomes a page of its own.
Before the next upgrade we will run every rule expression against a copy of the new version fed with the same scrapes, and compare which ones return data on both sides.
An upgrade can keep every metric and still rename the one value an alert depends on. Ours was a decimal point.
– Sergey Shinder
Top comments (0)