On a Saturday in August checkout returned errors to about one customer in eight for thirty five minutes. Our alert on checkout errors had a threshold of five percent and a for clause of five minutes. The error rate was above the threshold for the whole thirty five minutes, and the alert never fired.
Two weeks earlier somebody had improved the notification. The engineers on call wanted the current error rate in the message itself, so a label was added to the rule, error_rate, set to the templated value of the expression and formatted as a percentage.
In Prometheus an alert is identified by its labels. Our rules are evaluated every thirty seconds, and each evaluation that produces a label set Prometheus has not seen before starts a new alert in the pending state, with its own clock. The for clause means one alert must stay pending, with the same labels, for five minutes before it becomes firing. Our label changed whenever the error rate changed, and during an incident that is every evaluation. Each new value started a new pending alert, the previous one disappeared because its label set was no longer being returned, and none of them lived longer than thirty seconds. The Prometheus documentation warns against exactly this. Annotations exist for values that change, and the fix was moving one line.
The evidence had been in Prometheus all along. The ALERTS series for that rule showed about seventy distinct pending alerts across those thirty five minutes, each one a single sample long, and no firing alert at all.
The rate is an annotation now, where it is rendered into each notification without becoming part of the alert's identity. Our rules repository runs pint in CI, which rejects any label templated from the value. Every paging rule has a promtool unit test with an input series that rises and wobbles the way real errors do, asserting that the alert is firing after its for duration, so a rule that cannot fire fails before it is merged.
A label tells Prometheus which alert this is. We used one to say how bad it was, and every new reading became a different alert.
– Sergey Shinder
Top comments (0)