On a Tuesday night in July our public website returned errors for thirty five minutes. The external probe that checks it every minute fired the whole time, and nobody was paged. The alert was there in Alertmanager, active, and marked as inhibited.
Two years earlier we had added an inhibition rule: while a ClusterMaintenance alert is firing, mute warning and critical alerts that have the same value for the label cluster. The maintenance alert comes from an exporter that reads our change calendar. In June we replaced that exporter with one that reads the new calendar, and it labelled each maintenance window with target instead of cluster.
The rule compares the labels listed under equal, and Alertmanager treats a missing label as equal to another missing label. Its documentation says so: missing and empty are the same thing. A maintenance alert with no cluster label now matched every alert that also had no cluster label. Those were precisely the alerts that do not belong to any cluster: the external probes of the website, certificate expiry, the checks on our payment provider. Staging has a two hour maintenance window every Tuesday night. For five Tuesdays every global alert we had was muted for two hours, and on the fifth something actually broke inside the window.
Nobody saw it because an inhibited alert looks just like one that never fired. It does not reach the channel, the pager or the morning summary, and it shows up only to somebody who opens Alertmanager and asks for inhibited alerts.
The new exporter labels windows with cluster again, and the change calendar rejects a window without one. Every inhibition rule now requires each label in its equal list to be present and non empty on both sides, written as a matcher such as cluster=~".+", and a lint step in the repository fails any rule that leaves it out. Alerts that belong to no cluster carry scope="global", and no inhibition rule may target them. And a small job asks the Alertmanager API for inhibited alerts every five minutes and posts any page level alert that has been inhibited for more than half an hour, with the alert that is doing the inhibiting.
Two missing labels are equal in the same way that two empty envelopes hold the same letter. Our rule had been comparing envelopes for five weeks.
– Sergey Shinder
Top comments (0)