A broker node ran out of disk on a Thursday morning and stopped accepting writes. We found out from a product team asking why their events had gone quiet. The disk alert existed, was correctly written, had been evaluating for hours and was firing perfectly well. It was matched by a silence created five months earlier for a two hour maintenance window.
The person who made it had set an end time, typed a comment, done everything right except that the matcher was on cluster rather than on the specific instance, and the duration field had been left on a value that turned a two hour window into something much longer. Nobody reviewed it, because nobody reviews silences. They are created during an incident or a change window, by whoever is holding the keyboard, and then they are invisible. There is no pull request, no approval, no expiry notice, no owner.
Once we went and looked, there were thirty-one active silences in that Alertmanager. Nine were older than a month. Two matched on a label so broad they covered every rule in a whole namespace. Four referenced people who had left. Collectively they were suppressing a meaningful slice of our alerting and none of it appeared on any dashboard anybody opened.
What we run now is not clever. A bot posts the full list of active silences into the on-call channel every weekday morning, with age, matcher and creator. Anything older than seven days is called out individually. Creating a silence longer than twenty-four hours requires a ticket reference in the comment field, checked by a small service that reads the API and deletes silences that do not have one, after warning in the channel. And there is a meta-alert that fires when the number of alert rules currently suppressed goes above a threshold, because one silence that swallows forty rules is a different animal from forty narrow ones.
We also learned to prefer inhibition rules for the cases that come back regularly, since those live in configuration, in a repository, reviewed like anything else.
An alert you have muted is not a quiet system. It is an alert you have agreed in advance not to be told about, by someone who has since gone home.
– Sergey Shinder
Top comments (0)