Our alerts go through one Alertmanager. For years every critical alert went to the operations channel in Slack and to nobody's phone, because operations watched the channel in working hours and the on call engineer watched it at night. In September the payments team took over their own on call, and they asked for their critical alerts to page them.
The change was one new route: match team payments and severity critical, send to the payments pager receiver. It was reviewed, Alertmanager reloaded the configuration without complaint, and amtool check-config passed. We added the route at the bottom of the list, below the existing ones.
Five weeks later the card provider's callback endpoint started rejecting signatures at eleven at night. The alert fired within two minutes. It went to the operations channel, which nobody was reading, and the payments on call slept until a customer service lead rang him at half past seven.
Alertmanager walks its routing tree from the top. At each level it tests the child routes in the order they are written, and the first one that matches takes the alert. It does not look at the routes after it unless that route sets continue to true. Our second route matched every alert with severity critical, which was how operations got everything. The payments alert matched it first and stopped there. The new route was valid, loaded and correct, and no alert could ever reach it.
amtool can answer exactly this question. amtool config routes test takes a set of labels and prints the receiver an alert with those labels would go to. Had we run it with team payments and severity critical before merging, it would have printed the operations channel.
We moved the team routes above the generic ones and gave the generic critical route a matcher that excludes teams with their own rotations. Every pull request that touches the routing tree now runs a table of label sets through amtool config routes test and compares each result with the receiver we expect, so a route that cannot be reached fails the build. Each team also sends a synthetic critical alert through the real path once a week, and its own pager has to acknowledge it.
A configuration that loads tells you it was written correctly. It does not tell you where anything will go.
– Sergey Shinder
Top comments (0)