We run Opservo on our own servers first: 40+ production machines across the 30 companies in the Softuvo portfolio. On 28 September we found out that, for 59 days, not one alert meant for our own team had reached us.
What happened
Our team channel was a Discord server. Opservo sends to custom webhooks and Discord accepts webhooks, so on 31 July we pointed our alerts at one and moved on.
From that day on, the webhook rejected every alert with HTTP 400 Bad Request. Opservo spotted the problems on time and sent the alerts on time. Discord refused every one of them, and nobody was looking at the one place that would have told us.
The worst of it: an eight-day Redis outage sat inside that window. Opservo raised it. The alert hit a 400 and went no further.
Why nobody noticed
Silence looks exactly like health. A channel that gets nothing for a week reads as "a quiet week". Nothing ever tells you that you did not get a message.
Most teams test an alert channel once, on the day they add it. After that, the only proof it works is an alert arriving, and no alerts arriving proves nothing.
What we changed in Opservo
The webhook is fixed. More importantly, every notification channel in Opservo now records the result of its last delivery, so "is this channel actually working?" has an answer on the screen instead of a guess.
It was the second time in a month that a channel let us down. On 19 September a workstation disk remounted read-only. Opservo alerted within a minute, but nobody heard, because that workspace had no notification channel at all. Since then Opservo shows a banner when a workspace has no channel to alert through, and it raises a composite disk-failure alert.
Five checks for your own alert channels, whatever tool you use
- Send a test alert to every channel and confirm a person saw it. "Sent" is not enough. Someone has to have seen it.
- Look at the delivery result, not the configuration. A webhook URL that saves without error can still reject every payload.
- Test with a real alert, not a hello-world message. A receiver can accept one and reject the other.
- Give anything that pages at least two paths, for example Slack and email, so one broken channel does not mean total silence.
- Repeat check 1 every month. It takes five minutes. Put it in the calendar now.
The lesson
We build a monitoring tool and still missed this for two months. If it happened to us, it is worth five minutes to check yours.
Opservo is the AI ops engineer for teams without an SRE: plain-English server monitoring with one-click fixes. Free for 2 servers. Start at getopservo.com/welcome, or see what it monitors.
Originally published on the Opservo blog.
Top comments (1)
silence looking like health applies well beyond alerts. one cheap check is a scheduled test message with a known arrival time, plus an alert when it fails to arrive, so the absence of the message is itself the signal. otherwise a quiet week and a dead channel look the same.