A watchdog fails quietly in two ways: it says nothing when a source dies, or it mails so often that nobody reads it. Where the clock, limit and mail go wrong.
A source that has stopped answering produces no error, no stack trace and no red line anywhere. A feed that went quiet, a publisher that stopped publishing and a job that stopped running all look exactly like a calm day. The watchdog is therefore the only symptom such a failure will ever have, and it can go wrong in two opposite directions. Either a genuine death goes unreported, or healthy sources trigger so many messages that the recipient stops opening them. Both come from a small set of design mistakes, mostly in how silence is measured, how long a limit is, and how the message is written. The sections below take them one at a time and finish with a short test that can fail.
Read the full report on AISkills402: https://aiskills402.com/blog/watchdog-alerts-silent-and-noisy?utm_source=devto&utm_medium=social
Top comments (0)