Writing Good Alerts: How to Create Monitoring That Actually Helps Humans

Alerts. Those pesky notifications that interrupt your dinner, jolt you awake at 3am, or flood your inbox with false positives. I feel your pain. But robust monitoring is crucial for catching issues early. The key is designing alerts that help humans manage systems, not overwhelm them.

Here are some tips I've learned for creating effective, human-centric alerts:

1. Ensure alerts lead to concrete actions

A good alert should enable responders to quickly mitigate issues or escalate them. Ask yourself:

• Are the metrics clear indicators of a specific problem? Vague metrics lead to confusion.

• Does the alert contain enough context like IDs, snippets, or charts to diagnose root cause? Details speed investigation.

• Are runbooks provided for triage, debugging steps, and contacts? Documentation aids responders.

2. Set dynamic, tailored thresholds

Static thresholds fail as systems evolve. Instead, consider:

• Adjusting thresholds based on trends, seasonal usage patterns, and new behaviors. Weekends may differ from Tuesdays.

• Updating error budgets when introducing new failure scenarios. Additional fallback logic impacts counts.

• Tightening thresholds initially for new alerts, then relaxing once confident they work.

3. Use rigorous prioritization

Not every anomaly requires waking someone up. Thoughtfully prioritize based on:

• Severity of the incident. Reserve high priority for true emergencies needing immediate response.

• Likelihood it's a transient, self-healing glitch. Don't stress responders unnecessarily.

• Relevance of the error domain. Route alerts to appropriate teams to avoid distraction.

4. Continuously improve alerts

Analyze false positives, false negatives, feedback and tagged issues to iteratively refine alerts:

• Identify faulty metrics or thresholds by reviewing incorrectly triggered alerts.

• Surface missing alerts by correlating outages with alert gaps.

• Incorporate pain points and ideas from oncall engineers through surveys.

• Fix problematic alerts tagged with "noisy alert" or similar keywords.

Thoughtfully designed, continuously refined alerts provide responders valuable insights, not endless distractions. Now instead of battling alert noise, they can focus on quickly solving real problems. That's monitoring done right.

What strategies have worked for you when creating human-centric alerts? Share your wisdom below!