DEV Community

Vigilmon
Vigilmon

Posted on

How to Set Up On-Call Alerting with Vigilmon (Rotations, Escalations, and Alert Rules)

#sre #devops #monitoring #oncall

On-call rotations are stressful when your alerting setup is noisy or poorly configured. This guide shows how to set up intelligent on-call alerting with Vigilmon - so the right person gets paged for real incidents, not false positives.

The On-Call Alerting Problem

Most teams set up uptime monitoring with a single email address. Then:

  • Alert fatigue sets in: Too many false positives, team starts ignoring alerts
  • Wrong person gets paged: Everyone gets the alert, no one takes ownership
  • No escalation path: If the first responder doesn't acknowledge, the incident continues unresolved
  • Night alerts for non-critical issues: Staging is down at 2am, everyone gets paged

Good on-call alerting is about routing the right alert to the right person at the right time.

Vigilmon's Alert Architecture

Vigilmon alerts flow through:

  1. Monitors ? Check your endpoints
  2. Alert rules ? Threshold before alerting (e.g., 2 consecutive failures)
  3. Notification channels ? Where alerts go (email, Slack, webhook, SMS)
  4. Recipient configuration ? Who gets which alerts

Step 1: Define Your Alert Tiers

Before configuring Vigilmon, define your alerting policy:

Critical (page immediately)

  • Production services down
  • Checkout/payment endpoints failing
  • Authentication broken
  • Database unreachable

Warning (notify, don't page)

  • Staging environment issues
  • Response time degradation (not complete failure)
  • Non-critical background jobs missed heartbeat

Info (daily digest or ignore)

  • Development environment issues
  • Non-user-facing internal tools

Step 2: Configure Per-Monitor Alert Rules

For each monitor in Vigilmon, configure:

Consecutive failures before alert: 2 (production) / 3 (staging)
Enter fullscreen mode Exit fullscreen mode

This prevents single-transient-failures from triggering pages. For production critical services, 2 failures is the right balance - fast detection without excessive false positives.

For staging, require 3 consecutive failures. Staging environments are noisier.

Step 3: Set Up Notification Channels

In Vigilmon, create separate notification channels for each alert tier:

Production channel (Slack #prod-incidents)

Type: Slack webhook
Channel: #prod-incidents
Webhook URL: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
Mention: @oncall (current on-call person)
Enter fullscreen mode Exit fullscreen mode

Production SMS channel

Type: SMS
Recipients: +1-555-0100 (on-call engineer)
Enter fullscreen mode Exit fullscreen mode

Staging channel (Slack #dev-alerts)

Type: Slack webhook  
Channel: #dev-alerts
No @mention (not urgent)
Enter fullscreen mode Exit fullscreen mode

Step 4: Implement Escalation via Webhooks

Vigilmon supports webhook alerts, which lets you integrate with on-call management tools like PagerDuty, OpsGenie, or your own escalation system.

PagerDuty integration:

POST https://events.pagerduty.com/v2/enqueue
{
  "routing_key": "YOUR_PD_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "Vigilmon alert: {{monitor_name}} is down",
    "severity": "critical",
    "source": "vigilmon"
  }
}
Enter fullscreen mode Exit fullscreen mode

Configure Vigilmon to POST to these webhooks when a monitor fails.

Step 5: Configure Recovery Alerts

Just as important as failure alerts: knowing when your service recovers.

In Vigilmon, configure recovery notifications to:

  • Automatically resolve PagerDuty incidents when the monitor recovers
  • Post a recovery message to Slack (#prod-incidents)

This closes the loop without manual acknowledgment.

Sample On-Call Alert Flow

Here's a real example of a well-configured on-call flow:

T+0:00  Monitor detects failure (check location 1)
T+0:01  Monitor detects failure (check location 2) 
T+0:01  Alert threshold: 2 consecutive failures confirmed
T+0:01  Slack alert ? #prod-incidents (@oncall)
T+0:01  SMS ? on-call engineer's phone
T+0:01  Webhook ? PagerDuty creates incident
T+5:00  PagerDuty escalates to backup if unacknowledged
T+12:45 Fix deployed, service recovers
T+12:46 Slack: "api.yourapp.com recovered after 4m 23s"
T+12:46 PagerDuty incident auto-resolved
Enter fullscreen mode Exit fullscreen mode

Avoiding Alert Fatigue

The single biggest cause of on-call fatigue is false positive alerts. With Vigilmon:

  1. Multi-region confirmation: Alerts only fire when multiple locations confirm failure
  2. Consecutive failure threshold: Require 2-3 failures before alerting
  3. Separate channels by severity: Don't mix staging noise with production alerts
  4. Response time vs availability: Alert on availability separately from response time

Common On-Call Alerting Mistakes

Mistake 1: Same alert channel for all monitors - staging noise drowns out production alerts.

Mistake 2: Single-location monitoring - false positives every week, alert fatigue sets in.

Mistake 3: No recovery alerts - engineers don't know when to stop investigating.

Mistake 4: Threshold = 1 failure - every HTTP timeout pages the on-call.

Start Building Your On-Call Setup

Vigilmon provides the monitoring and alerting infrastructure. Pair it with a rotation tool (PagerDuty, OpsGenie) for full on-call coverage.

Start at vigilmon.online - free tier includes webhook support and Slack integration.

Top comments (0)