How to Set Up On-Call Alerting with Vigilmon
On-call alerting is the difference between catching a production outage in 90 seconds versus finding it at 9 AM from a support ticket. But getting on-call right is tricky: too many alerts and your team ignores them (alert fatigue); too few and real incidents slip through.
This guide shows you how to set up effective on-call alerting using Vigilmon — with strategies to minimize false positives while ensuring real outages always get through.
The Problem with Naive Alerting
Most monitoring tools fire alerts on the first failure. This causes:
- 3 AM pages for 30-second network hiccups — the site was never actually down for real users
- Alert fatigue — team members start ignoring alerts or muting notifications
- Missed real incidents — once alert fatigue sets in, real outages get buried in noise
Vigilmon solves this with multi-region consensus alerting: an alert only fires when 2+ geographic regions independently confirm the failure. This eliminates the vast majority of false positives from transient probe-side issues.
Step 1: Configure Multi-Region Monitors
First, set up monitors with consensus alerting:
- Log in to vigilmon.online → Add Monitor
- Type: HTTP(S)
- URL:
https://your-app.com - Interval: 60 seconds
- Alert condition: Status != 200 confirmed from multiple regions
This is the core: instead of alerting on the first failure from one location, Vigilmon confirms from multiple locations before firing. If your site is down in Tokyo but up everywhere else, that's a Tokyo network issue — not your problem.
Step 2: Set Up Alert Escalation via Webhooks
Vigilmon supports webhook alerts for integration with your preferred notification channels.
Slack Integration
In Vigilmon's notification settings, add your Slack webhook URL:
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
Vigilmon will POST a JSON payload to Slack when an alert fires:
{
"text": "🔴 ALERT: your-app.com is DOWN",
"attachments": [
{
"color": "danger",
"fields": [
{ "title": "Monitor", "value": "your-app.com" },
{ "title": "Status", "value": "DOWN" },
{ "title": "Regions confirming", "value": "US-East, EU-West, AP-Southeast" }
]
}
]
}
PagerDuty Integration
Vigilmon integrates with PagerDuty via webhook:
- In PagerDuty, create a new service → Integration Type: Events API v2
- Copy the Integration Key
- In Vigilmon, add a webhook notification pointing to:
https://events.pagerduty.com/v2/enqueue
- Configure the payload:
{
"routing_key": "YOUR_PAGERDUTY_INTEGRATION_KEY",
"event_action": "trigger",
"payload": {
"summary": "{{monitor_name}} is DOWN",
"severity": "critical",
"source": "vigilmon.online"
}
}
Step 3: Layer Your Alert Severity
Not all monitors need the same urgency. Create tiered alerting:
Tier 1 — Critical (immediate page):
- Your main app URL
- Auth/login endpoint
- Payment/checkout endpoint
- Database health check
Tier 2 — Warning (Slack notification, no page):
- Admin dashboard
- API documentation endpoint
- Non-critical internal tools
Tier 3 — Info (daily digest):
- Staging environment
- Non-customer-facing services
Configure different Vigilmon notification channels for each tier.
Step 4: Monitor Your Alerting System Itself
A common failure mode: your alerting infrastructure goes down and you don't know because there are no alerts.
Set up a Vigilmon heartbeat for your notification pipeline:
// In your monitoring service (runs every 5 minutes)
const healthPing = async () => {
try {
// Try sending a test Slack notification
await axios.post(SLACK_WEBHOOK, { text: 'Health check (delete me)' });
// Ping Vigilmon to confirm alerting pipeline is healthy
await axios.get('https://hb.vigilmon.online/ALERTING-PIPELINE-HB-ID');
} catch (err) {
// If we can't reach Slack, log it but don't crash
console.error('Alerting pipeline unhealthy:', err.message);
}
};
If Vigilmon doesn't receive the heartbeat, it can send an alert via email (fallback channel) telling you your primary alerting is broken.
Step 5: Define Response Playbooks
Alerts without playbooks create panic. For each critical monitor, document:
## Playbook: Main App DOWN
### First 5 minutes
1. Check https://vigilmon.online dashboard — which regions are down?
2. Check https://status.yourdomain.com — is this already posted?
3. Run: `curl -v https://your-app.com` to get raw HTTP response
4. Check VPS/container logs: `kubectl logs -l app=web --tail=50`
### Next 10 minutes
- If DB issue → check `/health/db` endpoint
- If deploy-related → run `git log --oneline main -5`
- If infra issue → check cloud provider status page
### Communication
- If > 2 minutes → post to #incidents Slack channel
- If > 10 minutes → update public status page at status.yourdomain.com
- If > 30 minutes → escalate to CTO
Link your Vigilmon alert notifications to these playbooks.
On-Call Alerting Coverage Table
| Monitor | Alert Channel | Response Tier |
|---|---|---|
| Main app | Slack + PagerDuty | Immediate page |
| Auth endpoint | Slack + PagerDuty | Immediate page |
| Payment endpoint | Slack + PagerDuty | Immediate page |
| Database health | Slack | 5-minute window |
| Admin dashboard | Slack | Business hours |
| SSL certificates | 14-day warning |
On-Call Best Practices
Rotate on-call weekly. No one should be on-call permanently. Weekly rotations spread the burden and prevent burnout.
Set alert quiet hours. Only wake people for Tier 1 critical alerts at 3 AM. Tier 2 can wait until morning.
Review and tune regularly. After every false positive, adjust the monitor threshold. After every missed incident, add a new monitor. On-call is a living system.
Post-incident reviews. After every significant incident, write a short post-mortem. What failed? What would have caught it sooner? Update monitoring accordingly.
Conclusion
Effective on-call alerting isn't about more alerts — it's about better signals. Vigilmon's multi-region consensus alerting eliminates false positives so that when your phone rings at 3 AM, it's always a real incident worth waking up for.
Top comments (0)