How do you set up monitoring alerts that wake the responsible on-call engineer, but do not train the team to ignore the pager? Your API can fail from two regions while Slack gets five duplicate warnings. A certificate can expire in 30 days and still page someone at 3 AM. A checkout outage can go to a general engineering channel because nobody set the owner.
Alerting fails when engineers have not defined which cases should wake the on-call engineer. A short warning gets treated like a production outage. Three signs of the same dependency failure become three incidents. Database alerts reach people who cannot fix the database. The fix starts with a clear decision: keep alerts that need action now, downgrade alerts that can wait, and remove alerts nobody uses.
Monitoring alerts connect a failed check to a human response. That connection is expensive. It can stop work, wake someone from sleep, or break a weekend. If the alert is real, that cost is worth it. If the alert is unclear, repeated, or impossible to act on, it creates alert fatigue.
This guide explains a simple rule: every page should be urgent, useful, and require a human. We will cover what to alert on, how severity should control routing, how a notification policy reduces noise, how alert quality affects the on-call engineer, and how to set up DevHelm so the right channel gets the right alert.
Why monitoring alerts matter
Monitoring answers "is the system healthy?" Alerting answers "who needs to act now?"
Those are different questions. A dashboard can show many metrics because people open it when they need details. An alert is pushed to a person. It needs a higher bar.
A healthy monitoring system usually has three output paths:
| Signal type | Destination | Example |
|---|---|---|
| Urgent and actionable | Page on-call | Checkout API returns 500s from multiple regions |
| Important but not urgent | Ticket or Slack | TLS certificate expires in 21 days |
| Useful context only | Dashboard or log | CPU crossed 70% for two minutes |
The mistake is sending all three to the same place. A Slack channel with deploy notes, warning alerts, dependency pings, and real outages becomes a junk drawer. A phone alert for non-urgent warnings becomes noise. The alerting layer should sort the message before a human sees it.
Start with good monitoring coverage: uptime checks, DNS checks, SSL checks, API checks, and the user journeys that prove your product works. This guide starts after those checks exist. The goal is to turn many check results into a small number of trusted monitoring alerts.
The right approach is easier to see side by side:
| Bad approach | Right approach |
|---|---|
| Every failed check pages the on-call engineer | Only urgent, user-impacting failures page |
| Warnings and outages use the same channel | Critical, warning, and info alerts use different routes |
| Alerts go to a general engineering channel | Alerts go to the responsible team or on-call rotation |
| One outage creates five separate alerts | Related alerts are grouped into one incident |
| Old noisy alerts stay forever | Unused alerts are deleted or downgraded |
| Planned maintenance pages the team | Maintenance windows mute only expected alerts |
Alert fatigue: what to page, downgrade, or remove
Alert fatigue happens when engineers receive so many low-value alerts that they stop trusting the alert channel. This is usually a process problem, not a personal problem, and the common causes are predictable.
- The alert has no clear action. "CPU is high" tells the responder almost nothing. Is customer traffic failing? Is a queue backing up? Is there a deploy in progress? If the alert does not point to a next step, it should not page.
- The alert is not urgent. A certificate that expires in 30 days matters, but it does not need a 3 AM phone call. It belongs in Slack, email, or a ticket queue. A certificate that expires tomorrow may need business-hours escalation. An expired production certificate needs a page.
- The alert repeats another alert. One database outage can trigger API errors, checkout failures, queue warnings, failed jobs, and status page changes. If those arrive as separate alerts, the on-call engineer spends the first ten minutes sorting them instead of fixing the issue.
- The alert has no owner. A general engineering channel is not ownership. Every alert should map to a service, a team, or an on-call rotation. "Someone should look" usually means nobody will.
- The alert has too little context. The responder needs the failed service, severity, region, status, runbook link, dashboard link, and escalation path. An unclear alert makes the engineer rebuild context while half awake.
The durable fix is a notification policy that sorts alerts before they reach people.
Monitoring alerts: signal vs noise
Good alerts start from user impact. Bad alerts start from a metric that was easy to graph.
Use this test: if the alert fires, can the responder take a clear action right now to protect users or revenue? If yes, it may deserve a page. If no, downgrade it.
Strong monitoring alerts usually look like this:
| Case | Why it matters | Best route |
|---|---|---|
| Customer-facing endpoint is down from more than one region | Users are blocked, not just one probe | Page the on-call engineer |
| Error rate crosses the service's SLO burn threshold | You are spending the error budget too fast | Page or Slack with notification, based on severity |
| P95 or P99 latency stays above the user-visible limit | The product is slow enough for users to feel it | Slack with notification, page if revenue path is affected |
| Core background job stops making progress | Work is stuck even if the API still returns 200 | Page if user data or billing is affected |
| Database load is high because one user runs repeated expensive queries | One customer can slow the whole product for everyone | Slack with notification, page if production health is at risk |
| Dependency needed for checkout, login, billing, or deployment is unreachable | Your product may fail because another service failed | Page for critical paths, Slack for non-critical paths |
| Certificate is expired or close enough to expiry that renewal probably failed | Users or API clients may be blocked soon | Slack or email early, page when expired |
| DNS records no longer match expected values | Traffic may go to the wrong place | Page for production domains |
For database load, make the alert specific. "DB load high" is weak. "User 123 is running a 10-year count query 80 times in 5 minutes and production latency is rising" is useful. That tells the responsible engineer what to check and what to stop.
Weak alert candidates should usually be downgraded:
| Case | Better route | Why |
|---|---|---|
| Single-server CPU or memory usage without customer impact | Dashboard or log warning | It may be useful context, but it is not a page |
| One failed check from one region that recovers on the next run | Log warning or silent Slack | One probe can fail because of network noise |
| A warning that has fired every week for months without action | Delete, tune, or make a ticket | Repeated ignored alerts create alert fatigue |
| Informational deploy events | Log or silent Slack | Useful history, not an alert |
| Errors from non-production routed to production on-call | Team Slack only | Staging should not wake production on-call |
| Runbook says "wait and see" | Dashboard or ticket | If no action is expected, it should not page |
Some important alerts start inside your code, not only in external monitoring. Use your logger and error tools to mark business-critical cases.
| Code-level signal | Log level | Sentry/Grafana rule | Best route |
|---|---|---|---|
| Parser fails for a top-tier customer feed | Critical/error | Alert when failures > 0 for that customer in 5 minutes | Page responsible engineer |
| Billing webhook signature fails repeatedly | Critical/error | Alert when failures spike above baseline | Page on-call engineer |
| Login token validation fails for many users | Critical/error | Alert on error rate by endpoint and status code | Page on-call engineer |
| Data export job misses its SLA | Warning | Alert when job age exceeds expected window | Slack with notification |
| Optional enrichment API times out | Warning | Track rate and latency on dashboard | Slack or ticket |
| Debug-only parsing mismatch in staging | Warning/info | Dashboard only | Log warning |
For example, if your app has parsers that extract data for top-tier clients, do not rely only on a generic "worker failed" alert. Add a clear code-level signal such as:
- client tier: top-tier, paid, trial
- parser name: invoices, shipments, uptime imports, HTML status parser
- input source: webhook, S3 file, API sync
- failure reason: schema mismatch, HTML changed, selector missing, auth error, empty response, timeout
- affected customer or workspace id
Then build rules on top of that signal:
- Critical: top-tier parser fails and no successful run happens within the next 5 minutes.
- Warning: paid-customer parser has more than 3 failures in 30 minutes but later recovers.
- Log only: trial-customer parser has one malformed optional field and the import still succeeds.
Parser alerts need a clear failure reason. "Parser failed" is too broad. If your parser reads HTML from a third-party site and that site changes its markup, the alert should say what changed:
- expected selector was not found
- table column count changed
- required JSON-LD field is missing
- date format changed
- login page appeared instead of data page
- response is empty or blocked
That detail makes the fix much faster. The responsible engineer can open the parser code, update the selector or schema handling, add a test case for the new HTML, and deploy the fix. Without the reason, they first have to reproduce the failure, compare old and new HTML, and guess what broke.
Tools like Sentry can turn repeated exceptions into issues and alert rules. Grafana can turn logs and metrics into dashboards and alert rules. The important part is the filter: do not alert on every exception. Alert on the exceptions that match a real business case, a responsible owner, and a clear action.
Logging is the raw material for many code-level alerts. If your team is still deciding how to structure logs in Node.js, see Winston vs Pino: Choosing a Node.js Logger in 2026. For the broader split between logs, metrics, and alerts, see Monitoring and Logging: How They Work Together.
Defining alert rules in DevHelm, Sentry, and Grafana
Most teams use more than one alert source. That is fine, but each tool should own the rule type it is best at.
| Tool | Best alert rules | Bad alert rules |
|---|---|---|
| DevHelm | External checks: uptime, API, DNS, SSL, heartbeat, dependency health | Internal stack traces that only exist inside app code |
| Sentry | Exceptions, failed jobs, parser errors, customer-impacting code failures | Simple uptime checks or "is the site reachable?" |
| Grafana | Metrics and logs: database load, queue depth, CPU, memory, error-rate trends | Single business events that need customer context |
Use DevHelm when the question is "does the service work from outside?"
| DevHelm rule | Example route |
|---|---|
| API returns 5xx from two regions | Page on-call engineer |
| SSL certificate is under 30 days | Slack or ticket |
| DNS record changed from expected value | Page platform on-call |
| Heartbeat job missed two runs | Slack with notification, page if billing/data is affected |
Use Sentry when the question is "did the code fail in a way that matters?"
| Sentry rule | Example route |
|---|---|
| New critical exception in checkout flow | Page on-call engineer |
| Parser fails for top-tier customer | Page responsible engineer |
| Same auth exception affects 20 users in 5 minutes | Slack with notification or page |
| Error appears only in staging | Team Slack or issue, no production page |
Use Grafana when the question is "is a metric moving into a dangerous range?"
| Grafana rule | Example route |
|---|---|
| Database CPU/load stays high for 10 minutes | Slack with notification, page if latency rises |
| Queue depth grows for 15 minutes | Slack with notification |
| Error rate crosses SLO burn threshold | Page or Slack based on severity |
| Memory rises slowly but users are fine | Dashboard or ticket |
The rule of thumb: DevHelm watches the outside user path, Sentry watches code failures, and Grafana watches system trends. Route them through the same notification policy model so the on-call engineer sees one clear incident instead of three disconnected alerts.
The stronger pattern is to alert on user-visible symptoms and keep causes as context. "Checkout is failing from us-east and eu-west" is the page. "Database connection pool is full" is useful detail. The on-call engineer needs both, but the user-visible problem should decide whether someone gets woken up.
Google's SRE guidance makes this point clearly: pages should be urgent, useful, new, and tied to user-visible problems. Prometheus Alertmanager follows the same idea with grouping, inhibition, and silences. The modern version is simple: classify first, notify second.
How severity levels route monitoring alerts
Severity connects detection to notification. Without severity, every alert looks equally urgent. With severity, critical outages can page people, while warnings stay out of the pager path.
Use a small scale. Most teams need three alert severities:
| Severity | Meaning | Notification behavior |
|---|---|---|
| Critical | Customer-facing outage, data loss risk, or broken revenue path | Page on-call immediately |
| Warning | Degradation, higher risk, or upcoming failure | Slack, email, or business-hours escalation |
| Info | Useful operational context | Dashboard, log, or digest |
This should match your incident severity model. A critical alert usually maps to a Sev1 or Sev2 incident. A warning may become a Sev3 if it continues or gets worse. Info should rarely become an incident unless a human promotes it.
Severity should be set at the source. Do not ask the notification channel to guess. The monitor, alert rule, or policy should know whether a failed check is critical or warning based on the service, environment, assertion, and confirmation window.
For a deeper incident triage model, see Incident Severity Levels: Sev1-Sev4 with Triage Matrix. The simple alerting rule is this: if two severities notify the same people, through the same channel, with the same urgency, one level is not needed.
Notification channels for monitoring alerts
Alert fatigue often gets worse because teams treat channels as personal choices instead of alerting tools. Slack, email, SMS, phone, webhooks, PagerDuty, and OpsGenie each have a job.
Slack is good for team awareness. It works for warnings, incident channels, resolved alerts, and routing context. It is weak as the only path for critical pages because people mute channels, miss threads, and close laptops.
Email is good for low-urgency follow-up. It works for reports, digests, and non-urgent warnings. It is poor for incident response because inboxes are crowded and delivery is not always fast.
SMS and phone are good for critical pages. They are loud by design. Save them for problems that need a human now.
Webhooks are good for automation. Send alerts to a workflow tool, ticket system, incident tool, or custom responder. A webhook should not be the only record of a critical incident unless the receiver is reliable.
PagerDuty and OpsGenie-style tools are good for on-call schedules. They know who is on-call, how to escalate, and whether the first responder acknowledged. They work best when the monitoring tool sends clean alerts with clear severity.
Here are ten common routing examples:
| Case | Best route | Why |
|---|---|---|
| Checkout API returns 500s from two or more regions | Page the on-call engineer | Users cannot pay, so action is needed now |
| Login is down for production users | Page the on-call engineer | Users cannot enter the product |
| Database is unavailable for the main app | Page the responsible backend or platform on-call | The owner can act fastest |
| Payment provider webhook is failing | Page on-call and post to incident Slack | Revenue path is at risk |
| API latency is high but requests still succeed | Slack with notification | The team should look soon, but it may not need a page |
| SSL certificate expires in 21-30 days | Email or ticket | Important, but not urgent |
| SSL certificate expires tomorrow | Slack with notification, then escalate during business hours | Renewal likely failed and needs fast follow-up |
| Staging monitor fails | Slack without notifying the whole team | Useful for developers, not production on-call |
| CPU crosses 70% for two minutes | Log warning or dashboard | Context only unless users are affected |
| A deploy starts or finishes | Log or silent Slack message | Useful history, not an alert |
Notification policy setup for monitoring alerts
A notification policy is a set of rules for alerts. It decides where an alert goes, who sees it, how it groups with related alerts, and whether it should be muted.
Think of it as a routing tree:
- What service or monitor generated the alert?
- What environment did it come from?
- What severity is it?
- Has the same incident already opened?
- Is a maintenance window active?
- Is quiet-hours behavior different for this severity?
- Which channel or escalation chain should receive it?
That tree gives you control. A critical checkout outage can page the primary on-call engineer, post to the incident channel, and update the status page. A warning on a staging monitor can go to Slack during business hours. A planned maintenance event can mute expected failures without turning the monitor off.
Good policies usually route by these attributes:
- Severity: critical, warning, info.
- Environment: production, staging, development.
- Service or tag: checkout, API, auth, billing, database.
- Region: global failure vs. single-region degradation.
- Monitor type: HTTP, DNS, SSL, TCP, heartbeat.
- Ownership: team or rotation responsible for the service.
- Time window: business hours, quiet hours, planned maintenance.
The default policy matters too. Alerts that do not match a specific rule should not disappear. Send them to a visible catch-all channel with enough context to fix the policy. A missing route is a config bug.
For teams using monitoring as code, notification policies should live beside monitor definitions. The monitor says what to check. The policy says who gets notified when that check fails.
Alert routing and escalation for on-call engineers
Routing answers one question: who should get this alert first? Escalation answers a different one: what happens if they do not respond?
Mixing these two ideas creates fragile alerting. A route should be based on ownership and context. An escalation chain should be based on time and responsibility.
Example escalation:
| Time since alert | Action |
|---|---|
| 0 minutes | Notify primary on-call |
| 5 minutes unacknowledged | Notify secondary on-call |
| 15 minutes unacknowledged | Notify engineering lead |
| 30 minutes unresolved for critical severity | Open broader incident response |
Test both layers. A good routing policy is useless if the backup never gets paged. A good escalation chain is useless if every database alert goes to the wrong team first.
Alert deduplication, grouping, and suppression
The most useful alerting work usually happens before the notification is sent.
Deduplication stops the same alert from notifying again and again. If the checkout monitor fails every 30 seconds for ten minutes, the on-call engineer should not receive twenty identical pages. They should receive the first alert, updates when the state changes, and a resolved message when the monitor recovers.
Grouping combines related alerts into one incident or notification. During a dependency outage, many monitors may fail for the same reason. Grouping by service, dependency, severity, and time window keeps the responder focused.
Suppression stops expected or lower-value alerts from notifying people. Maintenance windows are the common case. If you know the database will restart during a planned migration, mute that alert for the window but keep the monitor running.
Inhibition mutes smaller symptoms when a bigger alert already explains them. If "cluster unavailable" is firing, individual pod warnings do not need to page. They can stay visible as context without becoming separate pages.
These controls separate alert management from simple message delivery. A PagerDuty alternative, an incident management platform, or a monitoring tool can all send messages. The important question is whether the tool reduces duplicate human work before the alert reaches a human.
On-call engineers need trusted alerts
An on-call rotation depends on alert quality. You can design a fair schedule, pay engineers properly, and write good handoff notes. The team can still burn out if monitoring alerts are low value.
The target is trusted alerts. A quiet system may not monitor enough. A noisy system can still miss the failure that matters.
Credible alerts have five traits:
- The responder understands what broke.
- The alert maps to customer or business impact.
- The route points to the team that can act.
- The alert includes context and a runbook.
- The alert history shows that pages usually matter.
For schedule design, handoffs, pay, and rotation models, read On-Call Rotation Best Practices for Engineering Teams. This guide focuses on the input side: making sure the on-call engineer receives signal instead of noise.
Quiet hours and maintenance windows for alert fatigue
Quiet hours do not mean critical alerts go silent. They mean non-critical alerts wait until a better time.
A good quiet-hours policy usually looks like this:
| Alert severity | During business hours | During quiet hours |
|---|---|---|
| Critical | Page immediately | Page immediately |
| Warning | Slack or ticket | Queue for morning unless worsening |
| Info | Dashboard or digest | Dashboard or digest |
Maintenance windows mute expected alerts during planned work. The important word is expected. If you are restarting a database, mute the database availability alert for that window. Keep the rest of production alerting active.
Use narrow matchers:
- Specific service or monitor.
- Specific environment.
- Specific time window.
- Specific expected failure mode.
Broad silences are dangerous because they hide unrelated incidents. A maintenance window for checkout should not mute auth, DNS, or the public status page. Keep the policy narrow so an unrelated critical failure still reaches on-call.
Measuring monitoring alert quality
You can measure alert quality. If you do not measure it, alerting gets worse over time.
Track these metrics monthly:
| Metric | What it tells you |
|---|---|
| Alerts per on-call shift | Human load |
| Pages per incident | How well duplicate alerts are grouped |
| Actionable alert rate | Signal-to-noise ratio |
| MTTA | Whether alerts are trusted and routed to the right person |
| False positive count | Whether thresholds are too sensitive |
| Alerts with runbooks | Whether responders have clear next steps |
| Alerts by service owner | Ownership gaps |
| Suppressed alerts during maintenance | Whether planned work is being handled cleanly |
MTTA is especially useful. It measures the time between alert delivery and human acknowledgment. A rising MTTA can mean the alert is hard to see, routed to the wrong person, or ignored because the channel is noisy. For the full incident metric model, see MTTA, MTTR, MTBF, MTTF - The Four Incident Metrics, Compared.
Run a monthly alert review with three questions:
- Which alerts fired most often?
- Which alerts did not lead to action?
- Which incidents produced more than one page?
Delete, downgrade, group, or rewrite anything that fails the review. Alerting is not a one-time setup. Review it every month.
Monitoring alert setup example
Here is a simple policy for a small SaaS team with an API, web app, checkout flow, and public status page.
Start with monitors:
| Monitor | Failure behavior |
|---|---|
| Homepage uptime | Warning for one-region failure, critical for multi-region failure |
| API health endpoint | Critical on multi-region 5xx or timeout |
| Checkout endpoint | Critical on failed assertion or elevated latency |
| DNS records | Warning on record drift, critical on resolution failure |
| SSL certificate | Warning under 30 days, critical when expired |
| Background job heartbeat | Warning after one missed interval, critical after sustained failure |
Then define alert channels:
| Channel | Purpose |
|---|---|
| Engineering Slack | Warnings, resolved alerts, context |
| Primary pager | Critical production pages |
| Secondary pager | Unacknowledged critical escalation |
| Email or ticket queue | Non-urgent follow-up |
| Webhook | Automation, incident creation, or audit record |
Then define notification policies:
| Rule | Destination |
|---|---|
| production + critical | Primary pager immediately, engineering Slack for visibility |
| production + critical + unacknowledged after 5 minutes | Secondary pager |
| production + warning | Engineering Slack during business hours, queue after hours |
| staging + critical | Team Slack only |
| info | Dashboard or digest |
| maintenance window match | Mute matched warning/critical alerts for that service only |
Finally, attach runbook and ownership metadata:
| Metadata | Why it matters |
|---|---|
| Service owner | Prevents "who owns this?" delay |
| Runbook URL | Gives the first responder a next step |
| Dashboard URL | Speeds diagnosis |
| Status page component | Keeps customer updates tied to the same signal |
| Recent deploy link | Helps spot change-related incidents |
Runbooks are the most important link in this table. A good alert should not just say what broke; it should point to the first action. For a deeper format, see Runbooks: Anatomy, Examples, and the AI-Executable Format.
This setup is small on purpose. Start with a few alerts you trust. Add more only after the signal is good. Teams get into trouble when they create dozens of alerts before they have routing, severity, and review habits.
Setting up monitoring alerts with DevHelm
DevHelm splits alerting into two reusable parts: alert channels and notification policies.
Alert channels are destinations. They can be Slack, email, SMS, webhooks, PagerDuty, OpsGenie, or another place an alert can go. Create the destination once, test it, and reuse it across monitors. For the exact setup flow and supported channel types, see the alert channels reference.
Notification policies are routing rules. They decide which channel fires for each monitor, severity, tag, service, or escalation step. This is where you define quiet hours, severity routing, deduplication, escalation chains, and maintenance windows. For match rules, priority, and evaluation order, see the notification policies reference.
A clean DevHelm setup follows this order:
- Create the first monitor and first alert channel. If you are starting from zero, follow First HTTP monitor, then First alert.
- Create channels for the real destinations: engineering Slack, critical pager, secondary escalation, and ticket/email follow-up. For end-to-end setup, see the alerting guide.
- Tag monitors by service and environment: production, staging, checkout, API, auth, billing. If you route by ownership, use alert routing by tag.
- Set severity based on impact: critical for customer-facing outages, warning for degradation or upcoming failure, info for context.
- Build notification policies from specific to broad: checkout critical first, production critical next, warnings after that, catch-all last.
- Add escalation only where the first responder might miss the alert. For a three-step model, see tiered escalation.
- Add quiet-hours behavior for warnings and info.
- Add maintenance windows for planned work. For scoped planned downtime, see maintenance windows.
- Test the full path from failed monitor to delivered notification before trusting it in production. For the test flow, see testing your alerts.
The key is reuse. Do not configure custom alert behavior on every monitor unless the monitor is truly special. A policy should define the team's alerting model once. New monitors should follow that model through tags, severity, and ownership.
That is how you keep alerting clean. When the payments team changes its escalation chain, update the payments policy. When Slack channel names change, update the channel. When checkout becomes more critical, update the severity rule. Do not edit twenty monitors by hand.
PagerDuty alternative vs better monitoring alerts
Many teams searching for a PagerDuty alternative are really searching for less alert fatigue. These problems are related, but each layer has a different job.
PagerDuty, OpsGenie, Grafana OnCall, incident.io, Rootly, Better Stack, and other incident tools handle schedules, escalation, acknowledgment, and response workflows. They are useful when you need on-call schedules and escalation. They do not automatically fix bad monitoring alerts.
If the monitoring layer sends noisy alerts with little context, the on-call tool will just route noisy alerts faster. Signal quality has to improve before the pager.
The better evaluation question is:
- Can the monitoring layer classify severity before paging?
- Can related failures deduplicate into one incident?
- Can warnings avoid the pager path?
- Can maintenance windows suppress expected failures without hiding unrelated outages?
- Can the alert include owner, runbook, region, and service context?
- Can the incident tool escalate only after a clean alert has reached the right first responder?
For tool comparisons, see Best Incident Management Tools in 2026 and Opsgenie Alternatives in 2026. This guide has a narrower point: whatever pager you use, fix the signal before you optimize the route.
Monitoring alert audit checklist
Use this checklist for every paging alert:
| Question | What a good answer looks like |
|---|---|
| Is the condition user-visible or almost user-visible? | It affects users, revenue, data, or a critical dependency |
| Does it require action now? | Waiting until tomorrow would make the incident worse |
| Is the responder able to act? | The route goes to the team that owns the service |
| Is the owner clear? | The alert names a service, team, or on-call rotation |
| Is the severity correct? | Critical pages; warning goes to Slack or ticket; info stays quiet |
| Is the runbook linked? | The first responder has a clear next step |
| Will duplicate alerts group together? | One root cause creates one incident, not five pages |
| Is there a business-hours path for warnings? | Non-urgent warnings do not wake people at night |
| Is maintenance muting narrow? | Only the planned service and time window are muted |
| Did this alert lead to action in the last 90 days? | If not, delete it, tune it, or downgrade it |
If an alert fails the checklist, do not leave it as a page. Rewrite it, downgrade it, group it, or delete it.
The best alerting systems are boring in the right way. Most signals go to dashboards, tickets, or quiet Slack channels. A small number reach the on-call engineer. When they do, the engineer trusts the page.
FAQ
What is the difference between monitoring and alerting?
Monitoring collects health signals from systems, services, and user journeys. Alerting decides which signals need human attention and sends them to the right place.
What is the best way to reduce alert fatigue quickly?
Start with the ten noisiest alerts from the last 30 days. Delete alerts that never led to action, downgrade non-urgent warnings, and group duplicate symptoms from the same incident.
Should warning alerts ever wake up an on-call engineer?
Usually no. A warning should page only when it predicts a near customer-facing failure and waiting until business hours would make the incident worse.
How many notification policies should a small team have?
Most small teams can start with four notification policies: production critical, production warning, non-production, and catch-all. Add service-specific policies only when ownership or severity handling is different.
Is PagerDuty a monitoring tool?
PagerDuty is mainly an on-call and incident response tool. It routes, escalates, and tracks alerts from monitoring systems, but it usually depends on another tool to detect the failure first.
What should every alert include?
Every alert should include the affected service, severity, environment, current state, first failed time, owner, runbook link, dashboard link, and the notification policy that routed it.
DevHelm routes monitoring alerts through reusable alert channels and notification policies, with severity-based routing, escalation chains, quiet hours, deduplication, and maintenance windows. Start with your first production monitor at app.devhelm.io, then set up the alerting policy before the next incident tests it for you.
Originally published on DevHelm.
Top comments (0)