A misconfigured escalation policy has two failure modes, and both are bad. Either every engineer on the team gets paged for a minor hiccup, or nobody gets paged at all because the alert fell into a routing gap. The first creates fatigue. The second creates outages that last until a customer complains.
The difference between those outcomes is rarely the monitoring tool. It is the routing logic between "check failed" and "the right human is looking at it." That routing logic is the escalation policy.
This guide covers how to design escalation policies that match alert urgency to responder availability — without turning on-call into a sleep-deprivation experiment.
What an escalation policy actually does
An escalation policy is a set of rules that determines who gets notified about an incident, through which channel, and what happens if they do not respond within a defined window.
It sits between the alert rule and the human. The alert rule says "something is wrong." The escalation policy says "tell this person, then that person, then that person's manager, in this order, through these channels, at these intervals."
Without one, alerts go to a shared Slack channel where everyone assumes someone else is handling it. With a poorly designed one, alerts go to everyone simultaneously and nobody knows who owns the response.
A good escalation policy answers three questions:
- Who should respond first? Usually the on-call engineer closest to the affected service.
- What happens if they don't respond? The alert moves to the next tier after a defined timeout.
- When should you skip the chain entirely? Some incidents are severe enough to page multiple people from the start.
Tiered escalation design
Most teams benefit from a three- or four-tier model. Each tier represents a wider circle of responders with increasing authority and decreasing system-specific context.
Tier 1 — Primary on-call. The engineer currently carrying the pager for the affected service. They have the most context and are expected to acknowledge within 5 minutes. This is the person who should handle 80% of alerts without escalation.
Tier 2 — Backup on-call. A second engineer on the rotation, often the previous week's primary. If Tier 1 does not acknowledge within the timeout window, the alert moves here. The backup may have slightly less context on recent changes but knows the system well enough to investigate.
Tier 3 — Engineering lead or team lead. If both on-call engineers are unreachable — phone off, traveling, handling a separate incident — the alert reaches someone with authority to pull in additional help, initiate a broader response, or contact people outside the rotation.
Tier 4 — Management or incident commander. Reserved for extended outages where Tiers 1–3 are either overwhelmed or the incident has business-level impact requiring communication with stakeholders, customers, or leadership.
The key insight: each tier adds responders without removing the previous ones. When Tier 2 is paged, Tier 1 is still expected to respond if they become available. The chain widens the net; it does not shift ownership.
Time-based auto-escalation
The most common escalation trigger is an unacknowledged alert after a fixed window. A reasonable starting point:
- 0 minutes: Alert fires. Primary on-call is paged via push notification and phone call.
- 5 minutes unacknowledged: Backup on-call is paged.
- 15 minutes unacknowledged: Engineering lead is paged. The alert is now treated as potentially unattended.
- 30 minutes unacknowledged: VP or incident commander is notified. At this point the question is not "who can fix this" but "why has nobody responded."
These windows are not universal. A team with excellent acknowledgment rates might extend them to 10/20/40. A team covering a payment-processing system where every minute matters might compress them to 3/10/20. The right intervals come from your actual response data — track acknowledgment times for a month and set the first escalation threshold just above your p90.
One nuance that is easy to miss: acknowledgment is not resolution. "I see this and I am looking at it" is enough to pause the escalation clock. The goal of escalation is to confirm someone is aware, not to confirm the fix is deployed.
When to skip levels
Not every incident should walk the chain. A complete site outage affecting all customers should not wait 15 minutes for the backup to get a turn. Severity-based skip logic lets critical alerts bypass the queue.
A common pattern ties incident severity levels to escalation behavior:
| Severity | Escalation behavior |
|---|---|
| SEV4 / Informational | Slack notification only, no page |
| SEV3 / Low | Page primary on-call, normal escalation chain |
| SEV2 / High | Page primary and backup simultaneously |
| SEV1 / Critical | Page all tiers at once, open an incident channel |
The deciding factor is impact scope. A single failing health check from one region is SEV3 — the primary can investigate. All checks failing across all regions is SEV1 — you want three people looking at it in parallel, not in sequence.
Severity classification should happen at the alert rule level, not during the incident. If the on-call engineer has to manually decide "is this bad enough to wake up the lead," you have already lost minutes. Pre-define severity thresholds in your monitoring alerts configuration so the routing is automatic.
Override mechanisms
Schedules are plans. Reality is messier. Escalation policies need escape hatches:
Temporary overrides let an on-call engineer hand off the pager for a few hours — a doctor's appointment, a flight, a family event. The override routes alerts to the backup without changing the rotation schedule. Most on-call tools support this natively; if yours does not, a shared calendar with manual swaps works but introduces coordination risk.
Direct page allows anyone to invoke the escalation chain for an incident they observe but cannot fix themselves. A frontend engineer noticing database connection errors should be able to page the database on-call without navigating the org chart.
Escalation suppression prevents known maintenance from triggering the chain. If the team is performing a scheduled database migration that will cause transient errors for 20 minutes, suppress escalation for the affected monitors during the window. The alerts should still fire (so you have the audit trail), but they should not page anyone.
Manual escalation lets the responding engineer bump an incident to the next tier without waiting for the timeout. If the primary sees something they cannot diagnose — an unfamiliar service, a permissions issue, an infrastructure problem outside their domain — they should escalate immediately rather than burning the clock.
Channel routing by severity
Escalation is not just about who gets paged. It is about how. A 3 AM phone call carries a very different weight than a Slack message, and using the wrong channel for the wrong severity erodes trust in the system.
A practical channel matrix:
| Severity | Channels |
|---|---|
| SEV4 | Slack or email only |
| SEV3 | Push notification + Slack |
| SEV2 | Push notification + phone call + Slack |
| SEV1 | Phone call + SMS + push + Slack + incident channel |
The principle: as severity increases, add channels rather than switching them. A SEV2 alert still goes to Slack so the rest of the team has visibility, but the primary responder gets a phone call they cannot miss.
DevHelm's notification policies work this way — you map severity levels to channels so a warning goes to Slack while a critical failure pages via phone and creates an incident. The routing is defined once and applied to every alert that matches the severity, which keeps the configuration in one place instead of scattered across individual monitors.
Building the on-call / backup structure
The escalation policy is only as good as the rotation behind it. A few structural patterns reduce the chance of an alert falling through:
Always have a backup. A single-person on-call rotation with no escalation path means one unanswered phone call equals an unattended incident. Even a two-person team should have a primary/backup split.
Rotate backup independently. If the primary rotates weekly, the backup should too — but offset by one week. This means last week's primary is this week's backup, which maximizes context continuity.
Avoid hero culture. If the same senior engineer is always the final escalation tier, that person becomes a single point of failure and a burnout risk. Distribute the lead tier across multiple senior engineers on a rotation of their own. For more on sustainable rotation design, see on-call rotation best practices.
Document who is on call right now. It sounds obvious, but the most common escalation failure is not a misconfigured policy — it is an out-of-date schedule. Automate schedule publishing so the current on-call is always visible in your incident tooling.
Measuring whether it works
An escalation policy is not a set-and-forget artifact. Track these metrics monthly:
- Acknowledgment time (p50 and p90). If p90 exceeds your first escalation window, either the window is too short or the primary on-call is routinely unreachable.
- Escalation rate. The percentage of alerts that reach Tier 2 or beyond. A healthy rate is under 15%. Above 30% means Tier 1 is consistently failing to respond — investigate whether it is a coverage gap, alert fatigue, or tooling friction.
- False escalations. Alerts that escalated because they were not acknowledged but turned out to be non-issues. These train backup and lead responders to deprioritize escalated alerts, which defeats the purpose.
- MTTR by tier. If incidents that escalate to Tier 3 take dramatically longer to resolve, the escalation chain may be routing to people who lack context. Consider adding domain-specific escalation paths instead of a single generic chain.
Review these numbers in your monthly on-call retrospective, alongside the data from your runbooks and incident postmortems. The escalation policy should evolve as the team, the services, and the alert volume change. Google's Site Reliability Workbook covers similar principles in the context of large-scale operations.
Start with the basics
You do not need a four-tier policy on day one. Start with primary → backup → lead, a 5-minute acknowledgment window, and severity-based routing for critical alerts. Measure acknowledgment and escalation rates for a month. Adjust the windows, add override mechanisms, and refine channel routing based on what the data shows.
The goal is not a policy that covers every edge case. It is a policy that ensures every alert reaches someone who can act on it — without waking everyone else up in the process. Set up your first notification policy with severity-based routing and multi-channel escalation in a few minutes.
Originally published on DevHelm.
Top comments (0)