On-call is the tax engineering teams pay for running production systems. Every organization that ships software to users eventually reaches the point where someone needs to be reachable when things break at 2 AM. Done well, on-call is sustainable — a shared responsibility that distributes evenly across the team and improves with every incident. Done poorly, it burns people out, concentrates knowledge in a few overloaded individuals, and quietly degrades the product as exhausted engineers stop investing in reliability.
The difference between those outcomes is almost never the tooling. It is the structure: how rotations are designed, how escalations work, how burden is distributed, and how the team treats on-call as a first-class engineering practice rather than an afterthought.
Rotation structures
There is no single correct rotation model. The right choice depends on team size, geographic distribution, and the criticality of the services being covered.
Weekly rotation is the most common default. One engineer carries the pager for a full week, then passes it to the next person. Simple to understand, simple to schedule. The downside is that a bad week — multiple incidents, high alert volume — falls entirely on one person with no relief until the handoff.
Biweekly rotation extends the shift to reduce the overhead of context-switching between on-call and non-on-call weeks. It works when alert volume is low (fewer than five pages per week on average). Beyond that threshold, two weeks starts to feel punishing.
Follow-the-sun distributes coverage across time zones so no one takes overnight pages. A team with engineers in US Pacific, European, and Asian time zones can cover 24 hours without anyone waking up at 3 AM. The tradeoff is coordination cost: handoff quality becomes critical, and you need at least three engineers per timezone to avoid single points of failure.
Primary/secondary split assigns two engineers per shift. The primary takes the initial page; if they don't acknowledge within the escalation window (typically 5–10 minutes), the secondary gets paged. This provides redundancy without doubling the roster size.
Service-ownership rotation maps on-call to specific services rather than a team-wide roster. The payments team covers payments; the platform team covers infrastructure. This works at scale (50+ engineers) where generalist on-call produces too much context-switching, but requires well-defined service boundaries and ownership clarity.
Schedule design
Shift length matters. Seven-day shifts are standard, but consecutive days beyond seven correlate strongly with burnout and error rates. If your rotation runs longer than a week, build in explicit rest days or shorten the active hours.
Overlap periods between shifts prevent the "not my problem anymore" gap. A 30-minute handoff window — where both outgoing and incoming engineer are reachable — catches incidents that fire right at the boundary.
Handoff rituals are the difference between "good luck" and useful context transfer. At minimum, the outgoing engineer should communicate: open incidents, anything flapping or degraded, recent deploys that haven't fully baked, and upcoming maintenance windows.
Weekend coverage deserves explicit design. Some teams split weekends separately from weekdays; others absorb them into the weekly rotation. The key decision is whether weekend pages carry additional compensation or comp time. Leaving this ambiguous creates resentment.
Time-zone-aware scheduling goes beyond follow-the-sun. Even within a single-timezone team, shift start times matter. Starting on Monday morning rather than Sunday night means the outgoing engineer doesn't stay up late on their last day.
Escalation policies
A page that goes unacknowledged is worse than no page at all — it means the system thinks someone is handling the incident while no one is. Escalation policies exist to guarantee every alert eventually reaches a human who acts on it.
Basic chain: Primary (acknowledge within 5 min) → Secondary (acknowledge within 5 min) → Engineering Manager → VP Engineering. Each tier adds urgency without skipping the people closest to the code.
Time-based escalation increases the responder pool as time passes without acknowledgment. This is the minimum viable escalation policy. If you have nothing else, implement this.
Severity-based routing sends different severity levels to different responders. A P3 informational alert goes to Slack only. A P2 goes to the primary on-call. A P1 pages both primary and secondary immediately. A P0 pages the entire escalation chain simultaneously.
De-escalation matters too. When an incident resolves without needing backup, the secondary should be notified (not paged) so they know the situation is handled. Over-escalation erodes trust in the system and trains people to ignore pages.
Reducing on-call burden
The single highest-impact action for improving on-call is reducing alert noise. Teams that page 20+ times per week cannot retain engineers on the rotation. The target is fewer than five pages per on-call shift — ideally fewer than two that require actual intervention.
Alert quality over quantity. Every alert should be actionable. If the on-call engineer looks at a page and routinely says "I can ignore this," that alert should be tuned, suppressed, or deleted. Document what the responder should actually do in structured runbooks — not just what triggered the alert.
For the full policy model behind that cleanup, see Monitoring Alerts: The Definitive Guide to Alerting Without Alert Fatigue.
Auto-remediation for known responses. If the runbook for an alert is "restart the pod" or "clear the queue," that's a candidate for automation. Every automated response removes a page from the rotation permanently.
Blameless postmortems feed back into the system. When an incident occurs, the postmortem should ask: could this alert have been prevented? Could the responder have been given better information? Could the resolution have been automated? Each answer improves future on-call shifts.
Consolidation windows batch non-urgent alerts rather than paging for each one individually. A monitoring system that fires three separate alerts for related symptoms — high latency, increased errors, connection pool exhaustion — should consolidate into a single incident rather than paging three times.
Compensation and fairness
On-call carries real cost to the engineer: interrupted sleep, restricted evenings, constrained weekend plans. Teams that treat this as "just part of the job" with no explicit compensation build resentment and face retention problems.
Paid on-call is the clearest model. Common structures include a flat per-shift stipend ($200–500/week in US markets), a per-page bonus, or a combination. Some organizations pay a higher rate for pages outside business hours.
Comp time offers time off in exchange for on-call shifts — typically 0.5–1 day per week of on-call. This works well when the team values time flexibility more than additional pay.
Rotation equity tracking ensures burden distributes fairly over time. Track pages-per-person, incidents-per-person, and weekend-shifts-per-person quarterly. If one engineer consistently gets paged more due to scheduling luck, adjust the rotation.
Opt-out for life events. Moving, new baby, medical issues, family emergencies — there should be an explicit, no-questions-asked path to temporarily exit the rotation. The team absorbs the extra coverage. Building this in structurally prevents people from burning through goodwill to get relief.
Common anti-patterns
The hero. One engineer who "doesn't mind" being on-call all the time. They accumulate all the context, make the rotation smaller for everyone else, and become an invisible single point of failure. When they leave — and they eventually leave — the team discovers how much tribal knowledge walked out the door.
Alerts that cry wolf. Pages that fire but require no action train the on-call engineer to ignore alerts. This is the most dangerous anti-pattern: it directly causes real incidents to get slow responses because the engineer assumes "probably nothing again." See Opsgenie alternatives for tools that help with noise reduction at the routing layer.
No handoff notes. The incoming engineer starts their shift blind, with no context about what has been flapping, what was recently deployed, or what the previous shift was watching. Every shift start should include at minimum a 5-minute written or async handoff.
Scope creep punishment. "You're on-call anyway, so can you also handle this customer request / deploy this hotfix / review this PR?" On-call is for incident response. Loading unrelated work onto on-call engineers makes the rotation feel punitive and discourages participation.
No feedback loop. Pages fire, engineers respond, nothing changes. Without postmortems that feed back into alert tuning, runbook updates, and automation investments, on-call burden stays static or grows. The rotation should get measurably better each quarter.
Start with the signals
Rotation design determines who gets woken up and how quickly they respond. But the quality of on-call depends entirely on the quality of the alerts feeding it. A rotation staffed by great engineers still fails if the monitoring underneath generates noise instead of signal.
On-call works when the alerts feeding it are precise. Multi-region monitoring with configurable severity and notification policies reduces noise before it reaches your rotation — start at app.devhelm.io.
Originally published on DevHelm.
Top comments (0)