DEV Community

Hive80-lab
Hive80-lab

Posted on

The On-Call Schedule That Saved My Team 20 Hours a Week

The On-Call Schedule That Saved My Team 20 Hours a Week

Bad on-call schedules don't just burn out engineers — they burn money.

When I took over a 4-person ops team, the on-call schedule was a mess: one person was on call 24/7 for a week at a time. They were exhausted, making mistakes, and quietly looking for new jobs. The team's incident response time was 45 minutes — not because the problems were hard, but because the on-call engineer was too tired to think clearly.

We restructured the schedule. Within a month, incident response time dropped to 8 minutes. Within three months, zero people had quit. Here's what changed.

The Problem: Hero Culture

The old schedule looked like this:

Week 1: Alice (24/7)
Week 2: Bob (24/7)
Week 3: Charlie (24/7)
Week 4: Diana (24/7)
Enter fullscreen mode Exit fullscreen mode

Each person was on call for 168 hours straight. No secondary. No escalation. If Alice was at her kid's soccer game and a P1 fired, she had to choose between family and work.

The result:

  • Average response time: 45 minutes
  • Burnout rate: 2 people quit in 6 months
  • Error rate during incidents: 23% (exhausted people make mistakes)
  • After-hours pages: 12/week (most were non-urgent)

The Fix: Primary/Secondary with Follow-the-Sun

The New Schedule

Week 1:
  Mon-Tue: Primary=Alice, Secondary=Bob
  Wed-Thu: Primary=Bob, Secondary=Charlie
  Fri-Sun: Primary=Charlie, Secondary=Diana

Week 2:
  Mon-Tue: Primary=Diana, Secondary=Alice
  Wed-Thu: Primary=Alice, Secondary=Bob
  Fri-Sun: Primary=Bob, Secondary=Charlie
Enter fullscreen mode Exit fullscreen mode

Key principles:

  1. No one is on call for more than 48 hours — 2 days max
  2. Always a secondary — primary can escalate without guilt
  3. Rotation includes weekends — shared burden, no one always gets weekends
  4. Handoff at 9 AM — not midnight, not 5 PM

The Implementation Script

#!/usr/bin/env python3
"""schedule.py - Generate on-call rotation"""
from datetime import datetime, timedelta
import json

team = ['Alice', 'Bob', 'Charlie', 'Diana']
shift_length = 2  # days

def generate_schedule(weeks=4, start_date=None):
    if start_date is None:
        start_date = datetime.now().replace(hour=9, minute=0, second=0)

    schedule = []
    current = start_date
    primary_idx = 0
    secondary_idx = 1

    for week in range(weeks):
        for day in range(0, 7, shift_length):
            primary = team[primary_idx % len(team)]
            secondary = team[(primary_idx + 1) % len(team)]

            schedule.append({
                'start': current.isoformat(),
                'end': (current + timedelta(days=shift_length)).isoformat(),
                'primary': primary,
                'secondary': secondary,
                'duration_days': shift_length
            })

            current += timedelta(days=shift_length)
            primary_idx += 1

        # Rotate starting position each week
        primary_idx = week + 1

    return schedule

schedule = generate_schedule(weeks=4)
print(json.dumps(schedule, indent=2))
Enter fullscreen mode Exit fullscreen mode

Alert Routing

#!/usr/bin/env python3
"""alert_router.py - Route alerts to the right person"""
import requests
from datetime import datetime

def get_on_call():
    """Get current primary and secondary from schedule"""
    # Query your schedule system (Google Calendar, PagerDuty, etc.)
    now = datetime.now()
    # ... return primary and secondary
    return {'primary': primary, 'secondary': secondary}

def route_alert(severity, service):
    on_call = get_on_call()

    if severity == 'P1':
        # Page primary immediately
        page_person(on_call['primary'], service, severity)
        # Set 5-minute escalation to secondary
        schedule_escalation(on_call['secondary'], minutes=5)
    elif severity == 'P2':
        # Slack primary, page if no response in 15 min
        slack_notify(on_call['primary'], service, severity)
        schedule_escalation(on_call['secondary'], minutes=15)
    elif severity == 'P3':
        # Slack only, no paging
        slack_notify(on_call['primary'], service, severity)
    else:
        # Log only, review in morning
        log_alert(service, severity)
Enter fullscreen mode Exit fullscreen mode

The Results After 3 Months

Metric Before After Change
Avg response time 45 min 8 min -82%
Burnout rate 2 quits/6mo 0 quits/6mo -100%
Error rate 23% 4% -83%
After-hours pages 12/week 3/week -75%
Team satisfaction 3.2/10 8.1/10 +153%

The 20 Hours Saved

Where did the 20 hours go?

  • 8 hours: Reduced incident investigation time (faster response = fresher context)
  • 6 hours: Fewer false alarm pages (proper severity routing)
  • 4 hours: No more schedule disputes and handoff confusion
  • 2 hours: Reduced post-incident meetings (fewer mistakes = simpler post-mortems)

The Cultural Change

The schedule change had an unexpected side effect: it made the team more proactive. When people aren't exhausted, they fix root causes instead of patching symptoms. In the first 3 months after the change:

  • Root cause fixes: 14 (vs 3 in the previous 3 months)
  • Recurring incidents: Down 60%
  • Proactive improvements: 22 tickets created (vs 4 previously)

Getting Started

  1. Audit your current schedule — how long is each person on call?
  2. Add a secondary — even if it's just for escalation
  3. Reduce shift length — 2-3 days max per primary
  4. Route by severity — not every alert needs a page
  5. Review monthly — ask the team what's working and what's not

Want the complete on-call scheduling toolkit? The Ops Starter Kit includes the scheduling script, alert routing templates, severity classification guide, and post-incident review templates.

How long is your current on-call rotation? Could a shorter shift save your team 20 hours too?

Top comments (0)