The On-Call Runbook Every Small DevOps Team Needs (Free Template)
When the pager goes off at 3am, you need a runbook that tells you exactly what to do. Not a 200-page document nobody reads. A one-page runbook that gets you from alert to resolved as fast as possible.
This template is designed for teams of 1-10 people who run production without a dedicated SRE team.
When You Get Paged
- Acknowledge within 5 minutes - tells the team you're on it
- Check alert details - what service, what metric, what threshold
- Check the dashboard - is this real or a fluke? Look at trends
- Check recent changes - deployments in the last 2 hours?
- Determine severity - SEV1 (business down), SEV2 (degraded), SEV3 (minor)
- Create incident channel if SEV1 or SEV2
Quick Fixes by Symptom
Service Down (502/503)
- Check if process is running
- Check recent deployments
- Roll back if recent deploy caused it
- Check resource limits
- Check logs
- Restart if needed
High Error Rate (5xx)
- Check error logs for patterns
- Check database connections
- Check external API status pages
- Check for config changes
- Roll back if bad deploy
High Latency
- Check CPU/memory usage
- Check database slow queries
- Check network/DNS
- Check queue depth
- Scale up if needed
Database Issues
- Check connection pool
- Check slow queries
- Check disk space
- Check replication lag
- Kill long-running queries if blocking
Disk Space Full
- Check disk usage
- Find large files
- Check log files
- Rotate/compress old logs
- Clean up temp files
Escalation Paths
| Severity | Who | Response Time |
|---|---|---|
| SEV1 | On-call + Lead + CTO | 5 min |
| SEV2 | On-call + Lead | 15 min |
| SEV3 | On-call only | Next day |
Communication Template
[SEV2] [Service] experiencing elevated error rates
Started: [timestamp]
Impact: [who/what is affected]
Status: Investigating
Next update: [15 min from now]
Update every 15 minutes. Even if the update is "still investigating."
Post-Incident
- Schedule post-mortem within 48 hours
- Write timeline of events
- Identify root cause and contributing factors
- Create action items with owners and deadlines
- Update this runbook with learnings
Pro Tip
Keep this runbook next to your pager. When you're half-awake at 3am, you don't want to search for the runbook. Print it. Pin it. Use it.
Get the complete runbook with 20+ scenarios: Ops Starter Kit
Free resource: The First 30 Minutes - free incident quick-start checklist.
Use code LAUNCH50 for 50% off anything in the store.
Top comments (0)