Most teams practice their incident response plan exactly once: the day after a production outage. By then it is too late.
I run a 10-minute incident response drill every Monday morning. It is the single highest-ROI habit I have built in the last two years. Here is exactly how it works.
The Monday Drill (10 Minutes)
Minute 1-2: Pick a Scenario
Rotate through these scenarios:
- Database connection pool exhausted
- CDN goes down (static assets 404)
- API rate limit hit by a runaway script
- SSL certificate expires
- Disk full on the log server
- Third-party API returns 500s for 20% of requests
Write the scenario on a shared doc. No one looks at old runbooks — this keeps it fresh.
Minute 3-5: First Responder Walkthrough
The designated first responder (we rotate weekly) walks through what they would do:
- Detect: How would I know? (alert? user report? dashboard?)
- Triage: Is this real? What is the blast radius?
- Communicate: Who do I tell? (Slack channel? Status page?)
- Mitigate: What is the fastest way to stop the bleeding?
- Resolve: What is the actual fix?
Minute 6-8: Gap Check
The rest of the team asks two questions:
- What would slow you down? (missing access? unclear runbook? no dashboard?)
- What would you do if that tool was also down?
This is where the gold is. Every week someone says "I would check the dashboard" and someone else says "the dashboard is hosted on the same server that is down." That gap gets fixed before it becomes a 3 AM outage.
Minute 9-10: Action Items
Write down 1-2 improvements. Small ones:
- Add a fallback alert channel (Slack + email)
- Put the runbook link in the alert notification
- Add disk usage to the monitoring dashboard
Why It Works
| Without the drill | With the drill |
|---|---|
| First outage: 45 min to detect | First outage: 3 min to detect |
| Runbook exists but nobody read it | Runbook was practiced this week |
| "Who has access to X?" during the outage | Access was verified on Monday |
| Post-mortem reveals 5 gaps | Gaps were found before the outage |
The Template
I turned this drill into a reusable template with:
- 12 scenario cards (print and shuffle)
- A first-responder checklist
- A gap-tracking spreadsheet
- A 4-week rotation schedule
Get the Incident Response Template →
The best time to practice your incident response is when nothing is on fire. The second best time is right now.
More ops templates and checklists at hive80lab.gumroad.com.
Top comments (0)