A plan you never rehearsed is a wish: the small-team incident drill calendar

#devops #sre #security #smallbusiness

Every small team has an incident response plan. Almost none have rehearsed it — and the difference shows up at 2AM. The team that drills fumbles for fifteen minutes while they remember where things live. The team that doesn't fumbles for two hours while they discover the plan was fiction.

The fix is not more plan. It is a schedule. A drill that is not on the calendar does not happen, because drills are the first casualty of every busy week. Here is the calendar itself: four quarterly drills, a handful of fifteen-minute monthly ones, the roles, the pass criteria, and the loop that turns every drill into fixes instead of feelings.

Full checklist version (free, one page): Incident Response Drill Schedule Template for Small Teams

Why the calendar is the drill

Drills decay on their own schedule — people rotate off, tools get renamed, the runbook that worked in March is wrong by October. A quarterly cadence means nothing drifts more than six months before someone rediscovers it.
The budget is small on purpose: one big drill per quarter (~90 minutes) plus one fifteen-minute mini-drill per month. That is roughly four hours of practice a year. Small, frequent, and specific beats large, annual, and theatrical.
The schedule is public, with names and dates, next to the on-call rotation. A private calendar is a calendar that quietly dies.

The four quarterly drills

Q1 — the tabletop: one scenario, ninety minutes, no laptops. Pick the scenario the business actually fears (for most small teams: ransomware or a payment-path outage) and walk it end to end. The output is a list of stall points — places the room went quiet — not a grade.
Q2 — the restore: prove the backup can pay out. Restore one real system from backup onto a clean machine, timed, with the key ceremony included if backups are encrypted. A backup that has never been restored is a rumor, not a backup.
Q3 — the failover: rehearse the recovery path, not the disaster. A deployment rollback on staging, a DNS cutover walkthrough, or a vendor-down tabletop. The point is muscle memory for the path — at 2AM nobody reads a page they have never seen.
Q4 — break-glass and keys, hands on. Open the sealed emergency kit (or walk the steps that would open it), rotate one real credential, and confirm the alerting fires on emergency-account use. A break-glass kit that cannot survive being tested is a decoration.

The monthly fifteen-minute mini-drills

The severity recitation: the on-call person classifies two hypothetical events aloud against the severity matrix and starts a timeline entry for the fake incident. Two people, ten minutes, zero infrastructure.
The lost-device walk: one person narrates the MFA-device-lost runbook from memory while the other checks against the page. Nobody should meet that runbook for the first time holding a found phone in a parking lot.
The handoff under pressure: write a complete handover for a fictional open incident in ten minutes; the receiver restates it. The real 2AM version should be a copy-paste job.
The cleanup sandbox: fill a scratch disk once a quarter and walk the disk-full runbook on it. Disk-full is the most common small-team incident and the most preventable.

Drill-day roles (four people, three of whom can be the same person)

Facilitator: brings the scenario, keeps the clock, and refuses to answer questions the runbook should answer. The moment they rescue the room, the room stops learning.
Incident commander + scribe: someone drives (classifies, escalates, delegates) and someone writes the timeline as it would be written for real. Rotating these roles matters — your only IC should not be the person who is already on-call every quarter.
Observer with the checklist: holds the written pass criteria and marks every stall point with its reason: missing doc, missing access, missing tool, or missing skill. Grading against the runbooks, not the people, keeps drills blameless — and makes the findings actually get fixed.

Pass criteria and the fix loop

Write the pass criteria before the drill, not during it. One sentence: "restore of the billing database on a clean machine, under one hour, from sealed keys." A drill without pre-written criteria is a conversation with a timer.
Split findings into doc fixes and skill gaps. Every stall point is either the documentation failing the team (fix the doc this week) or a person missing a skill (schedule pairing, not a lecture). The doc-fix list is usually longer, and that is good news — docs are cheap and the fix pays out on every future drill.
Re-test the fix, not the whole drill. If the restore failed on the key ceremony, the next mini-drill is one key ceremony. Small re-tests keep the loop tight.
Let drills write to the plan. A plan that never changes after four drills a year is not stable — it is unread.

The one-page template

One row per drill: quarter · drill name · owner (a person, not a team) · scenario · pass criteria sentence · date · result. Plus the fix log: finding · type (doc/access/tool/skill) · owner · fix date · re-test date — same columns as a postmortem action list, so the two artifacts stay compatible.

The cadence rule that makes it real: never two consecutive quarters without a completed drill. A missed drill is rescheduled within thirty days or the miss goes on the same fix log.

Small-team honesty note: if the team is one founder and a part-time admin, this is one shared doc with four rows and a phone timer — and it still matters, because the founder is usually both the plan and the single point of failure inside it.

If you want the drill-day starter kit: the Ops Starter Kit ($14) bundles the incident-response checklists, and The First 30 Minutes is free.

DEV Community