DEV Community

Hive80-lab
Hive80-lab

Posted on

The 2 a.m. problem: why two-person IT teams fail incident response (and the 30-minute fix)

#devops #incidentresponse #sre #msp

Two-person IT teams don't fail incident response because they lack tools.
They fail because the runbook lives in someone's head — and that someone is asleep,
on holiday, or has quit.

The fix is not another monitoring tool. It's 30 minutes of boring documentation.

What actually breaks at 2 a.m.

  1. Nobody knows who is on call. A shared inbox is not an escalation path.
  2. Nobody knows what "down" means. Is the client's site down, or just the office Wi-Fi?
  3. The restore has never been tested. A backup you haven't restored from is a theory, not a backup.
  4. Passwords live in one person's head. That person is driving.

The 30-minute fix (do it this week)

Minutes 0–10: Write the one-page escalation ladder.
Who gets called first, second, third. Phone numbers. What counts as P1 vs P2.
One page. Print it. Tape it inside the server room door.

Minutes 10–20: Test one restore.
Pick the smallest client backup you have. Restore one file. If it works, write down
how long it took. If it fails, you just found your next P1 before it found you.

Minutes 20–30: Write the first-response checklist.
Six lines: confirm the outage is real, check the status page, check the last change,
roll back or escalate, post the client update, log the timeline.

That's it. No new tools. No new budget.

Why this pays

The average small MSP loses clients over communication during outages, not over
the outage itself. A one-page ladder + a tested restore turns a 2 a.m. disaster into
a boring story you tell at the next team meeting.

If you want the done-for-you version — printable incident field cards, the escalation
ladder template, and the restore-test log — I packaged them:

And if you'd rather audit your own ops first, the whole library of 167 ops notes is free:
https://hive80-lab.github.io/ops-notes/

Built by Hive80 Lab — we ship practical ops, automation and security resources for small teams.

Top comments (0)