DEV Community

Hive80-lab
Hive80-lab

Posted on

The 10-Minute Incident Response Checklist Every Solo Dev Needs

#devops #sre #python #productivity

When production breaks and you're the only one on call, panic wastes more time than the outage itself. After handling 100+ incidents solo, I distilled response into a 10-minute checklist that works for any stack.

Minute 0-2: Confirm and Contain

Before touching anything, answer three questions:

  1. Is it actually down, or just for me? Check from an external vantage (curl from a different network, status page, phone on LTE).
  2. What changed in the last hour? Deploys, config pushes, DNS changes, cert renewals. 80% of incidents trace to a recent change.
  3. Can I stop the bleeding without finding the root cause? Rollback beats debugging when users are locked out. 
# Quick change audit on a typical server
sudo find /etc /opt /srv -mmin -60 -type f 2>/dev/null | head -20
git -C /var/www/app log --since="1 hour ago" --oneline
Enter fullscreen mode Exit fullscreen mode

Minute 2-5: Communicate Before You Fix

A status page update or a single tweet takes 90 seconds and halves the angry messages. Template:

"We're aware of an issue affecting [service]. Working on a fix. Next update in 15 minutes."

Set a timer. Update even if there's nothing new — silence reads as abandonment.

Minute 5-8: Stabilize

Work the checklist in this order (fastest, least risky first):

  • Restart the service — systemctl restart <svc> (yes, really; it fixes ~40% of incidents)
  • Roll back the last deploy — if deploys are atomic, this is one command
  • Scale up resources — OOM and disk-full are the classic silent killers: df -h, free -m
  • Clear the queue/cache — stuck jobs and stale caches cause weird partial outages 
df -h            # disk full? log rotation is usually the culprit
free -m          # OOM kills show in dmesg
dmesg | tail -20
journalctl -u <service> --since "30 min ago" --no-pager | tail -30
Enter fullscreen mode Exit fullscreen mode

Minute 8-10: Decide Next Steps

Two outcomes, pick one deliberately:

  • Resolved → schedule the postmortem, write down what changed.
  • Not resolved → escalate or invoke your runbook. If you don't have a runbook, write one after this incident, while memory is fresh.

The Postmortem (30 Minutes, Same Day)

Three questions on one page:

  1. What was the user impact (duration, scope)?
  2. What was the root cause (the change, not the person)?
  3. What one automated check would have caught this?

One automated check per incident. After a year you have 20+ checks and your incident rate drops off a cliff.

Don't Build This Alone

If you're solo and want a head start, I maintain a set of ready-made incident response kits — pre-written runbooks, alert scripts, and checklists you can adapt in minutes:

They're cheap, instant, and built from real on-call experience. Even if you never buy anything: steal the checklist above. It works.

Top comments (0)