A database outage is the one incident where the fix can destroy the data. An app restart that goes wrong costs you five minutes. A database recovery that goes wrong costs you the day's orders — and the trust of everyone whose order was "processed" twice or not at all. The full runbook with the worked example is on our ops notes; here's the shape of it.
1. The two-minute triage before you touch anything
Half of "database down" is something else wearing its clothes. Four checks, two minutes each, and they pick your whole response:
| Symptom | Check | Verdict |
|---|---|---|
connection refused / too many connections
|
raw client from a second host | refused everywhere = the host; connects = pool or network config |
| every page 500s, even marketing pages | load a static asset | static works = data layer; static broken = wrong runbook |
| site slow but alive | lock waits, long queries | hung, not down — restarting is wrong here |
df -h shows the disk full |
— | it was never a mystery: fix the disk first |
Down, hung, slow, and disk-full get four different responses. The instinct — "restart the database" — is only right for one of them.
2. Stop the writes before you diagnose further
The ordering rule: a database you can't read from is an outage; data you corrupt while recovering is a catastrophe. Before any restart:
- flip the stop-write switch (maintenance flag, pause checkout — whatever your stack has),
- capture state: one dump of whatever is still readable, plus the logs, copied off the host. That dump is your post-mortem evidence and your insurance if the next step makes things worse,
- queue what customers send during the outage, with an honest acknowledgment: "your order is saved and in line — we'll email you when it's processed."
The payment exception matters: never let a queued payment auto-execute on recovery without a human eye. Hold the charge, email the customer, process after verification. A refund stampede is a worse incident than the outage.
3. Degraded mode: the subset that still works
Customers forgive "part of the site is down" far more than "the site is lying." The pre-decided subset: catalog and content served from cache with zero database on the path; checkout paused with a real message and a link to the status page; queue-and-ack for anything a customer must submit.
The one place degraded mode backfires: if your fallback writes to a second system of record, you've forked the data and bought a reconciliation project. Queue the writes, serve the reads, never split the truth.
4. The restore you rehearsed
Nightly dumps alone make you choose between "restore to last night, lose today" and "lose nothing, stay down." Point-in-time recovery (PITR) breaks the tradeoff: base backup plus write-ahead log replayed to minutes before the disaster. The 30-minute path:
- spin a fresh instance,
- restore the base backup,
- replay the log to the cut-off,
- verify with a real query (
SELECT max(created_at) FROM orders— notSELECT 1), - flip the connection string, lift the stop-write flag.
"How much data can we lose?" is a daylight decision. Write the recovery point objective down and buy the backup tier that matches — discovering your RPO is "the whole day" during the outage is how teams lose the whole day. And rehearse: an untested restore takes three hours and three mistakes. A quarterly drill cut ours from three hours to 27 minutes.
5. Back isn't healthy
Before declaring victory: connection count and pool config back to normal, replication lag at zero before you shift reads back, slow query log left on for 24 hours, disk headroom confirmed with an alert set, and — the one everyone forgets — the queue reconciled to the last order and the last acknowledgment.
Then close it like an incident, not an anecdote: post-mortem for the review, action tracker for the fixes. The pool config change and the disk alert become tracked rows, not meeting memories.
The full worked example (a 19-minute pool-exhaustion incident degraded to read-only with zero lost orders) is in the runbook. If you want the first-30-minutes chaos of any incident turned into a fillable plan, the Ops Starter Kit ($14) is the cheap version of calm, and Vol. 2 ($27) adds the DR plan and evidence log. Free First 30 Minutes checklist too.
Top comments (0)