data incident response is the discipline every DE team eventually has to build — the runbook, the on-call rotation, the sev-level playbook, the postmortem template. Every DE team eventually has an incident; knowing the difference between "the pipeline crashed" (Sev-2) and "the CFO saw a wrong number" (Sev-1), and having runbooks and MTTR discipline is what separates a mature data org from a chaotic one.
The tour walks (1) severity classification for data incidents, (2) runbook anatomy, (3) MTTR + on-call rotation, and (4) blameless postmortem template + action items + library.
1. Why incident response matters for DE in 2026
Where it shows up.
- Wrong numbers reaching execs.
- SLA misses on customer-facing analytics.
- Pipeline crash blocking hourly refresh.
- CDC lag causing stale data.
- Cost spike (data spend 3× overnight).
- Silent data quality regression.
Data incidents differ from SWE incidents.
- Impact is often silent — bad numbers, not 500 errors.
- Detection lag can be hours or days.
- Rollback is complicated (data can't be un-written).
- Root cause often external (source schema change, feed dropped).
2. Sev classification for data
Sev1 · Sev2 · Sev3 · Sev4
Slot 1 — Sev-1 examples.
- CFO sees wrong revenue number in exec dashboard.
- Customer-facing dashboard shows corrupted values.
- Data leak — PII exposed.
- Compliance report incorrect.
Response: within 15 min. Page primary + secondary + manager.
Slot 2 — Sev-2 examples.
- Hourly refresh SLA missed by > 30 min.
- Pipeline stuck; hasn't run in 2 hours.
- DQ test failing on prod.
- Internal dashboard degraded.
Response: within 1 hour. Page primary.
Slot 3 — Sev-3 examples.
- Small % of rows have wrong value; overall unaffected.
- Latency degraded but within SLA.
- Non-critical dashboard slow.
Response: within 1 day. Ticket + assign.
Slot 4 — Sev-4 examples.
- Cosmetic issue.
- Documentation drift.
- Log noise.
Response: within 1 week. Backlog.
Slot 5 — sev matrix.
| Aspect | Sev1 | Sev2 | Sev3 | Sev4 |
|---|---|---|---|---|
| Response | 15 min | 1 hr | 1 day | 1 week |
| Escalation | Manager | On-call | Team | Backlog |
| Postmortem | Required | Required | Optional | No |
| War room | Yes | Optional | No | No |
3. Runbook anatomy
Detection · triage · mitigation · root cause
Slot 1 — detection.
- Alerts fired: which one?
- Symptoms observed: what's wrong?
- Time of first detection.
Slot 2 — triage.
- Scope: how many rows / users / dashboards?
- Impact: financial, compliance, user-facing?
- Sev level assignment.
Slot 3 — mitigation.
- Stop the bleeding.
- Roll back deploy.
- Kill switch.
- Disable pipeline until fix.
Slot 4 — root cause.
- 5 whys.
- Data lineage tracing.
- Log correlation.
Slot 5 — communication.
- Incident channel in Slack.
- Status page for external.
- Manager + stakeholder updates every 30 min for Sev1.
Slot 6 — runbook template.
# Runbook: [Pipeline Name]
## Alert
- When: what triggers this runbook
- Signal: what to look for
## Triage
- Check X
- Check Y
## Mitigation
- If A: do B
- If C: do D
## Root cause investigation
- Where to look
- Common causes
## Rollback
- Command:
- Verification:
## Escalate to
- Team X for source system
- Team Y for infrastructure
Every hot pipeline has a runbook.
4. MTTR + on-call rotation
Time-to-detect · time-to-mitigate · PagerDuty pattern
Slot 1 — MTTR decomposition.
- TTD — Time to detect. From incident start to alert fires.
- TTA — Time to acknowledge. From alert to responder ack.
- TTM — Time to mitigate. From ack to service restored.
- TTR — Time to resolve (root cause fixed).
Slot 2 — improving TTD.
- Better monitors on data quality.
- Anomaly detection on metrics.
- Direct customer feedback loop.
Slot 3 — improving TTM.
- Runbooks reduce triage time.
- Kill switch for instant off.
- Feature flags for rollback without deploy.
- Practice drills quarterly.
Slot 4 — on-call rotation.
- Weekly rotation, primary + secondary.
- Follow-the-sun for global teams.
- Compensation (PTO or pay) for on-call.
- Handoff document at rotation start.
Slot 5 — PagerDuty patterns.
- Escalation: primary (5 min) → secondary (10 min) → manager (15 min).
- Auto-resolve after N min if no ack.
- Post-incident survey to responder.
Slot 6 — on-call culture.
- No blame for pages during shift.
- Fix the flappy alert, not silence it.
- Rotate all engineers (including managers).
- Retrospective on frequent alerts.
5. Postmortem + patterns
Blameless template · action items · library
Slot 1 — blameless template.
# Incident #123 — [Title]
## Summary
1-2 sentence description.
## Impact
- Users affected: N
- Duration: X min
- Financial impact: $Y
## Timeline
- 09:15 — Alert fired
- 09:18 — On-call acknowledged
- 09:22 — Root cause identified
- 09:35 — Mitigation deployed
- 09:40 — Service restored
## Root cause
[What actually broke]
## Detection
[How was it noticed]
## Mitigation
[What restored service]
## Contributing factors
[What made this bad]
## Action items
- [ ] Fix X (owner: @alice, due: 2026-08-01)
- [ ] Add monitor for Y (owner: @bob, due: 2026-07-20)
- [ ] Update runbook (owner: @carol, due: 2026-07-15)
## What went well
- What went well.
## What could go better
- Not blame; process improvement.
Slot 2 — blameless culture.
- Focus on process, not person.
- "How did the system allow this?" not "Who made the mistake?".
- Encourage disclosure of mistakes.
Slot 3 — action items.
- Each has owner + due date.
- Tracked in issue tracker.
- Reviewed at next team meeting.
- Not lost in the postmortem doc.
Slot 4 — postmortem library.
- All postmortems searchable.
- Tagged by root cause category.
- Reviewed quarterly for patterns.
- New engineers read top 10 during onboarding.
Slot 5 — repeat offenders.
- Same root cause 3× → engineering initiative.
- Same runbook gap 2× → runbook rewrite.
Slot 6 — external-facing writeup.
- For customer-visible incidents.
- Executive summary only.
- Reviewed by legal + PR.
Cheat sheet
- Sev1 = wrong CFO number.
- Sev2 = SLA miss.
- Sev3 = degraded.
- Sev4 = cosmetic.
- Runbook: detect, triage, mitigate, root cause.
- MTTR = TTD + TTA + TTM.
- Kill switch for instant off.
- On-call primary + secondary + manager.
- Weekly rotation.
- Blameless postmortem.
- Action items with owner + date.
- Postmortem library searchable.
- Practice drills quarterly.
- Compensate on-call.
- Runbook updated after every incident.
FAQ
Sev1 or Sev2 for a wrong number that hasn't been seen?
Depends. If it's in a dashboard the CFO checks daily and could be seen soon — Sev1. If it's in a rarely-viewed report and you can fix within 24 hours — Sev2. When in doubt, upgrade.
How do I make on-call sustainable?
Limit rotation to 1 week; primary + secondary + manager; compensation (comp time or pay); fix chronically flappy alerts so pages are meaningful. Retrospective on high-page shifts.
How much time on the postmortem?
Sev1: full retro within 1 week, 1-2 hours meeting + write-up. Sev2: writeup within 1 week, meeting optional. Sev3: writeup optional. Sev4: no postmortem.
How do I improve MTTR?
Better monitors reduce TTD. Runbooks reduce triage time. Kill switches enable instant mitigation. Feature flags enable rollback without deploy. Practice drills expose gaps.
What's a good page rate?
< 2 pages / week / responder for Sev1+2. > 5/week is unsustainable. Track and fix noisiest alerts.
How do I handle a public data incident?
Follow the sev process internally. External comm through PR/legal. Never blame customers or vendors publicly. Publish external postmortem after internal review.
Practice on PipeCode
Pipecode.ai is Leetcode for Data Engineering.
Top comments (0)