Incident Management & Collaboration for Data Quality

• Detecting the First Signal: Build monitors that surface actionable issues
• When Data Breaks, Who Does What: Roles, ownership, and communication paths
• How Runbooks, Automation, and Escalation Rules Keep MTTR Low
• Postmortems and Root Cause Analysis That Change Behavior
• Immediate Protocol: Practical triage checklist and runbook template

Data incidents are inevitable; silent ones are the most dangerous because they erode trust before anyone notices. You need a repeatable, auditable incident lifecycle — detection, triage, containment, remediation, and learning — that treats data like a first-class product and stitches monitoring, ownership, and post‑incident learning together.

The immediate symptoms you see are familiar: dashboards show bad numbers, reports get retracted, downstream ML models degrade, and business stakeholders tell you first — not your monitoring. Recent industry surveys show data downtime and mean time to resolution rising sharply, with business teams often discovering the issue before the data team does. That pattern — late detection, long resolution, and business-first discovery — is the precise friction the playbook below eliminates.

Detecting the First Signal: Build monitors that surface actionable issues

Your monitors must detect meaningful deviation, not spam on noise. For data systems that means a mix of technical and semantic checks placed at the right boundaries:

• Source / ingestion checks: arrival timestamps, row counts, file manifests, ingest latency.
• Schema & contract checks: column additions/removals, type changes, unexpected NULLs.
• Distributional checks: sudden shifts in cardinality, histograms, or categorical distributions.
• Business rule checks: conversion rates, revenue totals, enrollment counts — the metrics your consumers trust.
• Downstream invariants: referential integrity, uniqueness, freshness of aggregated datasets.

Implement checks as close to the change surface as possible — in the ingestion layer, in transformation runs (dbt tests), and as validation Checkpoints in a quality layer like Great Expectations. Checkpoints let you run suites of expectation_suite rules and chain Actions (post to Slack, hit a webhook, write to a quarantine table) so a failing expectation becomes an operational signal rather than an abstract test failure. dbt tests are the correct place for transformation assertions and integrate naturally into CI/CD so tests run pre-merge and in production runs.

Important: Prioritize signal-to-action. A successful alert includes the failing assertion, the minimal query to reproduce, relevant run metadata (commit, DAG run id), and an owner. Alerts that lack context become noise.

Example: a minimal Great Expectations Checkpoint that runs a suite and posts to Slack / webhook (trimmed for clarity):

name: users_daily_checkpoint
validations:
  - batch_request:
      datasource_name: prod_warehouse
      data_asset_name: users_daily
    expectation_suite_name: users_daily_suite
action_list:
  - name: post_to_slack
    action:
      class_name: SlackNotificationAction
      slack_channel: "#data-alerts"
  - name: pagerduty_webhook
    action:
      class_name: NotificationAction
      notifications:
        - webhook: "https://events.pagerduty.com/generic/2010-04-15/create_event.json"

Practical monitoring guidelines:

• Start with high-value checks (freshness, row counts, primary keys) that protect revenue or critical decisions.
• Use statistical baselines for distributional alerts, avoid hard thresholds for noisy metrics.
• Route alerts based on severity and context — small freshness delay ≠ critical revenue loss.

Citations: Great Expectations Checkpoints and Actions. dbt testing and placement of tests. Industry detection/resolution trends.

When Data Breaks, Who Does What: Roles, ownership, and communication paths

Clarity of ownership is the single most levered control you can add to incident response. Map dataset → pipeline → consumer ownership and make the routing deterministic.

Role	Primary responsibilities	Escalation / communication path
Data Owner / Domain Lead	Business intent, SLOs for datasets, acceptance criteria	PagerDuty → Domain on-call → Incident Commander
Data Steward	Data cataloging, metadata, consumer liaison	Slack channel & handbook
On‑call Data Engineer (DataRE / DRE)	First responder for pipeline and transformation failures	PagerDuty (primary)
Incident Commander (IC)	Coordinate cross-team response, assign leads, author status updates	IC channel (Slack) → Exec updates
Communications Lead	External/internal status, template ownership	Statuspage, support comms
Business Stakeholder / Consumer	Impact details, business context	Added to status updates; not on-call
Security / Legal	Involved when PII/exfiltration/regulatory risk suspected	Immediate escalation by IC

Operational rules that work in practice:

• Always page a named on‑call (not an alias) for dataset-level alerts. Use on-call schedules in PagerDuty to avoid ambiguity.
• For multi-team incidents, the IC pattern — borrowed from ICS and adapted for software — keeps delegation clear: IC focuses on orchestration while subject-matter leads handle domain fixes. Google SRE practices and Atlassian document this operating model.
• Register who to page in each dataset’s metadata: incident_owner_contact, runbook_link, sla_freshness_minutes.

Severity matrix (example):

Severity	Symptom	Who gets paged	Time-to-escalate
Sev 1 (Critical)	Core business metric wrong, exec impact	IC + Domain Lead + On-call	Immediate
Sev 2 (High)	Key pipelines failing, large subsets impacted	On-call + Domain Lead	15 minutes
Sev 3 (Medium)	Single dashboard wrong, scheduled job failing	On-call (ticket)	60 minutes

Citations: Incident Commander and ICS adaptation concepts. PagerDuty on-call tooling and routing.

How Runbooks, Automation, and Escalation Rules Keep MTTR Low

Runbooks are executable knowledge: a short, versioned document that lets a responder execute safe mitigation steps without hunting for context. Treat a runbook as code — versioned, reviewed, and invoked by automation or humans.

Essential runbook elements:

1. Symptom & detection query — exact check that failed and the diagnostic query (SELECT COUNT(*) ... WHERE partition_date = {{date}}).
2. Quick triage checklist (3–6 items) — e.g., check recent deploys, check upstream table arrival, check disk usage.
3. Safe mitigations — commands to re-run ingestion, steps to quarantine rows, backfill recipe with parameters, and rollback instructions.
4. Verification steps — precise queries and dashboards to prove recovery.
5. Communications templates — short status messages for support, internal stakeholders, and executives.
6. Escalation matrix — how long until the next escalation and to whom.

PagerDuty's Runbook Automation lets you transform manual runbook steps into secure, auditable automated tasks that responders can invoke from Slack or PagerDuty without shell access; that reduces human error and speeds resolution. Integrations with Slack let responders act in the channel, preserving context and creating a timeline for postmortems.

Example (minimal runbook template — YAML-like):

id: users_table_schema_drift_v1
symptom: "users_daily schema changed; new column 'x' present"
detection_query: "SELECT column_name FROM information_schema.columns WHERE table='users_daily';"
initial_checks:
  - check_ingestion: "SELECT COUNT(*) FROM raw.users WHERE ingestion_date = today"
  - check_recent_deploy: "git log -n 5 --pretty=oneline"
mitigations:
  - name: "quarantine_bad_partition"
    command: "INSERT INTO quarantine.users SELECT * FROM raw.users WHERE ingestion_date = today AND ...;"
  - name: "reingest_partition"
    command: "airflow dags trigger users_ingest --conf '{\"date\":\"{{date}}\"}'"
verification:
  - "SELECT COUNT(*) FROM curated.users_daily WHERE date = today;"
escalation:
  - after: 15m
    to: domain_lead
  - after: 60m
    to: incident_commander
communication_templates:
  - internal: "[SEV2] users_daily schema drift — investigating. Incident ID: {{incident_id}}"

Automation guardrails:

• All runbook automation must run through an auditable bridge (PagerDuty Runbook Automation) with RBAC and logging rather than giving wide terminal access.
• Use idempotent operations where possible (e.g., backfills that are safe to re-run).
• Log every automated action into the incident timeline so postmortem reconstruction is straightforward.

Citations: PagerDuty Runbook Automation and Slack integration.

Postmortems and Root Cause Analysis That Change Behavior

A postmortem's currency is clearly tied action items, not prose. The goal is to lock in changes that remove the entire causal chain that allowed the incident to occur.

A high‑value postmortem includes:

• Short incident summary with impact and duration.
• Precise timeline: timestamps of detection, paging, mitigation steps, and recovery. Timelines are the scaffolding for finding where the system failed.
• Proximate vs root cause analysis — separate the immediate trigger from deeper systemic weaknesses. Atlassian explicitly distinguishes proximate causes from optimal root causes. Use a Five Whys or causal tree to locate the leverage point.
• Action items that are specific, bounded, measurable, and owned (e.g., “Add source schema CI and test by 2026-02-15 — owner: data‑platform team”).
• Verification plan for each action (how you’ll validate the fix and when).
• Publication & follow-up: a postmortem owner drives approvals and tracks completion in your backlog. Atlassian prescribes approvals and SLOs for action resolution to ensure follow-through.

Blameless culture: frame all findings in systems and process terms; avoid naming individuals and instead reference roles and automation gaps. Blameless postmortems produce better RCAs and higher psychological safety. Google SRE’s incident playbook and case studies show that early incident declaration and a tight coordination model materially shorten incidents and simplify RCAs.

Copy‑paste postmortem skeleton (Markdown):

# Postmortem: [Short Title]
**Incident ID:** inc-2025-1234
**Date:** 2025-11-12
**Severity:** Sev 1
**Summary:** One-sentence summary of what failed and the impact.
## Timeline
- 09:12 UTC — Alert: users_daily rowcount fell 90%. (source: GE checkpoint)
- 09:18 UTC — On-call acknowledged; IC declared Sev1.
...
## Root cause analysis
- Proximate cause:
- Root cause:
## Action items
- [ ] Add source schema CI (owner: data-platform) — due: 2026-02-15
## Verification
- Query / dashboard URLs to confirm

Citations: Atlassian postmortem practices and templates. Google SRE incident response guidance.

Immediate Protocol: Practical triage checklist and runbook template

Here is a tightly scoped, time‑boxed protocol you can paste into an internal playbook and use in the first 48 hours of any data incident.

Quick triage (0–15 minutes)

1. Record incident_id and create an incident channel (Slack + PagerDuty incident). Capture the failing check, dataset, and DAG/commit id.
2. Run three reproduction queries: ingest counts, top 5 error messages, last successful run id.
3. If impact is customer-facing or revenue‑affecting, declare Sev 1 and page IC + domain lead. (Severity rules above.)

Containment & mitigation (15–60 minutes)

• Run safe mitigations from the runbook: quarantine, reingest a single partition, or revert the latest transformation deployment.
• Make a rollback decision if code change is root cause; use feature flags or revert commits via CI if safe.
• Communicate status to support and product teams using the template in the runbook.

Stabilize & restore (1–8 hours)

• Execute verified backfill if necessary. Mark datasets as quarantined in the catalog so consumers don’t unknowingly use partial data.
• Verify downstream dashboards and ML features; populate a "safe" read-only dataset for immediate needs.
• Track the incident resolution metrics: time-to-detect, time-to-ack, time-to-resolve.

Post‑incident (within 48–72 hours)

• Run timeline workshop; draft postmortem skeleton and assign owner.
• Convert priority actions to backlog items with SLOs, due dates, and owners. Use automation to remind approvers until closed.

Escalation quick table (copy into PagerDuty policy):

After	Action
0 min	Page on-call (primary)
15 min	Escalate to domain lead
60 min	IC engaged, exec‑level status if Sev1
4 hours	All-hands or incident war room if unresolved

Runbook verification checklist (for each action item):

• Does the runbook include the exact diagnostic query? yes/no
• Is the mitigation script idempotent? yes/no
• Is the verification query defined? yes/no
• Is a rollback plan documented? yes/no

Takeaway: The fastest wins come from small changes you can reason about fast: better ownership metadata, one reliable monitor, and a short, executable runbook for that monitor.

Citations: NIST lifecycle concepts for incident phases and recommended timelines. PagerDuty automation & runbook practices. Atlassian postmortem guidance for follow-up and approvals.

Treat incident management as a product — versioned runbooks, measurable SLOs, and regular drills — and you convert incidents from interruptions into the engine of continuous improvement. Data incident response is not a checklist you run once; it’s the operating rhythm that keeps your analytics trusted and your business confident.

Sources:
Data Downtime Nearly Doubled Year Over Year, Monte Carlo (Business Wire press release, May 2, 2023) - Survey findings on monthly incident frequency, detection & resolution times, and business-first issue discovery.

SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management (NIST, April 2025) - Framework for incident lifecycle phases and organizational incident response practices.

PagerDuty Runbook Automation (PagerDuty product documentation) - Capabilities for authoring, managing, and invoking automated runbook tasks and guidelines for auditable automation.

Postmortems: Enhance Incident Management Processes (Atlassian Incident Management Handbook) - Blameless postmortem guidance, templates, and approaches to root cause vs proximate cause and action tracking.

Incident Response (Google SRE Workbook / Incident Response chapter) - Operational patterns for incident command, timelines, and case studies illustrating effective coordination.

Checkpoints & Validation (Great Expectations documentation) - How to bundle validations with actions, and operate Checkpoints that produce actionable validation results.

Data quality testing: What it is, where and why you should have it (dbt Labs blog) - Principles for placing tests in the pipeline and using dbt tests for transformation-level assertions.

Slack Integration Guide (PagerDuty Support) - How to connect PagerDuty and Slack to support ChatOps workflows, in-channel actions, and incident channel automation.