Chaos Engineering for Data Pipelines: Fault Injection with LitmusChaos & Gremlin

chaos engineering data is the discipline of deliberately injecting failures — pod kills, network partitions, broker crashes, disk fills — into a running data pipeline to prove the pipeline stays inside its freshness, correctness, and lag budgets when the underlying infrastructure misbehaves. It is not a stunt, it is not load testing, and it is not "let's yank a cable and see what happens." It is a hypothesis-driven engineering practice with a written steady state, a stated hypothesis, a blast-radius-controlled experiment, and a verification step that either confirms the pipeline's resilience or produces a concrete backlog item to fix it — and in 2026 it has moved from "SRE toolbox for stateless microservices" to a table-stakes practice for any data platform team that owns a real freshness SLO.

This guide is the senior-data-platform-engineer walkthrough you wished existed the first time you had to answer "how do we prove our Kafka + Airflow + Spark stack survives a broker crash without our warehouse silently losing an hour of events?" It walks through why chaos engineering pipelines is qualitatively different from chaos on stateless services (state, ordering, exactly-once contracts change everything), how litmuschaos on Kubernetes uses a ChaosEngine custom resource + probe guardrails to run pod-delete / pod-network-loss / disk-fill experiments with auto-abort, how gremlin layers scenarios + a blast-radius dial + halt-on-breach health checks on top of raw attack primitives, and how to compose data fault injection — kafka chaos (broker kill + ISR shrink), airflow chaos (scheduler kill mid-DAG), spark chaos (executor loss) — into a repeatable game-day loop. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the ETL practice library →, and sharpen the architecture axis with the design practice library →.

---

On this page

• Why chaos engineering is different for data pipelines
• LitmusChaos on Kubernetes — pod / network / node kill experiments
• Gremlin — scenarios, blast-radius controls, workflows
• Data-plane fault injection — Kafka broker kill, Airflow scheduler kill, Spark executor loss
• Hypothesis-driven chaos loop — steady state → hypothesis → blast → verify
• Cheat sheet — chaos engineering data recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why chaos engineering is different for data pipelines

Stateful streams, exactly-once contracts, and freshness SLOs — data chaos plays by different rules

The one-sentence invariant: chaos engineering data is the practice of deliberately injecting infrastructure failures into a running data pipeline while a codified data SLO (freshness, lag, correctness) acts as the steady state that decides whether the experiment passes or halts — and every design choice in the experiment (blast radius, probe cadence, rollback path) has to account for the fact that streams are ordered, tables are stateful, and downstream consumers hold exactly-once contracts that a naive chaos experiment can violate. A chaos experiment against a stateless HTTP microservice that returns "hello world" is easy: kill a pod, watch p99 latency, done. A chaos experiment against a Kafka broker that owns 300 partition leaders with min.insync.replicas=2 is a different physics problem — one wrong move and you have unshipped events sitting in producer buffers, ISR shrinks that trigger leader-election storms, or a downstream Spark job that fails mid-shuffle and re-reads gigabytes it already processed.

The four "must-answer" axes for a data chaos experiment.

• State. Where does the pipeline hold state, and what happens if that state is unavailable for N seconds? Kafka partitions have leader + ISR replicas, Airflow schedulers own the metadata DB, Spark drivers own the DAG lineage, Iceberg tables own atomic commits. Every one of these is a distinct state-loss failure mode with distinct recovery semantics.
• Ordering. Does the pipeline require in-order processing, and does the chaos experiment preserve or violate that? A network partition between two Kafka brokers can cause events to arrive at consumers out of order relative to source-time. A Spark executor loss forces stage retry, which re-emits shuffle blocks that downstream sinks may have already committed.
• Exactly-once. Does the chaos experiment risk breaking the exactly-once contract? Kafka + idempotent producers + transactional consumers hold EOS via a coordinator; if you kill the coordinator broker at exactly the wrong moment during a commit, some records commit twice on retry. The chaos experiment must either prove EOS holds or clearly report where it broke.
• Freshness SLO. Is the steady-state SLI a data metric (freshness p99, end-to-end lag, DAG success rate, tombstone count) rather than an infra metric (CPU, memory, pod-ready count)? Data chaos experiments verify against data SLIs; infra chaos experiments verify against infra SLIs. Confuse the two and you'll ship "the pod restarted but the pipeline is fine" when actually the pipeline is 40 minutes behind.

The 2026 reality — chaos has crossed from stateless SRE into data platform SRE.

• LitmusChaos is the CNCF sandbox → incubating open-source chaos framework that runs entirely on Kubernetes. It exposes chaos experiments as ChaosEngine custom resources with a library of ready-made faults (pod-delete, pod-network-loss, node-taint, disk-fill, pod-cpu-hog, pod-memory-hog) plus a probe system that lets you attach httpProbe, promProbe, cmdProbe, k8sProbe guardrails that auto-abort on breach. It is the default choice when your data plane runs on Kubernetes and you want everything GitOps-managed.
• Gremlin is the commercial SaaS chaos platform with a control plane in the cloud and lightweight agents (gremlind) on every target host, VM, container, or Kubernetes node. It exposes attack primitives (CPU, memory, disk, IO, latency, packet-loss, blackhole, DNS, shutdown) that stitch into "Scenarios" — multi-step attack workflows with explicit blast-radius sliders and halt-on-breach Health Checks tied to Datadog / New Relic / Prometheus. It is the choice when your data plane spans VMs, on-prem, cloud, and containers, and you want a single control plane with SOC-2-ready audit logs.
• Kafka / Airflow / Spark chaos is the "data-plane" layer where you inject faults specific to the stateful data systems — broker kills, ISR shrinks, scheduler restarts, executor loss, shuffle-fetch failures. These are typically composed on top of LitmusChaos or Gremlin: the tool provides the primitive; you provide the guardrails that catch data-SLO breaches specific to your stack.
• Game days are the human ceremony that wraps all of the above — a scheduled hour where the team runs a chaos experiment against a production-like environment (or a small % of prod), watches the SLI dashboards live, halts on breach, and writes a post-mortem doc that lands as a backlog item. This is the pattern that turns chaos from an SRE side project into a resilience-forcing function.

What interviewers listen for.

• Do you name steady state, hypothesis, experiment, verify without prompting? — senior signal; this is the Netflix / PoP chaos framework in one line.
• Do you say "my steady state is a data SLO, not CPU" — freshness_p99, e2e lag, DAG success rate — rather than infrastructure metrics? — required answer.
• Do you name blast radius as the first control and explain the dev → staging → 1% prod → 10% prod graduation? — senior signal.
• Do you distinguish LitmusChaos (K8s-native, open source, GitOps) vs Gremlin (SaaS, cross-platform, commercial) rather than treating them as interchangeable? — senior signal.
• Do you name a halt condition — the automatic abort that fires when the steady-state SLI breaches — rather than describing chaos as an open-loop attack? — required answer.

Worked example — the four-axis data chaos comparison table

Detailed explanation. The single most useful artifact for a chaos-engineering interview is a memorised 4×N comparison table that maps each candidate chaos experiment against the four axes: state loss, ordering impact, exactly-once impact, freshness SLO impact. Every senior chaos discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical stack that has Kafka → Spark Structured Streaming → Iceberg → dbt → Snowflake.

• Stack under test. Kafka (MSK, 6 brokers) → Spark Structured Streaming on EKS → Iceberg on S3 → dbt scheduled by Airflow → Snowflake.
• Steady-state SLIs. end-to-end freshness p99 < 5 min from producer to Iceberg; DAG success rate > 99%; Kafka consumer group lag < 10k messages.
• Candidate experiments. pod-delete on a broker, pod-network-loss on the Spark driver, disk-fill on the Airflow scheduler pod, blackhole to S3 for 60 s.
• Question at hand. For each experiment, which axes are at risk and what is the halt condition?

Question. Build the four-axis chaos comparison table and derive the halt condition for each experiment.

Input.

Experiment	State loss	Ordering risk	EOS risk	Freshness risk
pod-delete on Kafka broker	leader for ~30 partitions	in-partition order preserved; cross-partition unaffected	low if acks=all + min.insync.replicas=2	~10-30 s bump in lag
pod-network-loss on Spark driver	driver metadata unreachable	streaming batch stuck; no re-order	medium — checkpoint may skip micro-batch	freshness p99 grows unboundedly
disk-fill on Airflow scheduler	scheduler pod OOMs on next heartbeat	irrelevant (DAGs are not ordered)	irrelevant	DAG start delayed; downstream freshness slips
blackhole to S3 (60 s)	Iceberg writes fail; retries	retries preserve order per Iceberg commit	high — commit + manifest write are atomic	freshness stalls for ~60 s

Code.

# ChaosEngine — pod-delete on Kafka broker with promProbe halt condition
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-chaos
  namespace: kafka
spec:
  engineState: 'active'
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: CHAOS_INTERVAL
              value: '30'
            - name: FORCE
              value: 'false'
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 10
              retry: 2
              probePollingInterval: 10
              stopOnFailure: true
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator:
                type: int
                criteria: '<'
                value: '50000'

Step-by-step explanation.

1. The ChaosEngine custom resource declares a chaos experiment scoped to applabel: app=kafka-broker in the kafka namespace. LitmusChaos discovers the matching pods and runs the referenced experiment (pod-delete) against a randomly-chosen one.
2. TOTAL_CHAOS_DURATION=60 bounds the chaos window to 60 s; CHAOS_INTERVAL=30 means the pod is deleted every 30 s within that window (so the broker will be killed once, then again 30 s later). FORCE=false means Kubernetes performs a graceful shutdown (SIGTERM → configured grace period → SIGKILL) rather than an immediate SIGKILL, so the broker can drain in-flight requests.
3. The promProbe attaches a Prometheus query as a continuous guardrail. Every 10 s during the chaos window, LitmusChaos runs max(kafka_consumergroup_lag{group="warehouse-sink"}) against Prometheus. If the returned value is not < 50000, stopOnFailure=true causes LitmusChaos to auto-abort the experiment and mark it Failed.
4. The < 50000 threshold is the halt condition — the point at which we've decided the experiment has produced too much data-SLO damage to continue. This is where infrastructure chaos and data chaos diverge: the halt condition is a data metric (consumer lag), not pod_ready.
5. When the experiment completes, LitmusChaos writes a ChaosResult CR with the verdict (Pass if all probes passed, Fail if any probe breached). The verdict + probe timeline is the artifact you attach to the GameDay doc.

Output.

Time (s)	State	Consumer lag	Probe verdict
0	broker-2 running	800	Pass
5	broker-2 killed (SIGTERM)	900	Pass
20	leader re-election underway	1800	Pass
35	broker-2 restarted; ISR healing	5200	Pass
60	broker-2 in ISR; lag draining	3400	Pass (final)

Rule of thumb. Never write a data chaos experiment without a continuous probe tied to a data SLI. The probe is the seatbelt; the SLI is what you're protecting; the halt condition is the crash sensor. Miss any one and you're doing chaos-as-a-stunt, not chaos engineering.

Worked example — steady state as a data SLO, not an infra metric

Detailed explanation. The steady state is the single most under-specified concept in most first-attempt chaos programs. Teams reach for pod_ready == true or cpu_percent < 80 because those metrics are easy to graph. But a Kafka broker can be "pod_ready" while its ISR is shrunk to a single replica and every consumer is 45 minutes behind — infra says green, data says on-fire. A senior data chaos program defines steady state as a data SLI first, then adds infra SLIs as secondary indicators. Walk through the four canonical data SLIs and how to instrument each.

• Freshness p99. For any table or topic, the 99th percentile age of the newest committed record vs. wall clock time. The single most useful pipeline SLI.
• End-to-end lag. Consumer group lag in messages, or commit_ts - source_ts in a streaming SQL. Not the same as freshness (a lag of 0 with a stale source is not fresh).
• DAG success rate. For scheduled batch pipelines, successful_runs / (successful_runs + failed_runs) over a rolling 24h window.
• Correctness invariants. Row-count parity vs source, no negative revenue values, foreign-key completeness — pipeline-specific business assertions.

Question. Define steady state as a data SLO for a Kafka → Iceberg pipeline and derive the halt conditions for a chaos experiment that kills a Spark executor.

Input.

SLI	Definition	Target	Halt threshold
freshness_p99_min	max(now() - iceberg.commit_ts) p99	< 5 min	> 15 min
lag_messages	sum(kafka_lag) across group	< 10k	> 100k
batch_success_rate	5-min rolling	> 99%	< 90% for 10 min
row_count_delta	source - iceberg over 1h	< 100	> 10000

Code.

# Prometheus recording rules — one file, four data SLIs
groups:
  - name: pipeline-slis
    interval: 30s
    rules:
      - record: pipeline:freshness_p99_min
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(iceberg_commit_age_seconds_bucket{table="events"}[5m])
            )
          ) / 60

      - record: pipeline:lag_messages
        expr: |
          sum(kafka_consumergroup_lag{group="warehouse-sink"})

      - record: pipeline:batch_success_rate
        expr: |
          sum(rate(spark_streaming_batch_success_total[5m]))
          /
          sum(rate(spark_streaming_batch_total[5m]))

      - record: pipeline:row_count_delta
        expr: |
          (source_events_total - iceberg_rows_total{table="events"})

# ChaosEngine — Spark executor kill with all four SLIs as halt probes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: spark-executor-chaos
  namespace: spark
spec:
  engineState: 'active'
  appinfo:
    appns: 'spark'
    applabel: 'app=spark-executor'
    appkind: 'pod'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '300'
            - name: PODS_AFFECTED_PERC
              value: '10'
        probe:
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:lag_messages'
              comparator: {type: int, criteria: '<', value: '100000'}
          - name: batch-success-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:batch_success_rate'
              comparator: {type: float, criteria: '>', value: '0.90'}
          - name: correctness-guard
            type: promProbe
            mode: EOT   # end-of-test only
            promProbe/inputs:
              query: 'pipeline:row_count_delta'
              comparator: {type: int, criteria: '<', value: '10000'}

Step-by-step explanation.

1. The four recording rules define the pipeline's steady state in Prometheus terms. Every one is a data metric — freshness age, consumer lag, batch success ratio, source-vs-sink row delta. None of them mention pod_ready or cpu_percent; those become infra SLIs, tracked separately for triage but not gating the chaos verdict.
2. pipeline:freshness_p99_min derives from a histogram of iceberg_commit_age_seconds — the age (in seconds) of the newest committed Iceberg snapshot at scrape time. Dividing by 60 gives minutes. histogram_quantile(0.99, ...) gives the p99, so short spikes don't panic the probe.
3. The ChaosEngine attaches all four SLIs as probes. Three are Continuous (evaluated every N seconds during chaos); one (correctness-guard) is EOT (end-of-test only) — you only compute row-count parity after the chaos window has closed and things have settled.
4. The PODS_AFFECTED_PERC=10 sets the blast radius to 10% of matching pods. If you have 20 Spark executors, LitmusChaos kills 2. This is the "start small, graduate up" principle applied to a single experiment.
5. When any continuous probe fails, LitmusChaos aborts the experiment. When the EOT probe fails, the experiment is marked failed even if nothing tripped during the run. The distinction matters: correctness bugs often surface after the chaos, when the pipeline catches up and reveals a divergence.

Output.

SLI dashboard	Baseline	During chaos	Post-chaos	Verdict
freshness_p99_min	2.1	4.8	2.3	Pass
lag_messages	800	45,000	1,200	Pass
batch_success_rate	1.00	0.94	1.00	Pass
row_count_delta	12	(n/a)	34	Pass (EOT)

Rule of thumb. Steady state = data SLI. Halt condition = threshold on that SLI. Anything else is chaos theatre. If you can't articulate what "the pipeline is healthy" means in a single Prometheus query, you're not ready to inject chaos.

Worked example — what interviewers actually probe on data chaos

Detailed explanation. The senior data-platform chaos interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you validate your Kafka + Spark stack is resilient?"), then progressively narrows to test whether you understand the framework. Candidates who name the steady-state / hypothesis / experiment / verify loop in sentence one score highest; candidates who describe "we'd kill some pods and see" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "How do you know your pipeline survives a broker crash?" — invites you to name the framework.
• Follow-up 1. "What's your steady state?" — probes SLI axis.
• Follow-up 2. "How do you decide the blast radius?" — probes blast-radius axis.
• Follow-up 3. "What if the experiment breaks production?" — probes halt-condition axis.
• Follow-up 4. "Where do you run this — dev, staging, or prod?" — probes graduation-of-trust axis.
• Follow-up 5. "How is this different from load testing?" — probes methodology understanding.

Question. Draft a 5-minute senior chaos engineering answer that covers all axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Framework named	"we run chaos scripts"	"steady state → hypothesis → experiment → verify"
Steady state	"everything green in Grafana"	"freshness_p99_min < 5 and lag < 10k for 15 min"
Blast radius	"one broker"	"1 of 6 brokers in stage; 1 in prod only after 3 passing stage runs"
Halt condition	"we watch it"	"promProbe on freshness > 15 min stops the experiment"
Graduation	"prod eventually"	"dev → staging → 1% prod → 10% prod → 100% prod, one week between steps"
vs load testing	"it's like load testing"	"load tests break capacity; chaos breaks unavailability"

Code.

Senior chaos engineering answer template (5 minutes)
====================================================

Minute 1 — name the framework up front
  "I run chaos as a four-step loop: define the steady state as a
   data SLO, state a hypothesis about how the pipeline should
   behave under a specific failure, run the experiment inside a
   bounded blast radius with automatic halt on SLO breach, and
   verify by comparing SLI curves before/during/after."

Minute 2 — steady state
  "For our Kafka → Spark → Iceberg stack the steady state is:
   freshness_p99_min < 5, kafka_consumergroup_lag < 10000,
   batch_success_rate > 0.99, row_count_delta < 100 over 1 hour.
   Those four Prometheus queries are the SLI definition."

Minute 3 — hypothesis + experiment
  "Hypothesis: 'if we kill 1 of 6 Kafka brokers, freshness stays
   under 15 min and lag under 100k.' Experiment: LitmusChaos
   pod-delete on a broker labelled kafka-broker with
   PODS_AFFECTED_PERC=17, TOTAL_CHAOS_DURATION=60s, promProbes
   on all four SLIs, mode=Continuous, stopOnFailure=true."

Minute 4 — halt + blast radius
  "The blast radius is one broker out of six. Halt condition:
   any promProbe reads outside its threshold. Graduation:
   3 passing runs in staging over one week before promoting to
   a 1%-of-prod broker set, then 10%, then 100%. We never
   start with a bigger blast radius than we've already passed."

Minute 5 — verify + post-mortem
  "Verify: overlay before/during/after SLI curves in Grafana,
   write a GameDay doc with the hypothesis, actual outcome,
   probe timeline, and any backlog items. If the hypothesis
   failed, the backlog item is the fix; if it passed, the
   backlog item is the next larger blast radius."

Step-by-step explanation.

1. Minute 1 names the four-step loop. This is the Netflix Principles-of-Chaos / PoP framework in one sentence; naming it up front signals you know the discipline exists and isn't just "kill some pods."
2. Minute 2 states the steady state as four concrete Prometheus queries. This is the hardest thing for junior engineers to do — they'll say "the pipeline is healthy" but not codify what that means. Concrete numeric thresholds are senior signal.
3. Minute 3 fuses hypothesis + experiment. The hypothesis is a falsifiable prediction ("stays under 15 min lag"), not vague ("should be fine"). The experiment is the smallest fault that tests the hypothesis; anything bigger is a shotgun.
4. Minute 4 covers halt + blast radius — the safety controls. Naming stopOnFailure=true and the graduation ladder (dev → stage → 1% prod → 10% → 100%) shows you understand that chaos is a controlled experiment, not a demolition.
5. Minute 5 covers verify + post-mortem — the learning step. This is what separates chaos engineering from chaos monkey: every experiment produces either "resilience confirmed, graduate the blast radius" or "resilience broken, here's the backlog item."

Output.

Grading criterion	Weak score	Senior score
Names the four-step loop in minute 1	rare	mandatory
Steady state as SLI queries	occasional	mandatory
Hypothesis is falsifiable	rare	required
Halt condition + blast radius	rare	senior signal
Verify + post-mortem + backlog	rare	senior signal

Rule of thumb. The senior chaos answer is a 5-minute monologue that walks the steady-state → hypothesis → experiment → verify loop with concrete SLI thresholds and a graduation ladder. Rehearse it once; deploy it every time.

Senior interview question on chaos engineering for data pipelines

A senior interviewer often opens with: "Your team owns a Kafka + Spark Structured Streaming + Iceberg pipeline that promises 5-minute freshness to the warehouse. Design a chaos engineering program that proves the pipeline meets that SLO under broker crash, executor loss, and S3 latency spikes. Walk me through the steady-state definition, the hypothesis for each experiment, the blast-radius controls, the halt conditions, and how you graduate from staging to production."

Solution Using an SLI-first chaos program with graduated blast radius and halt-on-probe-failure

# 1. Prometheus recording rules — the steady-state SLI catalog
groups:
  - name: pipeline-steady-state
    interval: 30s
    rules:
      - record: pipeline:freshness_p99_min
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(iceberg_commit_age_seconds_bucket{table="events"}[5m]))
          ) / 60
      - record: pipeline:kafka_lag_messages
        expr: sum(kafka_consumergroup_lag{group="warehouse-sink"})
      - record: pipeline:batch_success_rate_5m
        expr: |
          sum(rate(spark_streaming_batch_success_total[5m]))
          / clamp_min(sum(rate(spark_streaming_batch_total[5m])), 1)
      - record: pipeline:row_count_delta_1h
        expr: |
          (source_events_1h_total - iceberg_rows_1h_total{table="events"})

# 2. Three chaos experiments, one file, shared halt conditions
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pipeline-chaos-suite
  namespace: chaos
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '120'
            - name: PODS_AFFECTED_PERC
              value: '17'                # 1 of 6 brokers
            - name: TARGET_PODS
              value: 'app=kafka-broker'
        probe: &shared_probes
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:kafka_lag_messages'
              comparator: {type: int, criteria: '<', value: '100000'}

    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '120'
            - name: PODS_AFFECTED_PERC
              value: '10'                # 1 of ~10 executors
            - name: TARGET_PODS
              value: 'app=spark-executor'
        probe: *shared_probes

    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: NETWORK_LATENCY
              value: '2000'              # +2s to S3 calls
            - name: DESTINATION_HOSTS
              value: 's3.us-east-1.amazonaws.com'
            - name: TARGET_PODS
              value: 'app=iceberg-writer'
        probe: *shared_probes

# 3. GitOps promotion — kustomize overlay per blast radius stage
# overlays/staging/kustomization.yaml
resources: [../../base]
patches:
  - target: {kind: ChaosEngine, name: pipeline-chaos-suite}
    patch: |-
      - op: replace
        path: /spec/appinfo/appns
        value: staging
      - op: replace
        path: /spec/experiments/0/spec/components/env
        value:
          - {name: PODS_AFFECTED_PERC, value: '17'}   # 1/6 brokers in stage
---
# overlays/prod-1pct/kustomization.yaml — promoted only after 3 passing stage runs
resources: [../../base]
patches:
  - target: {kind: ChaosEngine, name: pipeline-chaos-suite}
    patch: |-
      - op: replace
        path: /spec/appinfo/appns
        value: prod
      - op: replace
        path: /spec/experiments/0/spec/components/env
        value:
          - {name: PODS_AFFECTED_PERC, value: '17'}
          - {name: NODE_SELECTOR, value: 'blast=1pct'}

Step-by-step trace.

Step	Before (no chaos program)	After (SLI-first chaos suite)
Freshness SLO measurement	ad-hoc dashboards	4 recording rules; single-query steady state
Failure discovery	production incidents	scheduled staging chaos + game days
Halt behaviour	on-call reacts manually	promProbe auto-aborts on SLI breach
Blast radius control	none	PODS_AFFECTED_PERC per experiment
Graduation	none	dev → staging → 1% prod → 10% → 100% via kustomize overlays
Post-mortem	only after outages	after every experiment, pass or fail
Backlog outcome	vague "resilience" tickets	concrete fix or "graduate radius" ticket

After deployment, the chaos suite runs on a nightly cadence in staging, weekly in prod-1pct, monthly in prod-10pct. Every run writes a ChaosResult CR consumed by a Grafana dashboard that overlays the steady-state SLIs on the experiment timeline; deviations surface as either "resilience regression" alerts or "blast-radius graduation" tickets.

Output:

Metric	Before	After
Time-to-detect resilience regression	production incident (~hours)	staging chaos run (~30 min)
Halt latency on SLO breach	operator-driven (~minutes)	promProbe (~10 s)
Blast radius audit trail	none	ChaosResult CR + GameDay doc
Production chaos exposure	none or too much	1% → 10% → 100% graduation
Post-mortem coverage	outage-driven	every experiment

Why this works — concept by concept:

• Steady state as SLI — the pipeline is "healthy" iff the four recording rules stay within their thresholds. This turns "how do I know if chaos broke things" from a debate into a Prometheus query. The four SLIs collectively cover freshness (age of the latest data), lag (queue depth), success rate (task health), and correctness (source-vs-sink parity) — the four dimensions of data-pipeline health.
• Hypothesis-driven experiment — every ChaosEngine encodes a falsifiable prediction ("if I kill 1 of 6 brokers, freshness stays < 15 min"). Falsifiability is the difference between science and stunt; without a stated hypothesis, "we ran chaos" has no verdict.
• Bounded blast radius — PODS_AFFECTED_PERC=17 (1 of 6 brokers) is the explicit dial. Every experiment starts with the smallest blast that still tests the hypothesis; graduation is an earned promotion, not a default.
• Halt on probe failure — stopOnFailure: true on the promProbe gives you an automatic seatbelt. The experiment aborts the moment the SLI breaches; you don't need a human watching the dashboard.
• GitOps promotion via kustomize overlays — the same base ChaosEngine ships to dev / staging / prod-1pct / prod-10pct via kustomize overlays. This is the audit trail — the CR you ran in prod is provably the same one that passed in staging, differing only in blast-radius and namespace.
• Cost — one Prometheus (already deployed), LitmusChaos control plane (~1 pod, cluster-scoped), 4 recording rules (~ms/scrape), N ChaosEngine CRs per pipeline (~KB each). Compared to unmanaged production incidents, this is trading a small ops overhead for measurable resilience data. O(1) per experiment; O(scheduled cadence) per pipeline.

Design
Topic — design
Design problems on resilient data pipelines

Practice →

Streaming Topic — streaming Streaming resilience and back-pressure problems

Practice →

---

2. LitmusChaos on Kubernetes — pod / network / node kill experiments

ChaosEngine CR + probe guardrails — the K8s-native chaos framework built for GitOps

The mental model in one line: litmuschaos runs chaos experiments as Kubernetes custom resources (ChaosEngine, ChaosExperiment, ChaosResult) that target labelled workloads via a service account, execute a fault from a versioned experiment library (pod-delete, pod-network-loss, node-taint, disk-fill, pod-cpu-hog, pod-memory-hog, and dozens more), and gate the experiment behind probes (httpProbe, promProbe, cmdProbe, k8sProbe) that auto-abort on breach — the entire pattern is GitOps-friendly, CNCF-graduated, and free. Every senior K8s-native data platform team has evaluated LitmusChaos; most have shipped it because the CR-driven model composes naturally with existing GitOps + RBAC + Prometheus tooling.

The four axes for LitmusChaos.

• Deployment surface. Native Kubernetes. Runs as a Helm-installable control plane (litmus-chaos-operator) plus a per-namespace litmus-admin service account with RBAC scoped to the target workloads. No SaaS control plane, no external agents.
• Latency. Chaos starts within seconds of applying the ChaosEngine CR. Probe evaluation is per-probe interval (typically 10 s). Auto-abort fires within one probe interval after breach.
• Fault library. ~60 pre-built experiments across pod, node, network, disk, CPU, memory, and cloud (AWS EC2, EBS, RDS, Azure VM, GCP VM). Custom experiments are defined as ChartServiceVersion CRs — you can ship your own fault.
• Cost. Free (Apache-2.0). Optional paid observability + control plane via chaos-center. Operational cost is one operator pod per cluster plus one chaos-runner pod per active experiment.

The four probe kinds — the guardrails that make chaos safe.

• httpProbe. Hits an HTTP endpoint on the target or a nearby service; checks response body / status / latency. Use for "is my /health endpoint returning 200?"
• promProbe. Runs a PromQL query against Prometheus; compares the result against a threshold. Use for every data SLI — freshness, lag, success rate, correctness.
• cmdProbe. Executes a shell command inside a chaos-runner pod (or a sidecar); checks the exit code or output. Use for "kafka-console-consumer sees a message" style checks.
• k8sProbe. Reads a Kubernetes API object; compares its status field. Use for "is my StatefulSet Ready", "is my PVC Bound", "is my Deployment at the expected replica count."

Probe modes — when in the chaos lifecycle the probe runs.

• SOT (start-of-test). Once, before chaos begins. Used to validate the pre-condition — "is the system healthy enough to inject chaos?"
• EOT (end-of-test). Once, after chaos ends. Used to validate the recovery — "did the pipeline catch up within N minutes?"
• Edge. Both SOT and EOT.
• Continuous. Every N seconds throughout the chaos window. Used with stopOnFailure: true to enforce halt conditions.
• OnChaos. Only during the chaos period (skips before/after). Used for "verify a specific behaviour appears while the fault is active."

Common interview probes on LitmusChaos.

• "How does LitmusChaos abort an experiment?" — required answer: probe stopOnFailure: true + mode: Continuous; the operator writes a ChaosResult verdict of Fail.
• "How do you scope which pods get killed?" — applabel selector + PODS_AFFECTED_PERC; both are enforced by the operator.
• "How do you version chaos experiments?" — ChartServiceVersion CRs in a Git repo; chaos-hub is the community index.
• "How is LitmusChaos different from chaos-mesh?" — LitmusChaos is CR + probe + GitOps; chaos-mesh is CR + Web UI + dashboard. Both are CNCF; pick based on team's GitOps preference.

Worked example — pod-delete on a Kafka broker with a promProbe guardrail

Detailed explanation. The canonical LitmusChaos data-pipeline experiment: delete a Kafka broker pod, watch consumer lag stay under a threshold, verify the broker rejoins the ISR within a bounded time. Build the ChaosEngine, install the experiment chart, and apply it against a stage cluster.

• Target. A StatefulSet/kafka-broker in the kafka namespace with 6 replicas (kafka-broker-0 through kafka-broker-5).
• Fault. pod-delete from the generic chart on chaos-hub.
• Guardrail. promProbe on max(kafka_consumergroup_lag) with a < 50000 threshold.
• Blast radius. 1 pod at a time (PODS_AFFECTED_PERC=17).

Question. Write the ChaosEngine CR, the RBAC for the chaos service account, and the kubectl apply that runs the experiment.

Input.

Object	Purpose
ServiceAccount litmus-admin	scoped identity for the chaos runner
Role/RoleBinding	RBAC on pods, chaosengines, chaosresults
ChaosExperiment pod-delete	fault library chart (installed once per namespace)
ChaosEngine kafka-broker-chaos	the actual experiment definition
promProbe consumer-lag-guard	halt on lag > 50k

Code.

# 1. RBAC — the chaos runner needs to delete pods in the kafka namespace
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litmus-admin
  namespace: kafka
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: litmus-admin
  namespace: kafka
rules:
  - apiGroups: [""]
    resources: [pods, events, configmaps, secrets, services]
    verbs: [get, list, watch, create, delete, update, patch, deletecollection]
  - apiGroups: [apps]
    resources: [statefulsets, deployments, replicasets]
    verbs: [get, list, watch]
  - apiGroups: [litmuschaos.io]
    resources: [chaosengines, chaosexperiments, chaosresults]
    verbs: [get, list, create, update, patch, delete]
  - apiGroups: [batch]
    resources: [jobs]
    verbs: [get, list, create, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: litmus-admin
  namespace: kafka
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: litmus-admin}
subjects: [{kind: ServiceAccount, name: litmus-admin, namespace: kafka}]

---
# 2. Experiment chart — installed once per namespace
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
  namespace: kafka
spec:
  definition:
    scope: Namespaced
    permissions: [{apiGroups: [""], resources: [pods], verbs: [create, delete, get, list, patch, update]}]
    image: litmuschaos/go-runner:3.10.0
    imagePullPolicy: Always
    args: [-c, ./experiments -name pod-delete]
    command: [/bin/bash]
    env:
      - {name: TOTAL_CHAOS_DURATION, value: '60'}
      - {name: RAMP_TIME, value: ''}
      - {name: FORCE, value: 'false'}
      - {name: CHAOS_INTERVAL, value: '30'}
      - {name: PODS_AFFECTED_PERC, value: ''}
      - {name: TARGET_CONTAINER, value: ''}

---
# 3. ChaosEngine — the experiment execution
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-chaos
  namespace: kafka
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '60'}
            - {name: CHAOS_INTERVAL, value: '30'}
            - {name: FORCE, value: 'false'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 10
              retry: 2
              probePollingInterval: 10
              stopOnFailure: true
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator: {type: int, criteria: '<', value: '50000'}

# 4. Apply and watch
kubectl apply -f rbac.yaml -f experiment.yaml -f chaosengine.yaml

# 5. Watch the ChaosResult verdict
kubectl -n kafka get chaosresult kafka-broker-chaos-pod-delete -w
# → phase: Running          → phase: Completed
# → verdict: (blank)         → verdict: Pass | Fail

# 6. Inspect the probe timeline
kubectl -n kafka describe chaosresult kafka-broker-chaos-pod-delete

Step-by-step explanation.

1. The RBAC block creates a litmus-admin service account in the kafka namespace with just-enough permission: delete pods, read statefulsets, manage its own chaos CRs, create runner jobs. Nothing cluster-wide, nothing outside kafka.
2. The ChaosExperiment chart is installed once per namespace. It declares the fault image (litmuschaos/go-runner), the entry-point, and the default env vars. Think of it as a Helm chart for a chaos primitive.
3. The ChaosEngine is the actual experiment execution. appinfo scopes the fault to pods labelled app=kafka-broker (which selects kafka-broker-0 through kafka-broker-5 in the StatefulSet). PODS_AFFECTED_PERC=17 = ceil(6 × 0.17) = 1 pod.
4. The promProbe runs max(kafka_consumergroup_lag{group="warehouse-sink"}) against Prometheus every 10 s. If the query returns anything not < 50000, stopOnFailure: true halts the experiment, the chaos-runner exits, and the ChaosResult records a Fail verdict with the probe timeline.
5. After the experiment completes, kubectl describe chaosresult shows the phase (Completed), verdict (Pass / Fail), and each probe's evaluation history. This is the CI-friendly artifact you check into GitOps or feed into your GameDay dashboard.

Output.

Time (s)	Event	Probe: consumer-lag	Verdict
0	ChaosEngine applied	800 (< 50k)	Pass
5	chaos-runner spawned	900	Pass
10	kafka-broker-3 deleted	1200	Pass
20	leader re-election	4800	Pass
30	kafka-broker-3 restarted	12000	Pass
45	broker rejoined ISR	8000	Pass
60	chaos window closed	3200	Pass (final)

Rule of thumb. For any Kubernetes-native data platform, model chaos as ChaosEngine CRs alongside your workload manifests. RBAC-scope the runner to the target namespace, ship the CR through GitOps, and gate every experiment behind a promProbe bound to a data SLI.

Worked example — pod-network-loss between Airflow scheduler and metadata DB

Detailed explanation. A subtler chaos experiment: introduce packet loss between the Airflow scheduler pod and the metadata Postgres, verify the scheduler survives without corrupting DAG state, and confirm task instances resume cleanly. This is a stateful chaos scenario — the scheduler is the sole writer to dag_run / task_instance tables and a partial write is much scarier than a clean pod kill.

• Target. Deployment/airflow-scheduler, 1 pod.
• Fault. pod-network-loss — introduces packet loss on the pod's network interface to a specific destination.
• Destination. The Airflow metadata Postgres service (postgres-airflow.airflow.svc).
• Guardrail. k8sProbe on the scheduler Deployment ready status + cmdProbe that runs airflow db check.

Question. Write the ChaosEngine that injects 50% packet loss to Postgres for 60 s and verify the scheduler recovers without a corrupted metadata write.

Input.

Setting	Value
Fault	pod-network-loss
NETWORK_PACKET_LOSS_PERCENTAGE	50
DESTINATION_HOSTS	postgres-airflow.airflow.svc
DESTINATION_PORTS	5432
TOTAL_CHAOS_DURATION	60
Post-chaos verification	airflow db check + no stale task_instance rows

Code.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-netloss
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'app=airflow-scheduler'
    appkind: 'deployment'
  experiments:
    - name: pod-network-loss
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '60'}
            - {name: NETWORK_PACKET_LOSS_PERCENTAGE, value: '50'}
            - {name: DESTINATION_HOSTS, value: 'postgres-airflow.airflow.svc'}
            - {name: DESTINATION_PORTS, value: '5432'}
            - {name: CONTAINER_RUNTIME, value: 'containerd'}
            - {name: SOCKET_PATH, value: '/run/containerd/containerd.sock'}
        probe:
          - name: scheduler-ready-guard
            type: k8sProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 2, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'
          - name: metadata-consistency-check
            type: cmdProbe
            mode: EOT
            runProperties: {probeTimeout: 20, interval: 5, retry: 1, stopOnFailure: true}
            cmdProbe/inputs:
              command: |
                psql -h postgres-airflow.airflow.svc -U airflow -d airflow \
                     -tAc "SELECT count(*) FROM task_instance WHERE state='running' AND queued_dttm < NOW() - INTERVAL '10 minutes'"
              comparator: {type: int, criteria: '=', value: '0'}
              source:
                image: 'postgres:16-alpine'
                imagePullPolicy: IfNotPresent
                inheritInputs: true

Step-by-step explanation.

1. pod-network-loss uses tc netem (kernel traffic control) inside the target pod's network namespace to drop 50% of packets destined for postgres-airflow.airflow.svc:5432. The fault is bounded by IP + port so only Postgres traffic is affected; the scheduler can still reach the k8s API, the DAGs Git-sync sidecar, etc.
2. The k8sProbe runs continuously and checks the Airflow scheduler Deployment stays in a valid state. If the Deployment goes to zero ready replicas (i.e. the scheduler crashed and can't restart), stopOnFailure aborts the experiment.
3. The cmdProbe runs at end-of-test, launching a temporary postgres:16-alpine pod that connects to the Airflow metadata DB and asks: "are there any task_instances in state 'running' with a queue time older than 10 minutes?" If the answer is not zero, we have stuck rows — the network loss corrupted a state transition and the experiment fails.
4. The cmdProbe pattern is powerful because it lets you assert data-shape invariants that Prometheus can't natively express. "No stuck task instances" is a domain-specific invariant of Airflow's operational state; a PromQL query can approximate it but the cmdProbe gives you the ground-truth SQL.
5. The fault duration (60 s) is deliberately short — enough to force Postgres retries but not so long that scheduled DAG runs will start missing SLAs. Longer network-loss experiments should have PODS_AFFECTED_PERC reduced accordingly and require the game-day escalation.

Output.

Phase	Event	Probe verdict
Setup	ChaosEngine applied	—
0-60 s	50% packet loss to Postgres	scheduler-ready-guard: Pass (deployment still Ready)
60-70 s	packet loss removed; connection retries drain	—
EOT	SELECT count(*) FROM task_instance WHERE stuck	metadata-consistency-check: Pass (0 stuck)
Result	ChaosResult verdict	Pass

Rule of thumb. For stateful-writer chaos (scheduler, metadata DB, coordinator), pair a k8sProbe (liveness during chaos) with an EOT cmdProbe (data consistency after chaos). One catches the crash, the other catches the corruption.

Worked example — disk-fill on the Airflow scheduler pod

Detailed explanation. A resource-exhaustion chaos experiment: fill the Airflow scheduler pod's ephemeral disk to 95% and observe how the scheduler handles the disk pressure. Kubernetes may evict the pod under DiskPressure, the scheduler may fail to write logs, or the dags/ folder Git-sync sidecar may fail to pull new DAGs. Every one of these is a real production failure mode; making it a scheduled chaos experiment turns it from a surprise into a learning.

• Target. Deployment/airflow-scheduler, 1 pod.
• Fault. disk-fill — writes a large file to the pod's ephemeral storage.
• Fill target. 95% of the pod's ephemeral-storage limit.
• Guardrail. promProbe on pipeline:batch_success_rate_5m + k8sProbe on scheduler deployment.

Question. Write the ChaosEngine that fills the scheduler's ephemeral disk to 95% for 90 s and confirm DAG success rate stays above 90%.

Input.

Setting	Value
Fault	disk-fill
FILL_PERCENTAGE	95
EPHEMERAL_STORAGE_MEBIBYTES	4096
TOTAL_CHAOS_DURATION	90
Halt SLI	batch_success_rate_5m > 0.90

Code.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-diskfill
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'app=airflow-scheduler'
    appkind: 'deployment'
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '90'}
            - {name: FILL_PERCENTAGE, value: '95'}
            - {name: EPHEMERAL_STORAGE_MEBIBYTES, value: '4096'}
            - {name: CONTAINER_RUNTIME, value: 'containerd'}
            - {name: SOCKET_PATH, value: '/run/containerd/containerd.sock'}
            - {name: DATA_BLOCK_SIZE, value: '256'}
        probe:
          - name: batch-success-rate-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:batch_success_rate_5m'
              comparator: {type: float, criteria: '>', value: '0.90'}
          - name: scheduler-alive-guard
            type: k8sProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 2, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'

Step-by-step explanation.

1. disk-fill writes a chaos-fill file inside the target pod's writable filesystem using dd if=/dev/zero of=... bs=... count=.... It writes until the pod's ephemeral-storage usage hits FILL_PERCENTAGE (95%) of the declared limit.
2. If the pod has an ephemeral-storage request/limit set in its spec, Kubernetes may trigger DiskPressure eviction. If not, the pod's writable layer just fills up and any component that writes (Airflow's log rotator, the SQLite session store, the Git-sync sidecar) starts erroring. Both failure modes are interesting.
3. The promProbe on batch_success_rate_5m catches the case where the scheduler survives but DAG runs start failing (e.g. because the scheduler can no longer write task logs). The k8sProbe catches the eviction case.
4. DATA_BLOCK_SIZE=256 tunes the dd block size for reasonable I/O behaviour — small enough not to lock the disk, large enough to fill quickly.
5. The 90-second window is long enough for the disk-pressure symptom to propagate through the scheduler's next-heartbeat, DAG-parse, and task-launch cycles. Shorter windows may not exercise the full failure envelope.

Output.

Phase	Event	Probe: success rate	Probe: scheduler alive
0 s	Chaos applied	0.99	Pass
30 s	Disk 90% full	0.98	Pass
60 s	Disk 95% full; log writes failing	0.95	Pass
90 s	Chaos removed; disk freed	0.96	Pass
Post	Log-writer catches up	0.99	Pass (final)

Rule of thumb. Every stateful pod deserves a disk-fill chaos test. The failure modes are legion (log rotator crashes, sqlite locks, git-sync stalls, eviction storms) and every one of them surfaces only under actual disk pressure — no unit test finds them.

Senior interview question on LitmusChaos

A senior interviewer might ask: "You've adopted LitmusChaos as the K8s-native chaos framework for your data platform. Design the end-to-end pattern for shipping a new chaos experiment — from a Prometheus SLI definition through the ChaosEngine CR, the RBAC, the GitOps promotion, and the ChaosResult post-processing that lands in the on-call GameDay doc. Include how you'd version experiments and how you'd deprecate a broken one."

Solution Using a GitOps chaos pipeline with SLI-first probes and PR-gated promotion

# repo layout
# chaos/
#   base/
#     _slis.yaml               # Prometheus recording rules (shared)
#     _rbac.yaml               # litmus-admin ServiceAccount + Role
#     experiments/
#       pod-delete.yaml        # ChaosExperiment chart (installed once)
#       pod-network-loss.yaml
#       disk-fill.yaml
#     engines/
#       kafka-broker-kill.yaml
#       airflow-net-loss.yaml
#       spark-executor-kill.yaml
#   overlays/
#     dev/kustomization.yaml
#     staging/kustomization.yaml
#     prod-1pct/kustomization.yaml
#     prod-10pct/kustomization.yaml
#     prod-100pct/kustomization.yaml

# base/engines/kafka-broker-kill.yaml — the source of truth
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-kill
  labels:
    chaos.pipecode.ai/owner: 'data-platform-sre'
    chaos.pipecode.ai/tier: 'staging'
    chaos.pipecode.ai/version: 'v3'
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
        probe:
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}

# post_chaos.py — a small daemon that watches ChaosResult CRs and pipes to Slack + Jira
import subprocess, json, time, requests

SLACK_URL = "https://hooks.slack.com/services/xxx"
JIRA_URL  = "https://acme.atlassian.net/rest/api/2/issue"

def watch_chaosresults():
    proc = subprocess.Popen(
        ["kubectl", "get", "chaosresults", "-A", "-w", "-o", "json"],
        stdout=subprocess.PIPE, bufsize=1, text=True,
    )
    for line in proc.stdout:
        try:
            evt = json.loads(line)
        except json.JSONDecodeError:
            continue

        name  = evt["metadata"]["name"]
        phase = evt.get("status", {}).get("experimentStatus", {}).get("phase")
        verdict = evt.get("status", {}).get("experimentStatus", {}).get("verdict", "")
        probe_status = evt.get("status", {}).get("probeStatuses", [])

        if phase != "Completed":
            continue

        if verdict == "Pass":
            requests.post(SLACK_URL, json={
                "text": f":white_check_mark: chaos {name} passed. Consider graduating blast radius."
            })
        else:
            requests.post(SLACK_URL, json={
                "text": f":rotating_light: chaos {name} FAILED. Probe timeline: {probe_status}"
            })
            requests.post(JIRA_URL, json={
                "fields": {
                    "project": {"key": "DATA"},
                    "summary": f"Chaos regression: {name}",
                    "description": json.dumps(probe_status, indent=2),
                    "issuetype": {"name": "Bug"},
                    "labels": ["chaos-regression"],
                }
            }, auth=("[e-mail]", "***"))

if __name__ == "__main__":
    while True:
        try:
            watch_chaosresults()
        except Exception as e:
            print("watch died:", e)
            time.sleep(30)

Step-by-step trace.

Step	Component	Action
1	Engineer opens PR	edit base/engines/kafka-broker-kill.yaml — bump PODS_AFFECTED_PERC
2	CI	kustomize build; kubeconform schema check; OPA policy check
3	Argo CD	sync PR branch to dev cluster; nightly chaos run
4	ChaosResult	verdict recorded; posted to Slack + optional Jira
5	PR gets 3 green nightly runs	reviewer approves promotion to staging overlay
6	Same PR extended	overlays/prod-1pct/kustomization.yaml bumped
7	Argo CD	sync to prod-1pct cluster; monthly chaos run during business hours
8	3 green monthly runs	promote to prod-10pct
9	Deprecation	old engine engineState: 'stop' + PR labelled chaos-retirement

After deployment, every chaos experiment flows through a PR → dev → staging → prod-1pct → prod-10pct → prod-100pct GitOps pipeline. The ChaosResult CR is the receipt; the Slack + Jira integration is the human interface; the RBAC is the safety boundary. New experiments cannot skip stages; failed experiments cannot self-graduate.

Output:

Artifact	Purpose	Consumer
ChaosEngine CR in Git	source of truth for the experiment	Argo CD / reviewer / audit
ChaosResult CR in cluster	verdict + probe timeline	Slack bot / Grafana / GameDay doc
kustomize overlays	blast-radius graduation ladder	PR reviewer / audit
Slack alert	live notification	on-call rotation
Jira ticket	failure follow-up	data-platform-sre backlog

Why this works — concept by concept:

• ChaosEngine as CR — the experiment is a Kubernetes object, versionable, reviewable, RBAC-scoped. Not a Terraform script, not a Jenkins job — a first-class cluster object. The GitOps toolchain treats it the same as any Deployment.
• Shared SLI recording rules — a single _slis.yaml file defines the four data SLIs; every experiment's probes reference pipeline:* metrics. New SLIs land in one place; new experiments reuse them for free.
• RBAC-scoped runner — litmus-admin has permission to delete pods only in the target namespace, and only for chaos CRs it created. A compromised chaos runner cannot escalate; the blast radius is bounded by the RBAC.
• kustomize overlays as the graduation ladder — the same base CR ships to five stages differing only in PODS_AFFECTED_PERC, node selectors, and namespace. The diff between dev and prod-100pct is auditable in a single git diff output.
• PR-gated promotion — you cannot promote an experiment to a bigger blast radius without three green runs at the previous stage and a peer-reviewed PR. This is the human safety net that makes chaos in production defensible.
• ChaosResult post-processing — the verdict + probe timeline pipes to Slack (visibility) and Jira (accountability). Every failed experiment produces a ticket; every passed experiment produces a "consider graduating" signal.
• Cost — one operator pod, one runner pod per active experiment, a few KB per CR, one Slack webhook, one Jira token. Compared to the incident-driven alternative (learn resilience during real outages), this is a 100× reduction in mean-time-to-learn. O(1) per experiment; O(N experiments × N stages) per pipeline.

Design
Topic — design
Design problems on Kubernetes-native reliability

Practice →

ETL Topic — etl ETL problems on scheduled DAG failure recovery

Practice →

---

3. Gremlin — scenarios, blast-radius controls, workflows

Attack primitives → Scenarios → Halt-on-breach — the SaaS control plane for cross-platform chaos

The mental model in one line: gremlin is a commercial SaaS chaos platform whose control plane runs in the vendor's cloud and whose lightweight agent (gremlind) runs on every VM, container, ECS/Fargate task, or Kubernetes node you want to attack — it exposes low-level attack primitives (CPU, memory, disk, IO, latency, packet-loss, blackhole, DNS, shutdown, time-travel) that stitch into higher-level Scenarios with explicit blast-radius sliders and Halt-on-Breach Health Checks tied to your observability platform (Datadog, New Relic, Prometheus, custom webhooks). Every senior data platform team evaluating chaos tooling faces the LitmusChaos-vs-Gremlin decision; the deciding factor is almost always deployment surface (K8s-only vs cross-platform) and audit posture (self-hosted OSS vs SaaS with SOC-2).

The four axes for Gremlin.

• Deployment surface. Cross-platform. Runs on Linux, Windows, container, K8s, ECS, Fargate, on-prem, EC2, GCE. The gremlind agent is a single ~50MB Go binary that phones home to the SaaS control plane; attacks are triggered from the web UI, Terraform provider, or API.
• Fault library. ~15 attack primitives, richer than LitmusChaos on node-level and network-level primitives (DNS, packet-loss, time-travel, IO throttle). Weaker on Kubernetes-object-aware faults (no direct k8sProbe analog).
• Blast-radius model. Explicit sliders: "attack N clients matching tag=X." Multi-dimensional selectors: tag:cluster=prod, tag:service=kafka, container:image=confluent-kafka. Percentage sliders per selector.
• Halt behaviour. "Halt-on-Breach" scenarios — a scenario references a Datadog / New Relic / Prometheus / webhook Health Check; the check runs on a cadence; if any check reports unhealthy the scenario aborts and the attack rolls back.

Attack primitives — the low-level faults Gremlin ships.

• Resource. cpu (peg N cores at X%), memory (allocate X GB), disk (fill X% of a volume), io (throttle read/write bandwidth).
• Network. latency (add X ms to selected traffic), packet-loss (drop X% of packets), blackhole (drop all packets to/from a destination), dns (return NXDOMAIN or spoofed IP), bandwidth (throttle Mbps).
• State. shutdown (halt the host), process-killer (SIGTERM/SIGKILL specific PIDs/regex), time-travel (shift the system clock forward/backward).

Scenarios — the workflow layer.

• Definition. A Scenario is an ordered list of Steps. Each Step is either an Attack (an attack primitive with parameters and blast radius) or a Delay. Scenarios can also declare Health Checks that run continuously.
• Halt-on-Breach. If any Health Check reports unhealthy during scenario execution, all remaining Steps are cancelled and any active attack is halted (reverting tc netem, freeing memory, releasing CPU pegs). This is Gremlin's equivalent of LitmusChaos stopOnFailure.
• Attack rollback. Every attack primitive has an idempotent rollback path — killing the attack ID via API restores the pre-attack state. This is the safety property that makes production chaos possible.

Common interview probes on Gremlin.

• "How does Gremlin differ from LitmusChaos?" — required answer: cross-platform SaaS with agents vs K8s-native OSS with CRs; Gremlin has richer network/state primitives, LitmusChaos has richer K8s-object probes.
• "How do you halt a Gremlin attack on SLO breach?" — required answer: Halt-on-Breach Scenario with a Health Check tied to your observability platform.
• "How do you audit who ran what chaos?" — Gremlin control plane logs every attack with user, timestamp, blast-radius, targets; export via API to your SIEM.
• "How do you version Gremlin scenarios?" — Terraform gremlin_scenario resource or the REST API; check the JSON into Git alongside your infrastructure.

Worked example — blackhole attack on Spark executors with halt on SLA breach

Detailed explanation. A stateful chaos experiment: blackhole all network traffic on 10% of Spark executor nodes for 60 s, forcing shuffle-fetch failures on running Spark jobs. Verify the job SLA (batch_processing_seconds_p99) stays under threshold and the driver reprovisions executors. Model the experiment as a Gremlin Scenario with three steps: a latency warm-up, the main blackhole, and a Datadog Halt-on-Breach check.

• Target. EC2 instances tagged service=spark-executor, env=stage.
• Blast radius. 10% of matched clients.
• Steps. 30 s of +200 ms latency to warm up, then 60 s of blackhole to a specific S3 endpoint, then 30 s of cooldown.
• Halt. Datadog metric spark.batch.processing.p99 < 30s (per 1-min tumble).

Question. Write the Gremlin Scenario JSON, the Terraform resource that manages it, and the Datadog synthetic used as the Halt-on-Breach check.

Input.

Setting	Value
Target selector	tag:service=spark-executor tag:env=stage
Blast radius	10%
Step 1	latency +200 ms, 30 s
Step 2	blackhole s3.us-east-1.amazonaws.com, 60 s
Step 3	delay 30 s
Health check	Datadog: spark.batch.processing.p99 < 30 for 5 min

Code.

// Gremlin Scenario — POST /v1/scenarios
{
  "name": "spark-executor-blackhole-stage",
  "description": "Blackhole S3 traffic on 10% of Spark executors; halt on p99 batch time breach.",
  "hypothesis": "If 10% of Spark executors lose S3 connectivity for 60s, batch processing p99 stays under 30s and the driver reprovisions.",
  "graph": {
    "nodes": {
      "start": {"type": "InfraAttack", "next": "warmup"},
      "warmup": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Latency",
          "attackType": "latency",
          "ms": 200,
          "length": 30,
          "targetType": "Random",
          "percent": 10,
          "targets": {
            "type": "Exact",
            "tags": {"service": "spark-executor", "env": "stage"}
          }
        },
        "next": "main"
      },
      "main": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Blackhole",
          "attackType": "blackhole",
          "hostnames": "s3.us-east-1.amazonaws.com",
          "length": 60,
          "targetType": "Random",
          "percent": 10,
          "targets": {
            "type": "Exact",
            "tags": {"service": "spark-executor", "env": "stage"}
          }
        },
        "next": "cooldown"
      },
      "cooldown": {"type": "Delay", "durationSeconds": 30, "next": null}
    }
  },
  "healthChecks": [
    {
      "id": "hc-spark-batch-p99",
      "haltOnUnhealthy": true,
      "provider": "datadog",
      "config": {
        "monitorId": 12345678,
        "state": "OK"
      }
    }
  ]
}

# Terraform — manages the same scenario as code
resource "gremlin_scenario" "spark_blackhole_stage" {
  name        = "spark-executor-blackhole-stage"
  description = "Blackhole S3 traffic on 10% of Spark executors; halt on p99 batch time breach."
  hypothesis  = "If 10% of Spark executors lose S3 connectivity for 60s, batch p99 stays < 30s."

  step {
    type = "latency"
    attack_definition = jsonencode({
      ms      = 200
      length  = 30
      percent = 10
      tags    = { service = "spark-executor", env = "stage" }
    })
  }

  step {
    type = "blackhole"
    attack_definition = jsonencode({
      hostnames = "s3.us-east-1.amazonaws.com"
      length    = 60
      percent   = 10
      tags      = { service = "spark-executor", env = "stage" }
    })
  }

  step { type = "delay"; duration_seconds = 30 }

  health_check {
    provider          = "datadog"
    monitor_id        = 12345678
    halt_on_unhealthy = true
  }
}

# Datadog monitor definition — the halt criterion
# monitor.yaml
type: metric alert
query: "avg(last_5m):p99:spark.batch.processing.seconds{env:stage} < 30"
message: |
  Spark batch processing p99 breached during chaos experiment.
  Gremlin will halt this scenario if this monitor goes OK -> Alert.
tags: [chaos, spark, stage]
options:
  notify_no_data: false
  evaluation_delay: 60

Step-by-step explanation.

1. The Scenario graph declares three ordered steps: warmup (latency), main (blackhole), cooldown (delay). Gremlin executes them in graph order; the next pointers describe the DAG.
2. Each InfraAttack node has an impactDefinition with targetType: Random and percent: 10, meaning Gremlin will randomly pick 10% of the clients matching the tag selector service=spark-executor env=stage. If you have 20 executors, 2 get blackholed.
3. The blackhole attack drops all IP traffic from the target host to s3.us-east-1.amazonaws.com (which resolves to a set of IPs; the attack tracks the resolution). Spark's S3A committer will retry with exponential back-off; after N failed retries, the executor's task fails and the driver reschedules it.
4. The healthChecks[] array references a Datadog monitor. Gremlin polls the monitor state every ~30 s; if it transitions from OK to Alert, haltOnUnhealthy: true causes the scenario to abort — all remaining steps cancelled, active attacks rolled back (blackhole removed, latency removed).
5. The Terraform resource wraps the same underlying REST API; checking the resource into Git gives you an audit trail (who bumped the percent from 5 → 10, when, with what justification). The Datadog monitor is defined outside the Gremlin resource because it's shared with other alerting; the Gremlin scenario only references its ID.

Output.

Time (s)	Step	Health check	Verdict
0	warmup start	OK (batch p99: 8s)	continue
30	main start (blackhole)	OK (batch p99: 12s)	continue
60	shuffle-fetch failures begin	OK (batch p99: 22s)	continue
75	driver reprovisioning	OK (batch p99: 28s)	continue
90	main end; blackhole rolled back	OK (batch p99: 18s)	continue
120	cooldown end	OK (batch p99: 10s)	scenario Passed

Rule of thumb. For cross-platform (mixed VM + container + K8s) data planes, model chaos as Gremlin Scenarios with explicit percent blast radius and a Datadog / New Relic Halt-on-Breach Health Check tied to a data SLI. The scenario graph is your chaos-as-code artifact; the halt behaviour is your safety net.

Worked example — latency attack on S3 for Iceberg writes

Detailed explanation. A subtler chaos experiment: add 500 ms of latency to all S3 API calls from the Iceberg writer nodes for 5 minutes. Verify the writer doesn't lose commits, doesn't corrupt metadata (Iceberg manifest atomicity), and the p99 commit latency stays within the SLO. Latency chaos surfaces retry storms, connection-pool exhaustion, and back-pressure paths that no functional test hits.

• Target. EC2 instances tagged service=iceberg-writer, env=stage.
• Blast radius. 25% of matched clients.
• Attack. latency +500 ms to s3.us-east-1.amazonaws.com and s3.dualstack.us-east-1.amazonaws.com.
• Duration. 5 minutes.
• Halt. Prometheus: iceberg_commit_seconds_p99 < 30.

Question. Write the Gremlin Scenario, and add a Prometheus-backed Halt-on-Breach using a webhook health check.

Input.

Setting	Value
Attack	latency +500 ms
Destinations	s3.us-east-1.amazonaws.com, s3.dualstack.us-east-1.amazonaws.com
Duration	300 s
Blast radius	25% of iceberg-writer nodes
Halt	Prom: iceberg_commit_seconds_p99 < 30 for 3 min

Code.

{
  "name": "iceberg-s3-latency-stage",
  "hypothesis": "If S3 gains 500ms of latency on 25% of writer nodes, commit p99 stays under 30s and no manifest corruption occurs.",
  "graph": {
    "nodes": {
      "attack": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Latency",
          "attackType": "latency",
          "ms": 500,
          "hostnames": "s3.us-east-1.amazonaws.com,s3.dualstack.us-east-1.amazonaws.com",
          "length": 300,
          "targetType": "Random",
          "percent": 25,
          "targets": {
            "type": "Exact",
            "tags": {"service": "iceberg-writer", "env": "stage"}
          }
        },
        "next": null
      }
    }
  },
  "healthChecks": [
    {
      "id": "hc-iceberg-commit-p99",
      "haltOnUnhealthy": true,
      "provider": "webhook",
      "config": {
        "url": "https://chaos-webhooks.internal/gremlin/iceberg-commit-p99",
        "expectedStatus": 200,
        "pollIntervalSeconds": 30
      }
    }
  ]
}

# chaos-webhooks/iceberg-commit-p99 handler
# Small FastAPI service that turns a PromQL query into a webhook Gremlin can poll.
from fastapi import FastAPI, HTTPException
import httpx

app = FastAPI()

@app.get("/gremlin/iceberg-commit-p99")
async def iceberg_commit_p99():
    async with httpx.AsyncClient() as c:
        r = await c.get("http://prometheus.monitoring:9090/api/v1/query",
                        params={"query":
                                "histogram_quantile(0.99, sum by(le)("
                                "rate(iceberg_commit_seconds_bucket{env=\"stage\"}[3m])))"})
    val = float(r.json()["data"]["result"][0]["value"][1])
    if val > 30.0:
        # Report unhealthy — Gremlin halts the scenario
        raise HTTPException(status_code=503, detail=f"p99 {val:.2f} > 30s")
    return {"p99_seconds": val}

Step-by-step explanation.

1. The latency attack uses tc netem on the target host to add 500 ms delay to any packet destined for the listed hostnames. Both the standard and dualstack S3 endpoints are covered so IPv4/IPv6 traffic both see the injected latency.
2. targetType: Random, percent: 25 picks a random quarter of the matching hosts. If you have 8 iceberg-writer nodes, 2 get the latency injection; the other 6 write to S3 normally, so the pipeline as a whole should self-balance if writer selection is uniform.
3. The webhook health check is Gremlin's escape hatch when your observability platform isn't natively integrated. Any HTTP endpoint that returns 200 for "healthy" and non-200 for "unhealthy" works. Here we wrap a PromQL query into a FastAPI endpoint.
4. pollIntervalSeconds: 30 sets the cadence — Gremlin hits the webhook every 30 s. If the endpoint returns 503, Gremlin halts the scenario. The webhook itself is stateless and can be co-located with any Prometheus-adjacent service.
5. The 5-minute duration is chosen because Iceberg commit p99 is a lagging indicator — you need a few commit cycles for the latency to propagate into the SLI. Shorter windows may pass with false-negative "everything's fine" verdicts.

Output.

Time (min)	Attack	webhook status	Iceberg p99 (s)	Verdict
0	applied	200 (2.1)	2.1	continue
1	~2 writers slow	200 (6.4)	6.4	continue
2	retries exhaust pool	200 (14.9)	14.9	continue
3	commit p99 spike	200 (22.3)	22.3	continue
4	reprovisioning	200 (18.7)	18.7	continue
5	attack ended	200 (7.2)	7.2	Passed

Rule of thumb. For every external dependency (S3, DynamoDB, Snowflake, external HTTP) inject a latency chaos experiment. Latency chaos is the single highest-yield chaos primitive — it surfaces retry storms, connection-pool exhaustion, timeout misconfigurations, and back-pressure paths that no functional test finds.

Worked example — the blast-radius graduation matrix

Detailed explanation. Every Gremlin scenario is a candidate for graduation from dev → staging → 1% prod → 10% prod → 100% prod. The graduation matrix is the artifact that codifies the promotion rules: how many green runs at each stage before the next, who approves, what the rollback plan is. Walk through the matrix for the Kafka broker kill scenario.

• Stage 1 (dev). Nightly automation, no on-call, no approval.
• Stage 2 (staging). Weekly automation, tag-approved by data platform SRE.
• Stage 3 (prod-1pct). Monthly ceremony, game-day format, senior SRE + platform lead required.
• Stage 4 (prod-10pct). Quarterly, executive sponsor + full on-call bridge.
• Stage 5 (prod-100pct). Reserved for annual DR drills; formal DR runbook + customer comms.

Question. Codify the graduation matrix as a decision table, and derive the CI check that gates promotion.

Input.

Stage	Cadence	Approvals	Green runs required to promote	Rollback plan
dev	nightly	automation	5	halt-on-breach
staging	weekly	1 SRE	3	halt-on-breach
prod-1pct	monthly	2 SREs + lead	3	halt + on-call bridge
prod-10pct	quarterly	lead + exec	2	halt + on-call bridge + customer comms
prod-100pct	annual (DR drill)	DR sponsor	N/A (single ceremony)	formal DR runbook

Code.

# ci/gate_chaos_promotion.py — enforces the graduation matrix on PR merge
import argparse, requests, sys

MATRIX = {
    "dev":         {"required_green": 5, "prev_stage": None},
    "staging":     {"required_green": 3, "prev_stage": "dev"},
    "prod-1pct":   {"required_green": 3, "prev_stage": "staging"},
    "prod-10pct":  {"required_green": 2, "prev_stage": "prod-1pct"},
    "prod-100pct": {"required_green": 1, "prev_stage": "prod-10pct"},
}

def gremlin_recent_runs(scenario_id, tag_env, n=10):
    r = requests.get(f"https://api.gremlin.com/v1/scenarios/{scenario_id}/runs",
                     params={"limit": n, "tag:env": tag_env},
                     headers={"Authorization": f"Bearer {TOKEN}"})
    return r.json()["runs"]

def can_promote(scenario_id, from_stage, to_stage):
    prev_required = MATRIX[to_stage]["required_green"]
    runs = gremlin_recent_runs(scenario_id, from_stage, n=prev_required)
    greens = [r for r in runs if r["state"] == "Successful"]
    if len(greens) < prev_required:
        return False, f"Only {len(greens)} green runs at {from_stage}; need {prev_required}"
    return True, f"OK: {len(greens)} green runs at {from_stage}"

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--scenario-id", required=True)
    p.add_argument("--to-stage", required=True, choices=MATRIX.keys())
    a = p.parse_args()
    prev = MATRIX[a.to_stage]["prev_stage"]
    if not prev:
        print("Stage has no prerequisite; auto-allow.")
        sys.exit(0)
    ok, msg = can_promote(a.scenario_id, prev, a.to_stage)
    print(msg)
    sys.exit(0 if ok else 1)

Step-by-step explanation.

1. The MATRIX codifies "how many green runs at stage N do you need before you can promote to stage N+1." Dev requires 5 green runs (nightly for a week); staging requires 3; prod-1pct requires 3; prod-10pct requires 2; prod-100pct is a single-ceremony DR drill.
2. gremlin_recent_runs calls the Gremlin API to fetch the last N runs of a scenario tagged with the previous stage's environment. The tag is your foreign key between the CI check and the historical run data.
3. The CI script is invoked from your PR pipeline: gate_chaos_promotion.py --scenario-id ... --to-stage prod-1pct. If insufficient green runs exist at the previous stage, the CI check fails and the PR cannot merge.
4. The design deliberately reads run history from Gremlin (source of truth) rather than tracking green-run counts in the repo. This prevents the "someone updated the counter without an actual run" failure mode.
5. Executive approval for prod-10pct is enforced via GitHub CODEOWNERS + branch protection — the platform/* codeowner (VP Data Platform) must approve any PR touching overlays/prod-10pct/. The matrix is codified in both machine-checkable (CI) and human-checkable (CODEOWNERS) forms.

Output.

PR event	Stage promotion	CI verdict	Human approval
Create engine	dev	auto-allow	1 SRE reviewer
Promote dev → staging	staging	required: 5 green dev runs	1 SRE reviewer
Promote staging → prod-1pct	prod-1pct	required: 3 green staging runs	2 SREs + team lead
Promote prod-1pct → prod-10pct	prod-10pct	required: 3 green prod-1pct runs	team lead + exec sponsor
Promote to prod-100pct	prod-100pct (DR)	required: 2 green prod-10pct runs	DR sponsor + formal runbook

Rule of thumb. Every chaos scenario in production had to earn its way there by passing N green runs at each prior stage. Codify the graduation matrix in both CI (machine-checkable) and CODEOWNERS (human-checkable). The matrix is the artifact your auditor will ask to see.

Senior interview question on Gremlin

A senior interviewer might ask: "You're bringing chaos engineering to a multi-cloud data platform (Kafka on-prem, Airflow on EKS, Spark on Databricks, Snowflake). Design the Gremlin-based chaos program: which primitives, which scenarios, which halt criteria, and how you manage the tag hierarchy so on-call knows what's being attacked at any moment."

Solution Using a tag-hierarchy-driven scenario library with SIEM-integrated audit trail

// 1. Tag hierarchy — every gremlind agent inherits these tags
// gremlind config at /etc/gremlin/config.yaml
{
  "team": "data-platform",
  "identifier": "auto",
  "tags": {
    "cluster": "kafka-onprem-1",
    "service": "kafka-broker",
    "env": "prod",
    "criticality": "tier-1",
    "chaos-eligible": "true",
    "chaos-tier": "1pct"
  }
}

# 2. Terraform module — reusable scenario definitions per stack
module "kafka_broker_kill_scenarios" {
  source = "./modules/gremlin-broker-kill"

  for_each = {
    "kafka-onprem-1-stage" = { env = "stage", percent = 17 }
    "kafka-onprem-1-1pct"  = { env = "prod", percent = 17, chaos_tier = "1pct" }
    "kafka-onprem-1-10pct" = { env = "prod", percent = 17, chaos_tier = "10pct" }
  }

  scenario_name = each.key
  env           = each.value.env
  percent       = each.value.percent
  chaos_tier    = try(each.value.chaos_tier, null)

  health_check_monitor_id = data.datadog_monitor.kafka_lag_slo.id
  halt_on_unhealthy       = true
}

# 3. SIEM exporter — mirror every Gremlin attack to Splunk for audit
# runs as a scheduled job every 5 minutes
import os, requests, json, hmac, hashlib
from datetime import datetime, timedelta, timezone

GREMLIN_TOKEN = os.environ["GREMLIN_TOKEN"]
SPLUNK_HEC    = os.environ["SPLUNK_HEC_URL"]
SPLUNK_HEC_TOKEN = os.environ["SPLUNK_HEC_TOKEN"]

def poll_gremlin_since(since_iso):
    r = requests.get("https://api.gremlin.com/v1/attacks",
                     params={"since": since_iso},
                     headers={"Authorization": f"Bearer {GREMLIN_TOKEN}"})
    return r.json()["attacks"]

def to_splunk_event(attack):
    return {
        "time":  attack["created_at"],
        "sourcetype": "gremlin:attack",
        "event": {
            "attack_id":   attack["id"],
            "scenario":    attack.get("scenarioId"),
            "user":        attack["createdBy"]["email"],
            "attack_type": attack["type"],
            "targets":     attack["targets"],
            "state":       attack["state"],
            "started_at":  attack["startedAt"],
            "ended_at":    attack.get("endedAt"),
            "halted":      attack.get("halted", False),
        }
    }

def push_to_splunk(events):
    for e in events:
        requests.post(SPLUNK_HEC,
                      headers={"Authorization": f"Splunk {SPLUNK_HEC_TOKEN}"},
                      json=e, timeout=5)

def main():
    since = (datetime.now(timezone.utc) - timedelta(minutes=10)).isoformat()
    attacks = poll_gremlin_since(since)
    push_to_splunk(to_splunk_event(a) for a in attacks)

if __name__ == "__main__":
    main()

Step-by-step trace.

Layer	Component	Responsibility
Agents	gremlind on every VM/container	inherit tag hierarchy; execute attack; report health
Control plane	Gremlin SaaS	scenario execution, blast-radius, halt
Scenario definitions	Terraform module	one module per fault type; per-env instances via for_each
Health checks	Datadog monitors	data SLI thresholds; Gremlin polls these
Halt behaviour	Gremlin halt_on_unhealthy	Datadog Alert → scenario abort
Audit trail	SIEM exporter → Splunk	every attack ID, user, targets, timing mirrored to SIEM
Approval matrix	GitHub CODEOWNERS + CI	machine-checked green-run count + human sign-off
On-call visibility	Slack #chaos-live channel	scenario start / end / halt posts via Gremlin webhook

After deployment, every chaos attack against the data platform is: (a) defined in Terraform (audit trail), (b) executed via the Gremlin API (blast-radius enforced), (c) gated by Datadog Halt-on-Breach (data-SLO seatbelt), (d) mirrored to Splunk (compliance receipt), (e) announced to #chaos-live (operator awareness). On-call always knows what's running; auditors always have the receipt.

Output:

Artifact	Purpose	Consumer
Terraform state	scenario definitions	infra reviewer / audit
Gremlin control plane	live scenario execution	data-platform SRE
Datadog monitor	halt criterion	Gremlin health-check poller
Splunk gremlin:attack events	audit trail	compliance / SOC-2
Slack #chaos-live post	live operator awareness	on-call
Post-mortem doc (Confluence)	scenario outcome + backlog	data-platform team

Why this works — concept by concept:

• Tag hierarchy on every agent — the service / env / criticality / chaos-tier tags define the selector language for every scenario. A well-designed tag hierarchy makes "blast radius = 10% of service=kafka-broker in env=prod chaos-tier=1pct" a one-line selector.
• Terraform module per fault type — a Terraform module encodes the fault + its parameters + its halt criterion. for_each over {stage, 1pct, 10pct} instantiates the same scenario at each blast tier with the right selectors. Chaos-as-code becomes chaos-in-Terraform.
• Datadog Halt-on-Breach — the halt criterion is a Datadog monitor, not a Gremlin-internal check. This decouples the SLI definition from the chaos tool — the same SLI monitor pages on-call and halts chaos. One source of truth.
• SIEM audit trail — every attack ID + user + targets + timing mirrors to Splunk. When a security auditor asks "who ran chaos in prod on 2026-07-30 at 14:00 and what did it hit," you point at a Splunk search. This is what makes prod chaos defensible in a SOC-2 environment.
• Slack live channel — #chaos-live gets a post at scenario start (with hypothesis + expected duration + rollback path), during halts, and at scenario end (with verdict). Operators can silence chaos-triggered alerts because they know a chaos window is active.
• CI-gated promotion — the same graduation-matrix pattern from LitmusChaos applies: dev → staging → 1% prod → 10% → 100%. CI counts green runs; CODEOWNERS enforces human approval. Chaos in prod is earned, not granted.
• Cost — Gremlin SaaS licence (per-agent per-month), gremlind footprint (~50 MB per host), one Terraform module per fault type, one Datadog monitor per SLI, one Splunk index for chaos events. Compared to the DIY equivalent (write attack primitives, build a control plane, build audit, build halt), Gremlin trades money for time-to-value. O(1) per attack; O(N scenarios × N stages) per pipeline.

Design
Topic — design
Design problems on multi-cloud data platform resilience

Practice →

API Topic — api-integration API integration problems on webhook and halt endpoints

Practice →

---

4. Data-plane fault injection — Kafka broker kill, Airflow scheduler kill, Spark executor loss

Stateful chaos — the three canonical data-plane experiments and their guardrails

The mental model in one line: data fault injection is the practice of applying chaos primitives (pod-delete, network-loss, blackhole, latency) to the stateful data systems — Kafka brokers with partition leadership + ISR + transactional coordinators, Airflow schedulers with sole write-access to a metadata DB, and Spark executors with in-memory shuffle blocks and driver-tracked lineage — where the correctness contract binds the experiment far more tightly than for stateless HTTP services. Every senior data platform team runs these three experiments; the difference between amateur and senior is whether the experiment ships with the specific data-system guardrail that makes the chaos survivable — min.insync.replicas=2 + acks=all for Kafka, catchup=False + max_active_runs=1 for Airflow, dynamicAllocation.enabled=true + blacklist.enabled=true for Spark.

The three canonical data-plane chaos experiments.

• Kafka broker kill. Kill a broker; verify the pipeline stays inside its freshness + lag SLO. Failure modes: unacked writes if acks!=all, ISR shrink storms, coordinator loss (transactional-EOS blast). Guardrails: min.insync.replicas=2, acks=all, producer.idempotence=true, retention.ms sized to slot lag.
• Airflow scheduler kill. Kill the scheduler mid-DAG; verify running tasks continue and new tasks are scheduled after the restart. Failure modes: mid-parse crashes leave dag_run in queued forever, task_instance rows stuck in running when the executor is a Celery worker, catchup floods the queue on restart. Guardrails: catchup=False, max_active_runs=1, retries>=2, dagrun_timeout set.
• Spark executor loss. Kill a Spark executor; verify the driver reprovisions and the batch/streaming job continues. Failure modes: shuffle-fetch failures cascade into stage restarts, driver OOM if too many executors lost, dynamic-allocation flap. Guardrails: spark.dynamicAllocation.enabled=true, spark.blacklist.enabled=true, spark.task.maxFailures=4, spark.stage.maxConsecutiveAttempts=4.

The stateful chaos multiplier — why data-plane chaos is different.

• In-flight state. A stateless HTTP service can be killed mid-request and the client just retries. A Kafka broker killed mid-transaction leaves the coordinator's state machine mid-way; recovery requires either a coordinator failover or a transactional abort. The chaos experiment must know which.
• Ordering contracts. Kafka partitions guarantee in-partition order; a broker kill preserves it. Spark shuffle blocks are not ordered but they are addressed by (mapId, reduceId); executor loss forces the driver to re-fetch from surviving executors, which the shuffle service handles transparently if configured. Airflow DAG runs are logical ordering only; the scheduler kill risks orphaning a mid-run task_instance if the executor is out-of-process.
• SLI selection. For each experiment the correct SLI differs. Kafka: consumer group lag + producer error rate. Airflow: DAG success rate + scheduler heartbeat age. Spark: batch processing time p99 + stage retry count. Getting the SLI wrong makes the experiment blind.
• Recovery topology. Kafka has a metadata quorum (ZooKeeper or KRaft) and a partition leader per topic-partition — recovery is per-partition. Airflow has a single scheduler + N executors — recovery is scheduler-restart-plus-heartbeat. Spark has a driver + N executors — recovery is executor-reprovision.

The three guardrails you cannot ship without.

• Kafka: acks=all on the producer + min.insync.replicas=2 on the topic. Without both, a broker kill can lose an unacknowledged write. This is the single most-cited Kafka correctness invariant in interviews.
• Airflow: catchup=False on the DAG + max_active_runs=1. Without both, a scheduler restart after a downtime replays every missed logical date and can OOM the queue.
• Spark: spark.dynamicAllocation.enabled=true + spark.blacklist.enabled=true. Without both, an executor loss either leaves the cluster under-provisioned or repeatedly retries onto the same bad node.

Common interview probes on data-plane chaos.

• "How do you verify a Kafka broker kill didn't lose writes?" — required answer: acks=all + min.insync.replicas=2 + producer-error-rate SLI + downstream row-count reconciliation.
• "What breaks when you kill an Airflow scheduler mid-DAG?" — orphaned task_instance rows in running state if executor is separate; dag_run stuck in queued if the parse loop crashed.
• "What's the Spark executor-loss failure mode?" — shuffle-fetch failures on downstream stages; without dynamicAllocation, the cluster shrinks and the job hangs.
• "How do you pick which broker to kill?" — random over applabel=kafka-broker avoiding the controller; the controller loss is a separate experiment (controller-kill) because it exercises different code paths.

Worked example — Kafka broker kill with min.insync.replicas guard

Detailed explanation. The canonical Kafka data-plane chaos experiment: kill one broker in a 6-broker cluster (replication.factor=3, min.insync.replicas=2) and verify no committed messages are lost and the producer's record-error-rate stays at zero. This is the interview go-to; every senior Kafka engineer has run it.

• Cluster. 6 brokers, replication.factor=3, min.insync.replicas=2, KRaft-mode metadata quorum.
• Producer. acks=all, enable.idempotence=true, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5.
• Fault. LitmusChaos pod-delete on 1 of 6 brokers (17% blast radius).
• Guardrails. promProbe on kafka_consumergroup_lag < 100000 + promProbe on rate(kafka_producer_record_error_total[1m]) < 0.1.

Question. Write the ChaosEngine + the producer / topic config that makes this experiment survivable, and quantify what happens if you drop the min.insync.replicas guard.

Input.

Setting	With guard	Without guard
replication.factor	3	3
min.insync.replicas	2	1
acks (producer)	all	1
enable.idempotence	true	false
Broker kill outcome	0 lost writes	~250 lost writes

Code.

# 1. Topic configuration — enforced by kafka-topics.sh at creation
# kafka-topics.sh --create \
#   --topic events \
#   --partitions 24 \
#   --replication-factor 3 \
#   --config min.insync.replicas=2 \
#   --config unclean.leader.election.enable=false \
#   --bootstrap-server localhost:9092

// 2. Producer configuration — the two lines that make the chaos survivable
Properties p = new Properties();
p.put("bootstrap.servers", "kafka-0:9092,kafka-1:9092,kafka-2:9092");
p.put("acks",                                 "all");
p.put("enable.idempotence",                   "true");
p.put("retries",                              Integer.MAX_VALUE);
p.put("max.in.flight.requests.per.connection", "5");
p.put("delivery.timeout.ms",                  "120000");
p.put("request.timeout.ms",                    "30000");
p.put("linger.ms",                                "20");
p.put("compression.type",                      "lz4");

# 3. ChaosEngine — kill 1 of 6 brokers with two data-SLI probes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-kill
  namespace: kafka
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker,role!=controller'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: CHAOS_INTERVAL, value: '60'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
            - {name: FORCE, value: 'false'}
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator: {type: int, criteria: '<', value: '100000'}
          - name: producer-error-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'sum(rate(kafka_producer_record_error_total[1m]))'
              comparator: {type: float, criteria: '<', value: '0.1'}
          - name: isr-integrity-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'min(kafka_topic_partition_in_sync_replica{topic="events"})'
              comparator: {type: int, criteria: '>=', value: '2'}

Step-by-step explanation.

1. The topic is created with replication.factor=3, min.insync.replicas=2, unclean.leader.election.enable=false. When a broker holding a partition leader dies, controller elects a new leader from the ISR; if the ISR has < 2 in-sync replicas the producer receives NotEnoughReplicasException and retries. With unclean-leader-election disabled, no out-of-sync replica is ever elected, so no committed writes vanish.
2. The producer sets acks=all (waits for the leader + all in-sync replicas to acknowledge) and enable.idempotence=true (attaches a producer-id + sequence, so retries don't duplicate). Together these give exactly-once-per-producer semantics — the exact property the chaos experiment verifies.
3. applabel: 'app=kafka-broker,role!=controller' explicitly excludes the controller pod. Controller loss is a different chaos experiment because the controller drives partition-leader election — testing controller loss requires different halt criteria (controller-fail-over time SLO instead of consumer lag).
4. The three probes cover three orthogonal SLIs: (a) consumer lag stays bounded (freshness); (b) producer error rate stays low (durability); (c) minimum ISR stays >= 2 (correctness contract). If any breaches, the experiment aborts.
5. FORCE: 'false' means the broker gets a graceful shutdown (SIGTERM → configured shutdown.timeout.ms → SIGKILL). Kafka brokers flush their memory-mapped log segments on graceful shutdown; a forced kill would test crash-recovery instead. Both are valuable; label them as different experiments.

Output.

Time (s)	Event	Consumer lag	Producer error rate	Min ISR
0	broker-3 alive	800	0.00	3
10	broker-3 SIGTERM	900	0.00	3
15	broker-3 gone; ISR shrinks	1400	0.02	2
30	new leaders elected	4200	0.01	2
60	broker-3 restarted	12000	0.00	2
90	ISR healing	8000	0.00	3
120	full ISR restored	3200	0.00	3

Rule of thumb. Kafka broker chaos without acks=all + min.insync.replicas=2 is theatre — you're not testing the resilient system, you're testing an unsafe one. Enforce both at the topic + producer config level before the chaos experiment ships. Add an isr-integrity-guard probe to make the min-ISR contract auditable during chaos.

Worked example — Airflow scheduler kill with catchup=False guard

Detailed explanation. The canonical Airflow chaos experiment: kill the scheduler pod while a DAG has tasks in-flight, verify running tasks complete (executor is Celery or Kubernetes, out-of-process), verify the scheduler restarts within HEALTHCHECK_MAX_AGE and doesn't flood the queue with catchup runs on restart, verify no task_instance rows are stuck in running state after the scheduler is healthy again.

• Setup. Airflow 2.9 with KubernetesExecutor, scheduler HA disabled (single scheduler), HEALTHCHECK_MAX_AGE=30s.
• DAG. Hourly ETL with catchup=False, max_active_runs=1, retries=2.
• Fault. pod-delete on scheduler pod.
• Guardrails. k8sProbe on scheduler Deployment + cmdProbe on select count(*) from task_instance where state='running' and queued_dttm < now() - interval '10 minutes'.

Question. Write the ChaosEngine and demonstrate what breaks if catchup=True is misconfigured.

Input.

Setting	With guard	Without guard
catchup	False	True
max_active_runs	1	unbounded
Scheduler downtime	30 s	30 s
Post-restart runs	1	~24 (one per missed hour × 24 hrs)

Code.

# The DAG with the correct guardrails
from airflow import DAG
from airflow.decorators import task
from datetime import datetime, timedelta

with DAG(
    dag_id="hourly_etl",
    start_date=datetime(2026, 1, 1),
    schedule="@hourly",
    catchup=False,                 # <-- guard 1: don't backfill missed runs
    max_active_runs=1,             # <-- guard 2: never more than one instance
    dagrun_timeout=timedelta(minutes=45),
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "execution_timeout": timedelta(minutes=30),
    },
) as dag:

    @task
    def extract(): ...
    @task
    def transform(x): ...
    @task
    def load(y): ...

    load(transform(extract()))

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-kill
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'component=scheduler'
    appkind: 'deployment'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '90'}
            - {name: CHAOS_INTERVAL, value: '45'}
            - {name: PODS_AFFECTED_PERC, value: '100'}
            - {name: FORCE, value: 'false'}
        probe:
          - name: scheduler-restart-guard
            type: k8sProbe
            mode: EOT
            runProperties: {probeTimeout: 30, interval: 5, retry: 3, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'
          - name: no-stuck-tasks
            type: cmdProbe
            mode: EOT
            runProperties: {probeTimeout: 30, interval: 5, retry: 1, stopOnFailure: true}
            cmdProbe/inputs:
              command: |
                psql -h postgres-airflow.airflow.svc -U airflow -d airflow -tAc \
                  "SELECT COUNT(*) FROM task_instance
                   WHERE  state='running'
                     AND  queued_dttm < NOW() - INTERVAL '10 minutes'"
              comparator: {type: int, criteria: '=', value: '0'}
              source:
                image: 'postgres:16-alpine'
                imagePullPolicy: IfNotPresent
                inheritInputs: true
          - name: no-catchup-flood
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 5, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'airflow_dagrun_active_count{dag_id="hourly_etl"}'
              comparator: {type: int, criteria: '<=', value: '1'}

Step-by-step explanation.

1. PODS_AFFECTED_PERC: 100 on a Deployment with 1 replica means the entire scheduler dies. Kubernetes will reschedule the pod immediately; the scheduler restarts, re-parses DAGs, resumes its scheduling loop. Executor tasks (Kubernetes pods) continue running because they're separate pods; the scheduler only manages state transitions.
2. catchup=False is the load-bearing invariant. On restart, the scheduler asks "what's the last logical date I scheduled for this DAG?" Without catchup, it schedules only the next logical date. With catchup=True and 24h of downtime, it would try to schedule 24 backfill runs immediately, potentially overwhelming the executor.
3. max_active_runs=1 is the second guard: even if the scheduler somehow tries to schedule multiple runs, only 1 is allowed active. This bounds the blast radius of a scheduler bug.
4. The EOT cmdProbe runs SQL against the metadata DB after chaos ends: "any task_instances stuck in 'running' state where their queue time is > 10 minutes ago?" If the scheduler died mid-transition on a task_instance (e.g. between queued and running), the row can be orphaned. The probe catches this.
5. The EOT promProbe on airflow_dagrun_active_count <= 1 verifies no catchup flood. If it fires, either the DAG has catchup=True (config bug) or the scheduler restart triggered multiple runs (Airflow bug).

Output.

Time (s)	Event	Scheduler ready	Stuck tasks	Active DAG runs
0	Chaos applied	1	0	1
5	Scheduler pod killed	0	0 (in-flight preserved)	1
20	Pod restarted; parsing DAGs	0	0	1
40	Scheduler resuming loop	1	0	1
60	Chaos window ends	1	0	1
EOT	Verification	1	0	1

Rule of thumb. For any Airflow chaos experiment, verify the DAG has catchup=False + max_active_runs=1 before running the experiment. Ship the two-line SQL probe that hunts orphaned task_instance rows as an EOT guard — it's the single most under-instrumented Airflow failure mode.

Worked example — Spark executor loss with dynamic allocation

Detailed explanation. The canonical Spark chaos experiment: kill 2 of 10 Spark executors mid-batch, verify the driver reprovisions replacements (dynamic allocation), verify shuffle-fetch failures don't cascade into unbounded stage retries, verify the job SLA (batch processing p99) stays under threshold. This tests both Spark's own fault tolerance and the surrounding YARN / K8s resource-manager's ability to reprovision.

• Cluster. Spark 3.5 on K8s, 1 driver + 10 executor pods, spark.dynamicAllocation.enabled=true.
• Guards. spark.blacklist.enabled=true, spark.task.maxFailures=4, spark.stage.maxConsecutiveAttempts=4, spark.dynamicAllocation.minExecutors=8.
• Fault. LitmusChaos pod-delete on 20% of executors (2 of 10).
• Halt. spark_streaming_batch_processing_seconds_p99 < 30.

Question. Write the ChaosEngine and the Spark submit config that makes executor loss survivable.

Input.

Setting	Value
spark.dynamicAllocation.enabled	true
spark.dynamicAllocation.minExecutors	8
spark.dynamicAllocation.maxExecutors	20
spark.blacklist.enabled	true
spark.task.maxFailures	4
spark.stage.maxConsecutiveAttempts	4
spark.shuffle.service.enabled	true (external shuffle service)

Code.

# 1. Spark submit config that survives executor loss
spark-submit \
  --master k8s://https://kubernetes.default.svc \
  --deploy-mode cluster \
  --conf spark.executor.instances=10 \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=8 \
  --conf spark.dynamicAllocation.maxExecutors=20 \
  --conf spark.dynamicAllocation.executorIdleTimeout=60s \
  --conf spark.blacklist.enabled=true \
  --conf spark.blacklist.timeout=1h \
  --conf spark.task.maxFailures=4 \
  --conf spark.stage.maxConsecutiveAttempts=4 \
  --conf spark.shuffle.service.enabled=true \
  --conf spark.kubernetes.executor.label.app=spark-executor \
  --conf spark.kubernetes.executor.deleteOnTermination=true \
  --conf spark.streaming.stopGracefullyOnShutdown=true \
  --conf spark.sql.streaming.gracefulShutdown.timeout=60s \
  --class com.acme.EventStreamJob \
  local:///opt/spark/jobs/event-stream.jar

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: spark-executor-kill
  namespace: spark
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'spark'
    applabel: 'app=spark-executor'
    appkind: 'pod'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '180'}
            - {name: CHAOS_INTERVAL, value: '60'}
            - {name: PODS_AFFECTED_PERC, value: '20'}
            - {name: FORCE, value: 'true'}
        probe:
          - name: batch-p99-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: |
                histogram_quantile(0.99,
                  sum by(le)(rate(spark_streaming_batch_processing_seconds_bucket{job="event-stream"}[3m])))
              comparator: {type: int, criteria: '<', value: '30'}
          - name: executor-reprovision-guard
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'spark_executor_active_count{job="event-stream"}'
              comparator: {type: int, criteria: '>=', value: '8'}
          - name: stage-retry-guard
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(spark_stage_consecutive_attempts{job="event-stream"})'
              comparator: {type: int, criteria: '<', value: '4'}

Step-by-step explanation.

1. spark.dynamicAllocation.enabled=true lets the driver request new executors from the resource manager when load requires. minExecutors=8 sets the floor — even after two are killed, the driver will request replacements to get back above 8.
2. spark.blacklist.enabled=true + spark.blacklist.timeout=1h prevents Spark from repeatedly retrying tasks onto a node that just crashed an executor. Without this, a bad node produces a cascade of retries as the driver keeps assigning tasks that keep failing.
3. spark.shuffle.service.enabled=true runs the external shuffle service (as a DaemonSet on K8s or a NodeManager auxiliary on YARN) which persists shuffle blocks outside the executor lifetime. Without it, executor death loses shuffle data and forces a re-compute; with it, surviving executors keep serving shuffle blocks.
4. spark.streaming.stopGracefullyOnShutdown=true on the streaming job ensures the driver drains active batches before exiting on a SIGTERM. Combined with spark.kubernetes.executor.deleteOnTermination=true, executor pods clean up their K8s resources on exit.
5. The three probes cover three orthogonal invariants: (a) batch processing p99 stays under 30 s (SLA); (b) executor count recovers to >= 8 within the chaos window (reprovisioning works); (c) no stage hits max consecutive attempts (retry storm didn't happen).

Output.

Time (s)	Event	Batch p99 (s)	Active executors	Max stage attempts
0	Chaos start	8	10	1
30	2 executors killed	12	8	1
60	Shuffle-fetch failures	22	8	2
90	Driver reprovisions	18	10	2
120	Job stable	10	10	2
EOT	Verification	9	10	2

Rule of thumb. Every Spark chaos experiment must ship with dynamicAllocation.enabled=true + blacklist.enabled=true + shuffle.service.enabled=true. Without the trio, executor loss produces either an under-provisioned job (no reprovisioning) or a retry storm (no blacklist) or a shuffle re-compute (no shuffle service).

Senior interview question on data-plane chaos

A senior interviewer might ask: "Design a game-day chaos scenario that covers your entire ingest stack — Kafka → Spark Structured Streaming → Iceberg — with one composed experiment that kills a broker, then loses an executor, then latency-injects S3. Include the halt criteria, the ordering, and how you'd staff the game day."

Solution Using a composed multi-step scenario with per-step data-SLI halts and a game-day runbook

# ChaosEngine — three sequential experiments, each with its own halt SLI
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: ingest-stack-game-day
  namespace: chaos
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
            - {name: TARGET_PODS, value: 'app=kafka-broker,role!=controller'}
        probe: &data_sli_probes
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:kafka_lag_messages'
              comparator: {type: int, criteria: '<', value: '150000'}

    # 2-minute gap between experiments — let the pipeline settle
    - name: pod-delete
      spec:
        components:
          env:
            - {name: RAMP_TIME, value: '120'}     # 2-min delay before this exp starts
            - {name: TOTAL_CHAOS_DURATION, value: '180'}
            - {name: PODS_AFFECTED_PERC, value: '20'}
            - {name: TARGET_PODS, value: 'app=spark-executor'}
        probe: *data_sli_probes

    - name: pod-network-latency
      spec:
        components:
          env:
            - {name: RAMP_TIME, value: '120'}
            - {name: TOTAL_CHAOS_DURATION, value: '300'}
            - {name: NETWORK_LATENCY, value: '500'}
            - {name: DESTINATION_HOSTS, value: 's3.us-east-1.amazonaws.com'}
            - {name: TARGET_PODS, value: 'app=iceberg-writer'}
        probe: *data_sli_probes

<!-- game-day-runbook.md — the human ceremony -->
# Ingest Stack Game Day — Runbook

**Duration.** 90 minutes (30 min setup + 30 min chaos + 30 min post-mortem)

**Roles.**
- Game Master (GM): drives the ceremony; runs the chaos apply command
- Observer 1 (Kafka): watches Kafka SLI dashboard; calls out anomalies
- Observer 2 (Spark): watches Spark SLI dashboard
- Observer 3 (Iceberg/S3): watches Iceberg + S3 dashboards
- Scribe: takes running notes into the post-mortem doc
- On-Call: shadows; can halt if real customer impact

**Pre-flight.**
- [ ] All observers connected to Zoom bridge
- [ ] All SLI dashboards open (freshness, lag, batch p99, iceberg-commit-p99)
- [ ] `#chaos-live` Slack channel posted with hypothesis
- [ ] Halt authority confirmed (any observer can shout "HALT" to trigger `kubectl delete chaosengine`)

**Sequence.**
- T+0 : GM applies `ChaosEngine`; observers narrate SLIs
- T+2 : Kafka broker killed (probes gate lag < 150k)
- T+4 : Kafka window ends; 2-min settle
- T+6 : Spark executors killed (probes gate batch p99)
- T+9 : Spark window ends; 2-min settle
- T+11: S3 latency injected (probes gate iceberg-commit-p99)
- T+16: S3 window ends
- T+18: Chaos complete; observers verify all SLIs recovered
- T+30: Scribe finalises post-mortem doc; GM assigns action items

Step-by-step trace.

Step	Component	Action	Halt criterion
1	ChaosEngine applied	kafka-broker-kill starts	lag < 150k, freshness < 15 min
2	120s chaos + settle	broker recovers	ISR heals
3	spark-executor-kill starts	2 executors killed	batch p99 < 30s
4	180s chaos + settle	executors reprovision	dynamic allocation
5	s3-latency starts	+500ms to S3	iceberg-commit-p99 < 30s
6	300s chaos ends	latency removed	commits catch up
7	GameDay doc	scribe finalises	action items filed

After the ceremony, the game-day doc lists every observed deviation, every halted probe (with reason), and every backlog item filed in Jira. The doc becomes the reference for the next ceremony; every recurring ceremony either graduates the blast radius or logs a resilience regression.

Output:

Artifact	Purpose	Consumer
ChaosEngine CRs (3 experiments)	scripted chaos	LitmusChaos operator
Data-SLI probes (2 per exp)	halt criteria	LitmusChaos runner
game-day-runbook.md	human ceremony script	GM + observers
Live SLI dashboards	real-time verdict	observers + on-call
Post-mortem doc	learning artifact	data-platform team
Jira backlog items	fix commitments	product + engineering

Why this works — concept by concept:

• Composed sequential experiments — three chaos primitives run in sequence with 2-min settle gaps. Each experiment exercises a different subsystem (broker, executor, external dependency) while the others get to recover. Sequence-with-gaps is safer than parallel — the pipeline never faces two novel failure modes at once.
• Shared data-SLI probe set (&data_sli_probes YAML anchor) — the freshness + lag guards are defined once and referenced by all three experiments. This keeps the halt criteria consistent across the ceremony; every experiment is judged by the same yardstick.
• Ramp time between experiments — RAMP_TIME: '120' delays each experiment start so the previous one has 2 minutes to fully settle. Without ramp gaps, back-to-back failures mask each other's SLI signal.
• Explicit game-day roles — GM, observers, scribe, on-call are named roles. This is the missing piece in most chaos programs: the ceremony has to have humans in the loop, or the automation looks like an unmanaged accident.
• Halt authority = any observer — any observer can shout "HALT" and trigger kubectl delete chaosengine ingest-stack-game-day. This is the human escape hatch that the automated probes can't provide (e.g. if a customer bug report comes in mid-ceremony).
• Post-mortem doc as source of truth — the doc lands in a Confluence space with a stable URL. Every game-day ceremony produces one; every subsequent ceremony references the prior; the compounding effect is a written history of your pipeline's resilience posture.
• Cost — 90 minutes of 5 engineers = ~7.5 engineer-hours per ceremony; quarterly cadence = 30 engineer-hours per year. Compared to the cost of one unplanned incident (~40+ engineer-hours + customer impact), the game day is a massive positive-ROI trade. O(1) per ceremony; O(quarterly cadence) per pipeline.

Streaming
Topic — streaming
Streaming problems on Kafka consumer resilience

Practice →

Event Topic — event-processing Event processing problems on exactly-once contracts

Practice →

---

5. Hypothesis-driven chaos loop — steady state → hypothesis → blast → verify

The four-step loop that turns chaos from a stunt into a repeatable engineering practice

The mental model in one line: blast radius data chaos becomes a practice (rather than a stunt) when every experiment cycles through four explicit stages — codify the steady state as a data SLI, state a falsifiable hypothesis about how the pipeline should behave under a specific failure, run the experiment inside a bounded blast radius with an auto-halt on SLI breach, and verify by comparing SLI curves before / during / after with a written post-mortem that either graduates the blast radius or produces a fix backlog item. Every mature chaos program lives inside this loop; every immature one skips one or more stages and produces "we killed something and it seemed fine" as its output.

The four stages, mapped onto concrete data-pipeline artifacts.

• Steady state. One or more Prometheus recording rules that define "the pipeline is healthy." For a Kafka → Spark → Iceberg stack: pipeline:freshness_p99_min, pipeline:kafka_lag_messages, pipeline:batch_success_rate_5m, pipeline:row_count_delta_1h. The SLI is data-first, infra-second.
• Hypothesis. A written, falsifiable prediction of the form "if X breaks, Y stays within Z." Example: "If we kill 1 of 6 Kafka brokers, freshness_p99_min stays under 15 and kafka_lag_messages stays under 100k for the 120s chaos window."
• Experiment. The chaos configuration that tests exactly the hypothesis — the smallest fault that exercises the failure mode, run inside the smallest blast radius that produces a signal, with halt-on-SLI-breach probes. The experiment is the ChaosEngine CR + probe set.
• Verify. Compare SLI curves before / during / after (Grafana overlay). Write the post-mortem doc: hypothesis, actual outcome, probe timeline, root cause of any deviation, decision — graduate blast radius, or file a fix backlog item, or repeat the experiment next cycle.

The graduation ladder — how blast radius escalates over calendar time.

• Stage 1 — dev. Nightly automated runs against a dev cluster. Blast radius = 100% of dev workloads (they're disposable). Failure = auto-file a Jira; no on-call impact. Goal: catch config errors before they touch shared environments.
• Stage 2 — staging. Weekly automated runs against staging. Blast radius = same percent as prod-target (17% of brokers, 20% of executors) so staging teaches you what prod will experience. Failure = Slack ping to data-platform-sre; no customer impact. Goal: validate the experiment's halt criteria against realistic-scale infra.
• Stage 3 — prod-1pct. Monthly ceremony in prod during business hours. Blast radius = 1 broker of 6, or 2 executors of 20, or 1% of writers. GM + observers required; formal game-day format. Goal: prove the SLO holds under real production load.
• Stage 4 — prod-10pct. Quarterly ceremony. Blast radius scales to 10% of tier-1 workloads. Executive sponsor required; on-call bridge open. Goal: verify the pipeline handles a "one AZ down" magnitude failure.
• Stage 5 — prod-100pct. Annual DR drill. Blast radius = full region outage or full-cluster loss. DR runbook + customer comms. Goal: annual DR compliance and executive-level confidence.

The written post-mortem template — what "verify" produces.

• Hypothesis (stated before the experiment). "If X breaks, Y stays within Z."
• Actual outcome. Screenshot of SLI dashboard with the chaos window highlighted; probe timeline extracted from ChaosResult.
• Deviation analysis. For each SLI, was the actual within predicted? If not, why? Was it a config bug, a code bug, or a genuine resilience gap?
• Decision. Graduate blast radius / repeat at same radius / halt this experiment class pending fix.
• Action items. Each deviation gets a Jira ticket with owner + ETA.

Common interview probes on the chaos loop.

• "How do you decide when to graduate blast radius?" — required answer: N green runs at the current stage (matrix), no red runs in the last M runs, peer-reviewed PR to promote.
• "What if a chaos experiment reveals a bug in production?" — halt-on-breach immediately, on-call takes over, file a P1 bug with the SLI evidence, roll back any partially-applied fault.
• "How do you write a falsifiable hypothesis?" — required answer: "if X breaks, Y stays within Z" with X, Y, Z all quantitative. Weak: "the pipeline is resilient." Strong: "freshness_p99_min stays < 15 for 120 s of pod-delete on 1 of 6 brokers."
• "What separates a chaos experiment from a load test?" — chaos tests unavailability failure modes; load tests capacity. Chaos gives you an SLO verdict under fault; load gives you a throughput ceiling under peak.

Worked example — writing a falsifiable chaos hypothesis

Detailed explanation. The hypothesis is the single most-skipped step in first-attempt chaos programs. Teams reach for a fault ("let's kill a broker") without writing down what should happen. Without a written hypothesis, "we ran chaos" has no verdict — you either observe "it seemed fine" (which is not verifiable) or "it broke" (which is not attributable). Every mature chaos ceremony writes the hypothesis in advance and pastes it into the game-day doc before the experiment runs.

• Falsifiable. The hypothesis must be a statement that concrete data can prove wrong. "The pipeline is resilient" is not falsifiable. "freshness_p99_min stays < 15 min for 120 s of pod-delete on 1 of 6 brokers" is.
• Quantitative. Every noun in the hypothesis must have a number. Not "some brokers" — "1 of 6." Not "low latency" — "< 15 min p99." Not "briefly" — "120 s."
• Bounded. The hypothesis includes an explicit time window and blast radius. Without bounds, "stays < 15 min" is trivially violable at some future time.
• Named SLIs. The hypothesis references specific Prometheus recording rules (pipeline:freshness_p99_min, pipeline:kafka_lag_messages) — not vague concepts.

Question. Write three chaos hypotheses for the ingest stack, and derive the halt criterion from each.

Input.

Hypothesis	Failure mode	SLI	Threshold	Duration	Blast radius
Kafka broker survives without lag spike	pod-delete	pipeline:kafka_lag_messages	< 100k	120s	1/6 brokers
Spark stays inside p99 under executor loss	pod-delete	spark_streaming_batch_p99_sec	< 30s	180s	2/10 executors
Iceberg commits queue but don't fail under S3 latency	pod-network-latency	iceberg_commit_seconds_p99	< 30s	300s	25% writers

Code.

# Hypothesis card — checked into Git alongside every ChaosEngine
# hypothesis-cards/kafka-broker-kill.yaml
apiVersion: pipecode.ai/v1
kind: ChaosHypothesis
metadata:
  name: kafka-broker-kill
spec:
  hypothesis: |
    IF one Kafka broker (of six) is deleted via pod-delete,
    THEN pipeline:kafka_lag_messages stays below 100,000
     AND pipeline:freshness_p99_min stays below 15 minutes
     AND min(kafka_topic_partition_in_sync_replica{topic="events"}) stays >= 2
    FOR the duration of the 120-second chaos window.

  falsifiable_by:
    - metric: 'pipeline:kafka_lag_messages'
      condition: 'exceeds 100000 at any 10-second sample'
    - metric: 'pipeline:freshness_p99_min'
      condition: 'exceeds 15 at any 10-second sample'
    - metric: 'min(kafka_topic_partition_in_sync_replica{topic="events"})'
      condition: 'falls below 2 at any 15-second sample'

  chaosengine_ref: chaos/base/engines/kafka-broker-kill.yaml

  owner: data-platform-sre
  reviewers: [alice, bob]

# hypothesis_check.py — a CI script that verifies every ChaosEngine has a hypothesis card
import glob, yaml, sys

engines = glob.glob("chaos/base/engines/*.yaml")
cards   = glob.glob("chaos/hypothesis-cards/*.yaml")

engine_names = {yaml.safe_load(open(f))["metadata"]["name"] for f in engines}
card_refs = {yaml.safe_load(open(c))["spec"]["chaosengine_ref"].split("/")[-1].split(".")[0]
             for c in cards}

missing = engine_names - card_refs
if missing:
    print(f"ERROR: ChaosEngines without hypothesis cards: {missing}")
    sys.exit(1)

# Verify each card has quantitative falsifiability
for c in cards:
    card = yaml.safe_load(open(c))
    fbs = card["spec"]["falsifiable_by"]
    for fb in fbs:
        cond = fb["condition"]
        if "some" in cond.lower() or "few" in cond.lower():
            print(f"ERROR: non-quantitative condition in {c}: {cond}")
            sys.exit(1)

print("All ChaosEngines have quantitative hypothesis cards.")

Step-by-step explanation.

1. The ChaosHypothesis CR is a documentation artifact — it doesn't drive execution, it drives review. Every ChaosEngine in Git must have a corresponding hypothesis card; the CI script enforces this.
2. The hypothesis text uses IF ... THEN ... AND ... FOR ... grammar. This isn't arbitrary — it forces the author to name the failure, name the SLI(s), name the threshold(s), and name the duration. Any missing piece becomes obviously wrong on review.
3. The falsifiable_by block enumerates the exact conditions that would falsify the hypothesis. Each metric + condition pair maps 1:1 to a promProbe in the ChaosEngine — if the two get out of sync, the experiment can pass while the hypothesis is falsified (a silent bug).
4. The CI check catches two failure modes: (a) ChaosEngine without a hypothesis card (execution without prediction), (b) hypothesis card with vague language ("some", "few", "briefly") — quantitative discipline enforced at commit time.
5. The reviewers field is not just administrative — chaos hypotheses need domain review. Killing a Kafka broker without ISR knowledge, or restarting the Airflow scheduler without catchup knowledge, is where teams ship dangerous experiments. Two reviewers is the norm; the more senior one should have shipped the specific system before.

Output.

Artifact	Purpose	Gate
ChaosHypothesis CR	falsifiable prediction	code review
CI hypothesis_check.py	every engine has a card	pre-merge
Grammar IF ... THEN ... FOR ...	forces quantitative discipline	review-time
falsifiable_by block	maps 1:1 to probes	consistency check
Two reviewers	domain expertise	governance

Rule of thumb. No ChaosEngine ships to any environment without a hypothesis card. The hypothesis is the falsifiable prediction; the probes enforce the halt; the post-mortem records the outcome. Skip the hypothesis and you're not doing chaos engineering — you're doing chaos.

Worked example — the quarterly ingest-stack game day

Detailed explanation. The quarterly game day is where the chaos loop scales beyond one experiment. Teams gather for 90 minutes, run a pre-planned sequence of experiments against the whole ingest stack, watch the SLI dashboards live, and produce one post-mortem covering the ceremony. The ceremony has a fixed rhythm: 30-min pre-flight, 30-min chaos, 30-min post-mortem. Walk through the artifacts and cadence.

• Attendees. GM (drives), 3 Observers (Kafka / Spark / Iceberg), Scribe, On-Call (shadow with halt authority), optional Product owner (learns).
• Environment. Prod-1pct (early quarters) → Prod-10pct (later quarters). Never dev — dev doesn't produce actionable learning.
• Pre-flight. SLI dashboards open, #chaos-live Slack post, hypothesis cards printed, halt authority confirmed.
• Sequence. Three experiments (broker kill, executor loss, S3 latency) with 2-min settle between.
• Post-mortem. Written same day; action items assigned; blast-radius graduation decision recorded.

Question. Design the quarterly ceremony agenda, the responsibilities of each role, and the post-mortem template.

Input.

Role	Responsibility
GM	apply ChaosEngine; drive tempo; call "next experiment"
Observer 1 (Kafka)	watch broker + lag + ISR panels; call anomalies
Observer 2 (Spark)	watch executor count + batch p99 + stage retries
Observer 3 (Iceberg)	watch commit p99 + write error rate + manifest count
Scribe	live-take notes into post-mortem doc
On-Call	shadow; halt authority; watches customer impact
Product	learns; represents customer perspective

Code.

<!-- game-day/2026-Q3-ingest-stack.md -->
# Ingest Stack Game Day — 2026-Q3

**Date.** 2026-07-30 14:00 - 15:30 UTC
**Environment.** prod-10pct
**GM.** Alice Chen (data-platform-sre)
**Halt authority.** Any observer via `#chaos-live` "HALT" message → GM runs `kubectl delete chaosengine`

## Pre-flight checklist (14:00 - 14:30)

- [ ] All observers on Zoom bridge
- [ ] SLI dashboards open in tabs:
  - [https://grafana/pipeline-freshness]
  - [https://grafana/kafka-brokers]
  - [https://grafana/spark-executors]
  - [https://grafana/iceberg-commits]
- [ ] `#chaos-live` posted with ceremony agenda + hypothesis cards
- [ ] `kubectl` context confirmed = `prod-10pct`
- [ ] Halt authority explicitly re-confirmed on the bridge
- [ ] Customer status page monitored (green expected)

## Experiment 1 — Kafka broker kill (14:30 - 14:35)

**Hypothesis.** IF one broker of six is deleted, THEN kafka_lag_messages stays < 100k AND freshness_p99_min stays < 15 AND min ISR stays >= 2 FOR 120s.

**Apply.** `kubectl apply -f chaos/overlays/prod-10pct/kafka-broker-kill.yaml`

**Observations.** (Scribe fills in live)
- Lag peak: ___
- Freshness peak: ___
- ISR minimum: ___
- Halt triggered? ___

**Verdict.** [ ] Pass  [ ] Fail  [ ] Halted

## Experiment 2 — Spark executor loss (14:38 - 14:45)

**Hypothesis.** IF 2 of 10 executors are deleted, THEN batch_p99 stays < 30s AND executor_count recovers to >= 8 within 60s AND max stage attempts stays < 4 FOR 180s.

...

## Experiment 3 — S3 latency (14:48 - 14:58)

...

## Post-mortem (15:00 - 15:30)

**Overall verdict.** ___ / 3 experiments Pass

**Deviations.**
| Experiment | Predicted | Actual | Root cause | Action item |
|---|---|---|---|---|
| ... | ... | ... | ... | ... |

**Blast-radius decisions.**
- Kafka broker kill: [ ] graduate to prod-100pct  [ ] repeat at prod-10pct  [ ] rollback to prod-1pct
- Spark executor loss: [ ] ...
- S3 latency: [ ] ...

**Action items (owner + ETA).**
1. ...
2. ...

**Follow-up date.** 2026-10-30 (next quarterly)

Step-by-step explanation.

1. The 90-min ceremony has a fixed shape: 30-min pre-flight, 30-min chaos, 30-min post-mortem. Deviating from the timing produces either "not enough setup" or "not enough learning" — the timings are load-bearing.
2. Each experiment has a printed hypothesis card that pastes verbatim into the doc; the observations table is filled in live by the scribe. This makes the ceremony reproducible — anyone reading the doc a year later can retrace what was tried and what happened.
3. The halt authority is deliberately human ("any observer via #chaos-live HALT message"). The automated probes catch SLI breaches; the humans catch business-impact signals that no probe covers (a support ticket, a Twitter mention, a customer email).
4. The post-mortem table forces attribution: for each deviation, name the root cause (config bug / code bug / genuine resilience gap) and file an action item. Without the table, deviations get forgotten.
5. The blast-radius decisions are the promotion signal for the next quarter's ceremony. Each experiment either graduates (next quarter runs at bigger radius), repeats (same radius, verify stability), or rolls back (fix a bug, retry at smaller radius). This is what makes chaos a compounding practice.

Output.

Artifact	Purpose	Cadence
Pre-flight checklist	operator readiness	per ceremony
Per-experiment hypothesis + observations	falsifiable outcome	per experiment
Post-mortem deviation table	attributed learning	per ceremony
Blast-radius decision matrix	graduation signal	per ceremony
Action items with owner + ETA	fix commitments	per deviation
Follow-up date	compounding cadence	per ceremony

Rule of thumb. Every game day produces exactly one written post-mortem, filed in a stable location, with one row per experiment and one action item per deviation. The doc is the compounding artifact — after 8 quarters you have 8 post-mortems that collectively tell the story of your pipeline's resilience trajectory.

Worked example — the "verify" step — SLI-before / SLI-during / SLI-after overlay

Detailed explanation. The verify step is where the hypothesis meets the data. Grafana overlays the SLI curves from a baseline (24h prior to chaos), the chaos window, and the post-chaos period. Deviations pop visually; agreement is boring (which is what you want). Every game-day doc includes screenshots of the overlays as the "here's what actually happened" evidence.

• Baseline. Last 24h of the SLI, same window-of-day, same day-of-week.
• Chaos. The exact chaos window with the ChaosEngine start / end times marked as vertical lines.
• Post-chaos. 30 min after chaos ends — verify recovery is complete.
• Overlay. Grafana time_range templating shows all three periods on the same panel.

Question. Design the Grafana panel that provides the verify-step evidence for a Kafka broker kill.

Input.

Panel	Metric	Overlay	Purpose
Consumer lag	sum(kafka_consumergroup_lag)	baseline vs chaos vs post	freshness verdict
Producer errors	sum(rate(kafka_producer_record_error_total[1m]))	baseline vs chaos	durability verdict
Min ISR	min(kafka_topic_partition_in_sync_replica)	chaos-only, annotated	correctness verdict
Broker count ready	count(kube_pod_status_ready{pod=~"kafka-broker.*",condition="true"})	chaos-only	infra recovery

Code.

// Grafana panel — Consumer lag with baseline overlay
{
  "type": "timeseries",
  "title": "Consumer lag — chaos overlay",
  "targets": [
    {
      "refId": "A",
      "expr": "sum(kafka_consumergroup_lag{group=\"warehouse-sink\"})",
      "legendFormat": "chaos",
      "datasource": "prometheus"
    },
    {
      "refId": "B",
      "expr": "sum(kafka_consumergroup_lag{group=\"warehouse-sink\"} offset 24h)",
      "legendFormat": "baseline (24h ago)",
      "datasource": "prometheus"
    }
  ],
  "options": {
    "legend": {"placement": "bottom"},
    "tooltip": {"mode": "multi"}
  },
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green",  "value": null},
          {"color": "yellow", "value": 50000},
          {"color": "red",    "value": 100000}
        ]
      }
    }
  },
  "annotations": {
    "list": [
      {
        "name": "chaos-window",
        "datasource": "prometheus",
        "expr": "ALERTS{alertname=\"ChaosEngineActive\", chaosengine=\"kafka-broker-kill\"}",
        "iconColor": "purple",
        "titleFormat": "chaos active",
        "tagKeys": "chaosengine"
      }
    ]
  }
}

# verify_generate.py — auto-generate the Grafana-URL for the verify step
# runs at ChaosResult completion; posts the URL to Slack + post-mortem doc
import subprocess, json, urllib.parse, requests

def get_chaosresult(name, ns):
    r = subprocess.check_output(["kubectl", "-n", ns, "get", "chaosresult",
                                  name, "-o", "json"])
    return json.loads(r)

def build_grafana_url(dashboard_uid, start_epoch_ms, end_epoch_ms):
    base = "https://grafana.internal/d/" + dashboard_uid
    # 30 min pre-chaos, chaos window, 30 min post-chaos
    from_ms = start_epoch_ms - 30*60*1000
    to_ms   = end_epoch_ms   + 30*60*1000
    qs = urllib.parse.urlencode({
        "from": from_ms,
        "to":   to_ms,
        "var-chaosengine": "kafka-broker-kill",
    })
    return f"{base}?{qs}"

def post_verify_link(chaosresult):
    exp   = chaosresult["status"]["experimentStatus"]
    name  = chaosresult["metadata"]["name"]
    start = int(exp["startTimestamp"])
    end   = int(exp["endTimestamp"])
    url = build_grafana_url("ingest-chaos-overlay", start*1000, end*1000)
    requests.post("https://hooks.slack.com/services/xxx", json={
        "text": f":mag: Verify-step overlay for {name}: {url}"
    })

if __name__ == "__main__":
    cr = get_chaosresult("kafka-broker-kill-pod-delete", "kafka")
    post_verify_link(cr)

Step-by-step explanation.

1. The Grafana panel plots two Prometheus series on the same y-axis: kafka_consumergroup_lag (chaos) and ... offset 24h (baseline). The offset shifts the query 24h back so we compare "now" to "same time yesterday" — same window-of-day traffic pattern.
2. The chaos-window annotation is driven by a Prometheus alert ChaosEngineActive fired by an ancillary exporter that watches for active ChaosEngine CRs. When the CR is active, the alert is firing, and Grafana overlays a shaded region on the panel.
3. The verify_generate.py script watches ChaosResult CRs; when one completes, it extracts start/end timestamps and generates a Grafana URL scoped to the chaos window ± 30 min. The URL posts to #chaos-live and lands in the post-mortem doc.
4. The 30-min windows before/after the chaos give visual context — you can see the baseline SLI curve, the chaos-induced deviation, and the recovery back to baseline. The recovery segment is often more informative than the chaos segment; a slow recovery means the pipeline was more fragile than the point-in-time SLI suggested.
5. The three-color threshold (green/yellow/red) on the y-axis gives an at-a-glance verdict — if the chaos curve stays in green throughout, hypothesis confirmed; if it dips into red, hypothesis falsified. The threshold values (50k warn, 100k halt) match the promProbe values.

Output.

Panel component	Purpose
Chaos series (query A)	actual behaviour under fault
Baseline series (offset 24h)	what "normal" looks like
Chaos-window annotation	visual scope of the experiment
Threshold bands	SLI verdict at a glance
Auto-generated URL	shareable evidence for post-mortem
Slack post	live operator awareness

Rule of thumb. Every chaos experiment ships with a paired Grafana panel that overlays chaos-vs-baseline SLIs with the chaos window annotated. The panel URL lands in the post-mortem doc as the "verify" evidence. Without the overlay, "verify" degenerates into "we watched the dashboard and it seemed fine."

Senior interview question on the chaos loop

A senior interviewer might ask: "Design the chaos-engineering practice for a new data platform team from scratch. Include the SLI catalog, the hypothesis format, the graduation ladder, the game-day ceremony, the tooling choice (LitmusChaos vs Gremlin), and how you'd measure the practice's own maturity over time."

Solution Using an SLI-first hypothesis-driven program with quarterly ceremonies and a maturity ladder

# chaos_maturity.py — score the practice itself, quarterly
# runs against the chaos GitOps repo + ChaosResult history

CRITERIA = [
    ("SLI catalog exists",           "grep -r 'record: pipeline:' chaos/base/_slis.yaml"),
    ("Every engine has a hypothesis card", "python hypothesis_check.py"),
    ("Every engine has continuous probes", "python probe_check.py"),
    ("Every hypothesis card is falsifiable", "python falsifiable_check.py"),
    ("Nightly dev chaos runs",       "kubectl get cronjob chaos-dev -n chaos"),
    ("Weekly staging chaos runs",    "kubectl get cronjob chaos-staging -n chaos"),
    ("Monthly prod-1pct ceremony",   "gh api repos/acme/chaos/actions/workflows/prod-1pct.yml"),
    ("Quarterly prod-10pct ceremony", "gh api repos/acme/chaos/actions/workflows/prod-10pct.yml"),
    ("Post-mortem doc per ceremony", "ls game-day/*.md | wc -l"),
    ("Action-item follow-through > 80%", "python action_item_check.py --window 90d"),
    ("Blast-radius graduation events > 0", "python graduation_check.py --window 90d"),
    ("SIEM audit trail",             "curl -s splunk-hec/health"),
]

def score():
    passed, total = 0, len(CRITERIA)
    for name, cmd in CRITERIA:
        try:
            subprocess.check_output(cmd, shell=True, stderr=subprocess.DEVNULL)
            passed += 1
            print(f"  [x] {name}")
        except subprocess.CalledProcessError:
            print(f"  [ ] {name}")
    print(f"\nMaturity: {passed}/{total}")
    return passed, total

if __name__ == "__main__":
    score()

# CronJob — the practice's own heartbeat
apiVersion: batch/v1
kind: CronJob
metadata:
  name: chaos-maturity-score
  namespace: chaos
spec:
  schedule: '0 9 1 */3 *'   # quarterly, 9am on the 1st
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: score
              image: acme/chaos-maturity:latest
              command: ['python', '/app/chaos_maturity.py']
          restartPolicy: OnFailure

Step-by-step trace.

Layer	Component	Responsibility
SLI catalog	_slis.yaml	one recording rule per pipeline SLI
Hypothesis cards	hypothesis-cards/*.yaml	one per ChaosEngine; enforced by CI
ChaosEngine CRs	engines/*.yaml	one per experiment; probes reference SLIs
Blast-radius overlays	overlays/{dev,staging,prod-*}/	per-stage promotion
Game-day doc	game-day/*.md	one per ceremony; scribe live-writes
CI checks	hypothesis_check.py, probe_check.py, falsifiable_check.py	pre-merge gates
Maturity CronJob	quarterly	scores the practice itself
SIEM export	siem_exporter.py	attack audit trail

After deployment, the practice grades itself quarterly. A team scoring 12/12 has a mature chaos practice: SLI catalog, hypothesis-driven experiments, probe-gated halt, graduated blast radius, written post-mortems, follow-through on action items, audit trail. A team scoring 4/12 has the tools but not the discipline — usually because they skipped the hypothesis step and can't produce a verdict.

Output:

Practice level	Score	Meaning
0-3	Absent	no chaos program
4-6	Tooling-only	ran experiments; no discipline
7-9	Hypothesis-driven	SLIs + hypotheses + halts; no ceremony
10-11	Ceremony-driven	quarterly game days + post-mortems
12	Compounding	maturity score itself is monitored

Why this works — concept by concept:

• SLI catalog before any experiment — the practice starts by defining what "healthy" means. The catalog is the single source of truth referenced by every probe and every hypothesis card. Without it, each experiment invents its own criteria and comparability dies.
• Hypothesis-driven experiments — every experiment ships with a falsifiable prediction. CI enforces the pairing (hypothesis_check.py), the grammar (falsifiable_check.py), and the probe consistency (probe_check.py). Chaos becomes engineering, not stunts.
• Graduation ladder with matrix-gated promotion — dev → staging → prod-1pct → prod-10pct → prod-100pct, with N green runs required at each stage. CI enforces the counts; CODEOWNERS enforces the human approval. Prod chaos is earned, not granted.
• Quarterly game-day ceremony — the human ritual that reads the SLI evidence, discusses the deviations, and decides the next quarter's blast-radius adjustments. Ceremony frequency is the practice's heartbeat; skipping ceremonies collapses the compounding.
• Written post-mortem per ceremony — the doc is the compounding artifact. After 8 quarters you have 8 post-mortems that collectively tell the story of your pipeline's resilience trajectory. New team members read the docs to onboard.
• Action-item follow-through as an SLI — the practice measures whether its own action items are being closed. A team that files 20 action items and closes 3 is doing theatre; a team that files 8 and closes 7 is doing engineering. The 80% threshold in chaos_maturity.py is the guard.
• Maturity score as a monitored SLI — the practice grades itself. When maturity drops (a ceremony was skipped, a check regressed), the CronJob output triggers a Slack alert. The practice's own health becomes a metric on the same dashboard as pipeline health.
• Cost — one quarterly ceremony (7.5 engineer-hours) + nightly/weekly automation (~0 marginal ops) + one CI check bundle (~5 minutes per PR) + one CronJob (~1 minute per quarter). Compared to the cost of one unplanned incident, this is a 100× positive-ROI trade. O(1) per ceremony; O(quarterly) per pipeline; O(annual) per program review.

Real-time
Topic — real-time-analytics
Real-time analytics problems on SLI-first design

Practice →

Data-Validation
Topic — data-validation
Data validation problems on post-chaos correctness

Practice →

---

Cheat sheet — chaos engineering data recipes

• Which tool when. LitmusChaos is the 2026 default when your data plane is Kubernetes-native and you want everything GitOps-managed — free, CNCF, ChaosEngine CR + probe guardrails + ChaosResult verdict. Gremlin is the choice when your data plane spans VMs / on-prem / ECS / K8s and you want one SaaS control plane with SOC-2 audit, richer network/state primitives (DNS, time-travel, blackhole), and a Terraform provider. Pick both if your platform has both surfaces — LitmusChaos for K8s workloads, Gremlin for cross-platform infra chaos.
• Steady-state SLI catalog template. Define four Prometheus recording rules per pipeline: pipeline:freshness_p99_min (histogram_quantile of commit_age_seconds / 60), pipeline:kafka_lag_messages (sum of consumer group lag), pipeline:batch_success_rate_5m (success / total ratio over 5-min window), pipeline:row_count_delta_1h (source − sink parity). Every probe on every ChaosEngine references these — one source of truth for "the pipeline is healthy."
• Halt-condition template. Halt criterion = promProbe in Continuous mode with stopOnFailure: true, threshold set 2-3× tighter than the customer-facing SLO (so chaos aborts before customers notice), probeTimeout: 5, interval: 10, retry: 1 (short retry — one flaky query shouldn't abort). Pair with an EOT cmdProbe for correctness invariants Prometheus can't express (row counts, orphaned rows, manifest integrity).
• LitmusChaos ChaosEngine template. apiVersion: litmuschaos.io/v1alpha1, spec.chaosServiceAccount: litmus-admin, spec.appinfo.applabel selector, experiments[0].name: pod-delete (or pod-network-loss, disk-fill, pod-cpu-hog), env TOTAL_CHAOS_DURATION + PODS_AFFECTED_PERC + FORCE: 'false' (graceful), probe[] with 2-3 promProbe + optional k8sProbe + optional cmdProbe. RBAC-scope the runner service account to the target namespace only.
• Gremlin scenario YAML template. scenario.name + hypothesis (falsifiable), graph.nodes.{start,attack,cooldown} with InfraAttack type + impactDefinition (kind, attackType, params, percent blast radius, targets.tags selector), healthChecks[] with provider: datadog|webhook|newrelic|prometheus and haltOnUnhealthy: true. Version via Terraform gremlin_scenario resource; mirror every attack to SIEM via the API for audit.
• Kafka broker chaos guardrails. Topic: replication.factor >= 3, min.insync.replicas = 2, unclean.leader.election.enable = false. Producer: acks = all, enable.idempotence = true, retries = Integer.MAX_VALUE, max.in.flight.requests.per.connection <= 5. Chaos: PODS_AFFECTED_PERC <= 17 (1 of 6), FORCE: 'false', halt on kafka_consumergroup_lag > 100000 OR min(in_sync_replica) < 2 OR rate(kafka_producer_record_error_total[1m]) > 0.1.
• Airflow scheduler chaos guardrails. DAG: catchup=False, max_active_runs=1, retries >= 2, dagrun_timeout set. Chaos: pod-delete on component=scheduler with FORCE: 'false'. Halt on k8sProbe on scheduler Deployment ready + EOT cmdProbe SQL SELECT count(*) FROM task_instance WHERE state='running' AND queued_dttm < now() - INTERVAL '10 minutes' = 0 (orphan check) + EOT promProbe on airflow_dagrun_active_count <= max_active_runs (no catchup flood).
• Spark executor chaos guardrails. Spark submit: dynamicAllocation.enabled=true, dynamicAllocation.minExecutors=<floor>, blacklist.enabled=true, blacklist.timeout=1h, task.maxFailures=4, stage.maxConsecutiveAttempts=4, shuffle.service.enabled=true (external shuffle service DaemonSet or NodeManager aux). Chaos: PODS_AFFECTED_PERC <= 20 on app=spark-executor. Halt on batch_processing_p99 > 30s + EOT check active_executor_count >= minExecutors + max_stage_consecutive_attempts < 4.
• Blast-radius graduation matrix. Dev = 100% (disposable), 5 green runs to promote; Staging = same percent as prod-target (17% brokers, 20% executors, 25% writers), 3 green weekly runs to promote; Prod-1pct = 1 broker of 6 / 2 executors of 20 / 1% writers, monthly, 3 green ceremonies to promote; Prod-10pct = 10% of tier-1, quarterly, 2 green ceremonies to promote; Prod-100pct = annual DR drill only. CI script counts green runs; CODEOWNERS enforces human approval per stage.
• Game-day ceremony script. 90 minutes total: 30-min pre-flight (SLI dashboards open, #chaos-live posted, halt authority re-confirmed, kubectl context verified) → 30-min chaos (three composed experiments with 2-min settle gaps, scribe live-writes observations, any observer can shout HALT) → 30-min post-mortem (deviation table, root cause per deviation, blast-radius decisions, action items with owner + ETA, follow-up date). Doc lives in Confluence at a stable URL.
• Hypothesis card template. IF <fault at blast radius X> THEN <SLI Y> stays <comparator threshold Z> AND <SLI Y'> stays <comparator threshold Z'> FOR <duration>. Every noun quantitative. Every SLI a Prometheus rule reference. Every falsifiable_by entry maps 1:1 to a promProbe. Two named reviewers, one with production experience on the specific system.
• SIEM audit trail. Every chaos attack (LitmusChaos ChaosResult, Gremlin attack event) mirrors to your SIEM (Splunk, Datadog, Elastic) with fields: {attack_id, scenario, user, targets, tags, started_at, ended_at, halted, verdict, probe_timeline}. Retention: 1 year minimum (audit typical); 7 years for SOC-2 environments. Alert on any prod-scope chaos with no matching approval PR — that's an unauthorised attack.
• Post-mortem action-item follow-through. Track action-item close-rate as a chaos-practice SLI: (closed_within_90d / total_filed) > 80% is healthy; < 50% is theatre. Report the ratio in the quarterly maturity score; regress the ratio into the team's OKRs. An unclosed action item is a resilience gap that will surface as a real incident.
• Chaos maturity ladder. 0-3 criteria = absent (no chaos program); 4-6 = tooling-only (running experiments, no discipline); 7-9 = hypothesis-driven (SLIs + hypotheses + halts, no ceremony); 10-11 = ceremony-driven (quarterly game days + written post-mortems); 12 = compounding (maturity score itself monitored, action-item follow-through > 80%, blast radius graduating quarter over quarter). Score the practice quarterly; target one level per year.

Frequently asked questions

What is chaos engineering for data pipelines in one sentence?

Chaos engineering for data pipelines is the discipline of deliberately injecting infrastructure failures — Kafka broker kills, Airflow scheduler restarts, Spark executor loss, S3 latency spikes, disk fills, network partitions — into a running pipeline while a codified data SLO (freshness_p99, end-to-end lag, DAG success rate, source-to-sink row parity) acts as the steady state that decides whether the experiment passes or halts, all inside a bounded blast radius that graduates from dev → staging → 1% prod → 10% prod → 100% prod. The four canonical tools in 2026 — litmuschaos (K8s-native, open source, ChaosEngine + probe CRs), gremlin (SaaS, cross-platform, scenarios + blast-radius dial + halt-on-breach), plus the data-plane primitives (kafka chaos broker kill, airflow chaos scheduler kill, spark chaos executor loss) — differ in deployment surface and fault library but converge on the same four-step loop: steady state → hypothesis → experiment → verify. Getting the axes right (data-first SLI, falsifiable hypothesis, halt-on-breach probe, written post-mortem) is what separates a mature practice from a stunt, and it's the load-bearing skill senior data platform interviewers probe.

LitmusChaos vs Gremlin — when do I pick each?

Default to LitmusChaos when your data plane runs entirely on Kubernetes and you want everything GitOps-managed — it's CNCF-graduated, Apache-2.0, and models chaos as first-class Kubernetes custom resources (ChaosEngine, ChaosExperiment, ChaosResult) with a probe system (httpProbe, promProbe, cmdProbe, k8sProbe) that auto-aborts on breach. Ship it via Argo CD, RBAC-scope the runner to the target namespace, and the entire experiment lives in Git. Pick Gremlin when your data plane spans VMs, on-prem, ECS, Databricks, mixed cloud — a single SaaS control plane with lightweight gremlind agents on every target host, richer network/state attack primitives (DNS, blackhole, time-travel), and SOC-2-friendly SIEM-exportable audit logs. Gremlin is commercial (per-agent-per-month licence), so the trade is money-for-time-to-value vs LitmusChaos's ops-cost-for-flexibility. Larger platforms often run both — LitmusChaos for the K8s data plane, Gremlin for the cross-platform infra chaos — with shared Datadog / Prometheus SLIs as the common halt criteria.

How do I define steady state for a data pipeline?

Steady state must be a data SLO, not an infra metric. For a Kafka → Spark Structured Streaming → Iceberg → warehouse stack, four Prometheus recording rules define steady state: pipeline:freshness_p99_min (99th percentile age in minutes of the newest committed record vs wall clock), pipeline:kafka_lag_messages (sum of consumer group lag across all partitions), pipeline:batch_success_rate_5m (successful streaming batches / total batches over a 5-min window), and pipeline:row_count_delta_1h (source events count − sink rows count over the last hour). Every chaos probe references these — no pod_ready, no cpu_percent < 80, because a pod can be Ready while the pipeline is 40 minutes behind. Set halt thresholds 2-3× tighter than the customer-facing SLO (chaos aborts before customers notice) and pair Continuous promProbe guards with EOT cmdProbe checks for correctness invariants Prometheus can't express (orphaned task_instance rows, Iceberg manifest integrity, tombstone parity). If you can't articulate "the pipeline is healthy" in a single Prometheus query, you're not ready to inject chaos.

What's a safe blast radius for a Kafka broker kill?

The 2026 answer is one broker at a time, scoped via PODS_AFFECTED_PERC: '17' on a 6-broker cluster (or the equivalent selector on your topology), with a mandatory topic contract of replication.factor >= 3 + min.insync.replicas = 2 + unclean.leader.election.enable = false and a producer contract of acks = all + enable.idempotence = true. Under those settings, killing one broker leaves each affected partition with 2 in-sync replicas — enough to satisfy min.insync.replicas and continue accepting writes; the controller re-elects a new leader from the remaining ISR within seconds and the producer's retry loop absorbs the transient errors. Halt on kafka_consumergroup_lag > 100000 OR min(in_sync_replica) < 2 OR rate(kafka_producer_record_error_total[1m]) > 0.1 — any of these means the guardrails failed and further chaos will lose writes. Never kill the controller in the same experiment (label the controller pod separately and exclude via applabel: 'app=kafka-broker,role!=controller'); controller loss is a distinct experiment with different SLIs (controller failover time SLO).

Should I run chaos engineering in production?

Yes, but only after the graduation ladder is climbed and the halt criteria are proven. The mature 2026 pattern runs chaos through dev → staging → prod-1pct → prod-10pct → prod-100pct with N green runs required at each stage before promotion (5, 3, 3, 2, 1 respectively) and CI-gated + CODEOWNERS-approved PRs promoting each experiment between stages. Prod-1pct chaos is a monthly ceremony in a game-day format (GM, observers, scribe, on-call with halt authority); prod-10pct is quarterly with an executive sponsor; prod-100pct is reserved for annual DR drills with a formal runbook and customer comms. Every prod experiment ships with stopOnFailure: true promProbe guards tied to data SLIs 2-3× tighter than customer-facing SLOs and mirrors every attack to your SIEM (Splunk, Datadog, Elastic) for audit. The alternative to production chaos is not zero chaos — it's unplanned chaos when a real incident hits an untested failure mode. Scheduled, bounded, halted chaos is quantifiably safer than the incident lottery.

How is chaos engineering different from load testing?

Load testing measures capacity — how many requests per second, how many concurrent users, how big a batch — the pipeline can handle before latency/throughput degrades. Chaos engineering measures availability — whether the pipeline stays inside its SLO when a specific infrastructure component fails. Both are essential and both are complementary, but they answer different questions: load testing answers "how much load survives?" and chaos answers "which failure modes survive?" Load tests typically run in a dedicated environment with synthetic traffic; chaos experiments typically run in real environments with real traffic and inject unavailability rather than volume. A pipeline can pass a load test at 10× peak and still fail a chaos experiment because a Kafka broker kill exposes a min.insync.replicas misconfig that only surfaces during failure. In the 2026 senior interview, describing chaos as "load testing but for failures" is a weak answer; describing it as "hypothesis-driven falsification of resilience assumptions under bounded infrastructure faults with SLO-tied halt criteria" is a senior signal.

Practice on PipeCode

• Drill the streaming practice library → for the Kafka broker chaos, ISR contract, and consumer-lag-under-failure problems senior interviewers love.
• Rehearse on the ETL practice library → for the Airflow scheduler chaos, DAG catchup, and orphaned task_instance recovery patterns.
• Sharpen the architecture axis with the design practice library → for graduated blast radius, game-day ceremony design, and SLO-first steady-state selection.
• Practise the event-processing library → for exactly-once contracts, coordinator loss, and idempotent consumer patterns under chaos.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-step chaos loop against real graded inputs.

Lock in chaos engineering muscle memory

Docs explain chaos tools. PipeCode drills explain the decision — when a `promProbe` should halt, when `min.insync.replicas` earns its config line, when Airflow's `catchup=False` prevents a queue flood, when Spark's `dynamicAllocation` saves an executor-loss experiment, when a hypothesis is falsifiable versus vague. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data platform engineers and SREs actually face.

Practice streaming problems →
Practice design problems →