DEV Community: Gowtham Potureddi

From Data Engineer to Data Architect: Skill Ladder, Certifications & Interview Signals

Gowtham Potureddi — Sat, 01 Aug 2026 13:35:22 +0000

The move data engineer to data architect is the single most misunderstood step on the data career ladder, because it looks like a promotion inside the same job when it is actually a change of job entirely — the day you become an architect you stop being measured on the pipelines you ship and start being measured on the decisions other people ship because of you. A strong data engineer is graded on throughput: DAGs that run green, latency that stays under SLA, tables that land on time, bugs that get fixed fast. A data architect is graded on judgement: whether the blueprint they drew survives three years of scaling, five business units, two acquisitions, and a governance audit — and whether the twelve engineers building against that blueprint could do their jobs without asking the architect what to do next. The gap between those two grading rubrics is the whole subject of this guide, and it is why so many excellent engineers stall at the boundary: they keep sharpening the skill that got them here instead of building the skill ladder the new role demands.

That new ladder has five load-bearing rungs — data modeling, governance and security, integration patterns, cost and performance, and cloud platforms — sitting under a sixth skill that ties them together: the ability to explain a trade-off out loud, in numbers, to a room that includes a CFO and a compliance officer as well as engineers. This walkthrough is the senior-DE map for climbing all six: what actually changes when the mandate shifts from implementation to blueprint, which data architect skills are worth building first and which are noise, which data architect certification paths genuinely move the needle versus which are résumé decoration, how the architecture design interview is scored (and why "governance-first" beats "clever" almost every time), and a concrete 18-month transition roadmap that turns a good data engineer into a credible enterprise architecture hire. Each section pairs a teaching block with a worked interview answer — code or a text template, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the design practice library →, build modeling muscle on the dimensional-modeling practice library →, and keep your query foundations sharp with the SQL practice library →.

On this page

DE vs Data Architect: what actually changes
The architect skill ladder: modeling, governance, cost
Certifications that actually move the needle
Architecture interviews: blueprints & trade-offs
Signals architects are graded on & the transition plan
Cheat sheet — DE → architect recipes
Frequently asked questions
Practice on PipeCode

1. DE vs Data Architect: what actually changes

The mandate shift from implementation to blueprint — you stop owning the DAG and start owning the decision

The one-sentence invariant: the data engineer owns the implementation and the data architect owns the blueprint — the architect is accountable for the shape of the system, the standards the engineers build against, and the trade-offs the business signs off on, while being deliberately one level removed from the day-to-day code that ships. This is not a seniority bump inside the same lane; it is a lane change. The failure mode that kills most transitions is the engineer who gets the architect title and keeps doing the engineer's job — reviewing every PR, writing the tricky DAGs personally, being the on-call hero — while the actual architect responsibilities (the reference designs, the governance model, the data-contract standards, the cost envelope) go unowned because nobody was told they were now theirs.

What changes across the four dimensions interviewers probe.

Scope of accountability. A DE is accountable for a component — a pipeline, a table, a service. An architect is accountable for a system — how components fit, what standard they share, how the whole thing evolves. Interviewers test this by asking you to zoom out: "don't tell me how you'd build the pipeline, tell me how you'd decide whether this data even belongs in the warehouse."
Breadth vs depth. The DE goes deep on a stack — Spark internals, Airflow scheduling, Postgres query plans. The architect goes broad across domains — modeling, governance, integration, cost, cloud, security — trading some depth in each for the ability to reason about all of them at once. You cannot be an architect who only knows the ingestion layer.
Time horizon. The DE optimises for the sprint and the incident. The architect optimises for the three-year horizon — the migration cost, the vendor lock-in, the schema decisions that are cheap now and expensive later. "What will this decision cost us in year three?" is an architect question, never a DE question.
Communication surface. The DE talks to other engineers and a manager. The architect talks to engineers, product, finance, security, and often the C-suite. The blueprint only matters if the people funding it and building it both understand it, so the architect's output is as much a document and a conversation as it is a diagram.

The skill-ladder overview — the six things an architect is expected to hold.

Data modeling. Dimensional (star / snowflake), normalized (3NF), and Data Vault — knowing which fits which workload and why. This is the pillar DEs are closest to and still underrate.
Governance and security. Catalog, lineage, access control, PII classification, data contracts, retention. The pillar DEs are furthest from and interviewers weight highest.
Integration patterns. Batch, streaming, CDC, API integration, event-driven — the menu of how systems exchange data and the trade-offs of each.
Cost and performance. Storage tiering, partitioning, compute right-sizing, the FinOps of a warehouse. The pillar that turns a "cool design" into a "fundable design".
Cloud platforms. The reference services on at least one major cloud, and enough of the other two to reason about portability and lock-in.
Communication and stewardship. ADRs, reference designs, trade-off narratives — the connective tissue that makes the other five visible to the org.

What interviewers listen for in the DE-to-architect screen.

Do you frame answers as decisions with trade-offs rather than tools with features? — senior signal.
Do you lead with governance and cost unprompted, not just latency and correctness? — required answer.
Do you say "it depends, and here's the axis it depends on" instead of naming a single right answer? — senior signal.
Do you describe your role as "setting the standard other teams build against" rather than "building the hardest pipelines"? — required answer.
Do you show you can zoom out to the business — cost, risk, compliance — not just the stack? — senior signal.

Worked example — the DE-vs-architect responsibility table

Detailed explanation. The most useful artifact for the transition conversation — and the first thing a hiring manager wants to see you understand — is a clean responsibility split between the two roles. Getting this table right proves you know what you are signing up for, which is half the battle in an architect screen.

The axis. For each responsibility, who is accountable (owns the outcome) versus who is responsible (does the work).
The trap. Candidates who say the architect "does everything the DE does, plus design" have not understood the lane change — the architect is less hands-on by design.
The signal. The strongest answers show the architect delegating implementation and owning standards, review of designs (not code), and cross-team consistency.

Question. Build the responsibility split between data engineer and data architect for a typical warehouse-plus-lakehouse platform team, and identify the two responsibilities most likely to be dropped in a botched transition.

Input.

Responsibility	Data engineer	Data architect
Pipeline / DAG implementation	owns	reviews design, not code
Table / schema design	owns per-table	owns modeling standard
Reference architecture	consumes	owns
Governance & data contracts	applies	defines
Cost envelope / FinOps	reports usage	owns budget & tiering policy
Cross-team consistency	local	owns platform-wide

Code.

Responsibility split — DE vs Data Architect (RACI-style)
========================================================

R = Responsible (does the work)   A = Accountable (owns the outcome)

Area                         | DE   | Architect
-----------------------------+------+----------
Build a pipeline             |  R   |   -
Design one table             |  R   |  (A on the standard)
Set the modeling standard    |  -   |   A
Choose batch vs streaming    |  R*  |   A     (* recommends; architect decides)
Define data contracts        |  R   |   A
Classify PII / access tiers   |  R   |   A
Own the cost envelope        |  -   |   A
Approve a reference design    |  -   |   A
Cross-team schema consistency |  -   |   A
On-call for a pipeline       |  R   |   -

Most-dropped in a botched transition:
  1. "Own the cost envelope"  — nobody picks it up; spend drifts.
  2. "Cross-team consistency" — each team re-invents; the platform fragments.

Step-by-step explanation.

Read the table top-down and notice the pattern: the DE column is full of R (does the work), the architect column is full of A (owns the outcome). The architect is accountable for far more than they are responsible for — that is the definition of the role.
The single row that flips both — "choose batch vs streaming" — is where DE and architect overlap. The DE recommends based on hands-on knowledge; the architect decides based on the trade-off across cost, latency, and governance. Naming this handoff explicitly is a senior signal.
"Set the modeling standard" has no DE responsibility at all — the DE designs individual tables within a standard the architect owns. If you cannot articulate a modeling standard, you are not ready for the architect column.
The two most-dropped responsibilities — cost envelope and cross-team consistency — are dropped precisely because they are pure-architect (no DE fallback owns them). A transition that leaves these unowned produces spiralling cloud bills and a fragmented platform within two quarters.
The whole table is your answer to "what changes when you become an architect": you move from the R column to the A column, and you inherit responsibilities that nobody else in the org is positioned to hold.

Output.

Botched-transition symptom	Root cause	The architect responsibility that was dropped
Cloud bill grows 40% YoY unexplained	no budget owner	cost envelope / FinOps
Every team models "customer" differently	no shared standard	modeling standard + consistency
Governance audit fails on lineage	no contract owner	data contracts / classification
Architect is the bottleneck on every PR	stayed in the R column	delegation of implementation

Rule of thumb. Draw the RACI split before you take the title. If you cannot name at least three responsibilities that move from you do the work to you own the outcome others do the work on, you are interviewing for a senior-DE role with an architect label, not for an architect role.

Worked example — a skill-gap self-assessment rubric

Detailed explanation. Before you can close the gap you have to measure it honestly. The self-assessment rubric scores you on each of the six pillars from 0 (no exposure) to 4 (can set the org standard), which turns "I want to be an architect" into a ranked list of what to build first. Most DEs discover they are a 3–4 on modeling and integration and a 0–1 on governance and cost — which is exactly the profile interviewers screen out.

The scale. 0 none, 1 aware, 2 can apply under guidance, 3 can lead independently, 4 can set the org standard.
The bar. An architect hire is expected at 3+ across all six, with at least one 4. A 4 in modeling and a 1 in governance is a rejection, not a strength.
The output. Your two lowest scores are your transition plan for the next two quarters.

Question. Score a typical mid-senior DE against the six-pillar rubric and derive the first two areas to invest in.

Input.

Pillar	Typical mid-senior DE	Architect bar
Data modeling	3	3+
Governance & security	1	3+
Integration patterns	3	3+
Cost & performance	2	3+
Cloud platforms	2	3+
Communication & stewardship	2	3+

Code.

# Skill-gap self-assessment — score each pillar 0..4, rank the gaps
PILLARS = {
    "data_modeling":     3,
    "governance":        1,
    "integration":       3,
    "cost_performance":  2,
    "cloud_platforms":   2,
    "communication":     2,
}
ARCHITECT_BAR = 3

def transition_plan(scores: dict[str, int], bar: int = ARCHITECT_BAR) -> list[tuple[str, int]]:
    """Return pillars below the architect bar, largest gap first."""
    gaps = [(p, bar - s) for p, s in scores.items() if s < bar]
    gaps.sort(key=lambda x: (-x[1], x[0]))   # biggest gap first, then alphabetical
    return gaps

for pillar, gap in transition_plan(PILLARS):
    print(f"{pillar:18s} gap = {gap}  -> invest now")
# governance          gap = 2  -> invest now
# cloud_platforms     gap = 1  -> invest now
# communication       gap = 1  -> invest now
# cost_performance    gap = 1  -> invest now

Step-by-step explanation.

The rubric forces a number on each pillar, which defeats the natural bias to over-rate the pillars you enjoy (modeling, integration) and ignore the ones you avoid (governance, cost). The number is the point.
transition_plan filters to pillars below the bar and ranks by gap size. The largest gap — governance at 2 — is the first investment, because it is both the furthest from the bar and the pillar interviewers weight highest.
Ties (cloud, communication, cost all at gap 1) are broken alphabetically here, but in practice you break ties by interview frequency: cost and governance come up in almost every architect screen, so they jump the queue over the others.
The plan is deliberately short — two to four items — because a transition that tries to level all six pillars at once levels none. Fix the biggest gap to a 3, re-score, then move to the next.
Re-run the assessment every quarter. The transition is done when every pillar reads 3+ and you have earned at least one 4 through a real project you can talk about in an interview.

Output.

Priority	Pillar	Gap	First concrete action
1	Governance & security	2	Design a data-contract + PII-classification standard for one domain
2	Cost & performance	1	Build a cost model for one warehouse workload; propose tiering
3	Cloud platforms	1	Earn one associate-level cloud data cert on your primary cloud
4	Communication	1	Write three ADRs and one reference design for real decisions

Rule of thumb. Score all six pillars honestly, sort by gap, and fix the largest gap first — and remember that a single 4 with a 1 elsewhere reads as specialist, not architect. Breadth at 3 beats a spike at 4 with a hole beside it.

Worked example — the "why architect" positioning template

Detailed explanation. Every architect screen opens with some version of "you're a strong data engineer — why architect?" The weak answer is "it's the next level up" (which frames it as a title grab). The strong answer names the specific decisions you already influence informally, the gap you have deliberately closed, and the scope you want to be accountable for. This is a 90-second scripted answer you rehearse once and deploy every time.

Part 1 — the pull. The kind of problem you already gravitate to: the cross-team decision, the standard, the trade-off — not the hard pipeline.
Part 2 — the proof. A concrete decision you already drove that was architect-shaped (a modeling standard, a build-vs-buy call, a governance model).
Part 3 — the gap closed. The pillar you knew you were weak on and what you did about it — this pre-empts the "but you've never owned governance" objection.

Question. Draft a 90-second "why architect" answer that a hiring manager reads as ready, not ambitious.

Input.

Component	Weak answer	Strong answer
Framing	"it's the next promotion"	"I already make architect-shaped decisions"
Proof	"I built hard pipelines"	"I set our modeling standard across 3 teams"
Gap awareness	"I know everything already"	"I was weak on governance; here's what I did"
Scope wanted	"more responsibility"	"accountability for the platform blueprint"

Code.

"Why architect?" — 90-second answer template
============================================

Beat 1 — the pull (20s)
  "The parts of my current role I gravitate to aren't the pipelines —
   they're the decisions. Which modeling pattern the team standardises
   on, whether we buy or build ingestion, how we keep 'customer'
   consistent across three squads. I'm already doing that informally."

Beat 2 — the proof (30s)
  "Last year I wrote the star-schema standard our analytics teams now
   build against, and led the build-vs-buy call on CDC — we picked
   log-based over a vendor and I owned the trade-off doc that got it
   funded. That's architect work; I want the title to match the scope."

Beat 3 — the gap I closed (25s)
  "I knew governance was my weak pillar — I'd applied contracts but
   never designed the standard. So I built a PII-classification and
   data-contract model for our billing domain and got it through a
   security review. That closed the gap I'd have been screened on."

Beat 4 — the scope I want (15s)
  "What I want to be accountable for is the platform blueprint — the
   reference designs, the standards, the cost envelope — so twelve
   engineers can move fast without re-litigating the same decisions."

Step-by-step explanation.

Beat 1 reframes the ambition from "next level" to "already doing the work" — the single most important move, because it turns the question from are you ready? into why isn't your title accurate yet?
Beat 2 supplies proof in the architect vocabulary: a standard (not a pipeline), a build-vs-buy call (not a bug fix), a trade-off doc that got funded (business outcome, not technical output). Every noun signals the lane change.
Beat 3 is the counter-intuitive move: you name your weakest pillar out loud. This disarms the interviewer's biggest objection before they raise it, and it demonstrates the self-assessment discipline the role requires.
Beat 4 states the scope you want in architect terms — accountability for the blueprint — and ties it to a business value ("twelve engineers move fast without re-litigating"). You are selling leverage, not seniority.
The whole script is under two minutes and rehearsed to sound unrehearsed. It does the job of the entire first round: it proves you understand the role, have done the work, know your gaps, and want the right scope for the right reason.

Output.

Beat	Purpose	What the interviewer concludes
Pull	reframe ambition as fit	"the title is lagging the scope"
Proof	evidence in architect vocabulary	"they've done real architect work"
Gap closed	disarm the objection	"self-aware; already closed the weak pillar"
Scope	sell leverage over seniority	"wants accountability for the right reasons"

Rule of thumb. Never answer "why architect?" with "it's the next level." Answer with the architect-shaped decisions you already drive, the weakest pillar you deliberately closed, and the blueprint-level scope you want to own — leverage over the org, not a rung on the ladder.

Senior interview question on the DE-to-architect transition

A senior interviewer often opens with: "You're clearly a strong data engineer. Convince me you're ready to be a data architect — not just senior — and tell me the one gap you'd close in your first ninety days and how."

Solution Using a decisions-owned narrative plus a scoped 90-day governance closeout

Answer structure — "ready for architect" in four moves
======================================================

Move 1 — decisions I already own (proof of lane change)
  "I already own the modeling standard for our analytics domain and I
   drove the log-based-vs-vendor CDC call. Both were decisions, not
   deliverables — that's the architect column of the RACI."

Move 2 — the honest gap (self-assessment)
  "My weakest pillar on the six-pillar ladder is governance. I score
   myself 3+ on modeling, integration, and cost, but only a 2 on
   governance because I've applied contracts, not designed the standard."

Move 3 — the scoped 90-day closeout (the plan)
  "First 90 days I'd close it with one real deliverable, not a course:
   a data-contract + PII-classification standard for one high-value
   domain, reviewed by security, adopted by one team. Scoped to one
   domain so it ships; templated so it generalises."

Move 4 — how I'd measure it (accountability)
  "Success = one domain with published contracts, a classification
   applied to every column, and a lineage view the auditor accepts.
   That converts my governance score from a 2 to a 3 with evidence."

# 90-Day Governance Closeout — Billing Domain (scoped)

## Goal
Move governance from "applied" to "standard-setting" with one shippable artifact.

## Weeks 1-3 — inventory & classify
- Catalog every table/column in the billing domain
- Classify columns: public / internal / confidential / PII
- Output: a classification matrix (column -> tier -> masking rule)

## Weeks 4-8 — data contracts
- Define a contract schema (owner, SLA, schema version, breaking-change policy)
- Write contracts for the 5 highest-traffic billing tables
- Wire a CI check that fails a PR on a breaking schema change

## Weeks 9-12 — lineage & sign-off
- Stand up column-level lineage for the domain (catalog tool or dbt exposures)
- Security review + auditor walkthrough
- Output: a governed domain other teams can copy as a template

-- The CI data-contract check (Weeks 4-8): fail on a breaking change.
-- Compares the live schema against the committed contract snapshot.
WITH contract AS (      -- expected columns from the checked-in contract
    SELECT column_name, data_type, is_nullable
    FROM   contract_billing_orders_v3
),
live AS (               -- actual columns in the warehouse right now
    SELECT column_name, data_type, is_nullable
    FROM   information_schema.columns
    WHERE  table_schema = 'billing' AND table_name = 'orders'
)
SELECT 'BREAKING: column removed or retyped' AS violation,
       c.column_name, c.data_type AS expected_type, l.data_type AS actual_type
FROM   contract c
LEFT JOIN live l ON l.column_name = c.column_name
WHERE  l.column_name IS NULL          -- column dropped
   OR  l.data_type   <> c.data_type   -- type changed
   OR  (c.is_nullable = 'NO' AND l.is_nullable = 'YES');  -- nullability relaxed

Step-by-step trace.

Step	Input	Reasoning
Move 1	two owned decisions	proves you're already in the A column
Move 2	six-pillar self-score	names the gap before the interviewer does
Move 3	one scoped domain, 90 days	shows delivery discipline, not a wish list
Move 4	measurable success criteria	frames governance as an accountable outcome
Contract CI check	schema vs contract	governance made executable, not aspirational
Lineage sign-off	auditor accepts	evidence the gap is truly closed

After this answer, the interviewer has heard you name architect-shaped decisions you already own, self-assess honestly on the exact pillar they were about to probe, and propose a scoped, measurable 90-day closeout that produces a reusable template rather than a personal certificate. The SQL contract check turns "I'll improve governance" into "here is the mechanism that enforces it in CI" — the difference between an aspiration and a design.

Output:

Signal probed	Weak candidate	This answer
Lane change understood	"it's more senior"	owns decisions, not just deliverables
Self-awareness	"no real gaps"	names governance as the weak pillar
Delivery discipline	"I'd take a course"	scoped 90-day shippable standard
Governance depth	vague	executable contract check + lineage
Accountability	none stated	measurable success criteria

Why this works — concept by concept:

Decisions over deliverables — the architect is accountable for outcomes others implement. Leading with two owned decisions (a modeling standard, a build-vs-buy call) proves you already live in the accountable column, which is the whole test.
Honest self-assessment — naming governance as your weakest pillar before the interviewer probes it converts a potential rejection into a demonstration of the self-awareness the role requires. Hiding the gap is the losing move.
Scoped closeout — one domain, 90 days, one shippable standard. Scope is the architect's discipline: a plan that boils the ocean signals you don't yet think in deliverable increments.
Governance as executable — the contract CI check and column-level lineage turn governance from a slide into a mechanism. Architects are graded on whether the standard is enforceable, not whether it sounds good.
Cost — the transition itself is the cost: closing one pillar to a 3 is roughly one quarter of deliberate, project-based effort per pillar. Budget two-to-four quarters to move a strong DE across the boundary, front-loaded on governance and cost — the two pillars DEs are furthest from and interviewers weight highest.

Design
Topic — design
System-design problems for aspiring architects

Practice →

SQL Topic — sql SQL problems on schemas, contracts, and metadata

Practice →

2. The architect skill ladder: modeling, governance, cost

The five competency pillars — where DEs are strong, where they're screened out, and how each is scored

The mental model in one line: the architect skill ladder is five competency pillars — data modeling, governance and security, integration patterns, cost and performance, and cloud platforms — and the hiring bar is not a spike in one but a 3-out-of-4 across all five, because an architect who is brilliant at modeling and blind to governance produces designs that ship a compliance incident. Every senior DE is already climbing two or three of these rungs at work; the transition is about the ones they never touch.

The five pillars, and what "architect level" means on each.

Data modeling. Not "can write a CREATE TABLE" — can choose between dimensional (star/snowflake), fully normalized (3NF), and Data Vault for a given workload, and defend the choice on query pattern, change rate, and auditability. Architect level = you set the modeling standard and the team's tables conform to it.
Governance and security. Catalog, column-level lineage, access control, PII classification, data contracts, retention and residency. This is the pillar DEs score lowest on and interviewers weight highest — because a governance miss is the one architecture mistake that ends up in a regulator's inbox.
Integration patterns. Batch, micro-batch, streaming, CDC, request/response API integration, event-driven — the menu of how systems exchange data, and the latency/cost/complexity trade-off of each. Architect level = you match the pattern to the requirement instead of defaulting to the one you know.
Cost and performance. Partitioning, clustering, storage tiering, compute right-sizing, and the FinOps discipline of attaching a dollar figure to a design. Architect level = every reference design carries a cost model, not just a data-flow diagram.
Cloud platforms. Deep on one major cloud's data stack, literate on the other two. Architect level = you can reason about portability, lock-in, and the managed-vs-self-hosted trade-off, not just wire up one vendor's services.

Data modeling is the pillar DEs underrate — the three canonical models.

Dimensional (star schema). Fact tables surrounded by denormalized dimension tables. Optimised for analytical read patterns — wide scans, aggregations, slice-and-dice BI. The default for a warehouse's gold layer. Fewer joins, faster reads, controlled redundancy.
Normalized (3NF). Every fact stored once, relationships expressed by foreign keys. Optimised for write-heavy operational (OLTP) systems and for a warehouse's staging/silver layer where integrity matters more than read speed. More joins, less redundancy, cleaner writes.
Data Vault. Hubs (business keys), links (relationships), satellites (attributes over time). Optimised for auditability, source-system agility, and highly regulated environments — you can add a source without breaking the model. Verbose to query; typically feeds a dimensional presentation layer on top.

Governance is the pillar that gets you hired or rejected — the five sub-competencies.

Cataloging & lineage. Every dataset discoverable; column-level lineage answers "where did this number come from?" for the auditor.
Access control & classification. Every column tagged (public / internal / confidential / PII) with a masking and row-access policy attached to the tag.
Data contracts. Producer-consumer agreements with schema, SLA, and a breaking-change policy enforced in CI.
Retention & residency. How long data lives and which jurisdiction it lives in — GDPR/CCPA-shaped requirements that constrain the physical design.
Quality & observability. Freshness, completeness, and validity checks with owners and alerts — governance is not just access, it's trust.

Worked example — the competency-pillar rubric

Detailed explanation. The pillar rubric defines what novice, practitioner, and architect look like on each of the five pillars, so both you and an interviewer can place you precisely. It is the scoring key behind the self-assessment from Section 1.

Novice. Aware of the concept; can apply an existing pattern under guidance.
Practitioner. Applies the pattern independently on their own component.
Architect. Sets the standard the whole platform builds against and defends the trade-off to the business.

Question. Fill in the rubric for the modeling and governance pillars, and state the observable behaviour that distinguishes a practitioner from an architect.

Input.

Pillar	Practitioner behaviour	Architect behaviour
Data modeling	designs a correct star schema for their mart	sets the org's modeling standard; picks star vs 3NF vs Vault per workload
Governance	applies the existing classification tags	designs the classification + contract + lineage standard
Integration	builds the pipeline the design specifies	chooses batch vs streaming vs CDC on trade-offs
Cost	reports their pipeline's spend	owns the cost model and tiering policy
Cloud	uses the cloud services set up for them	reasons about portability and lock-in

Code.

Pillar rubric — the practitioner -> architect line
==================================================

Pillar         | Novice (1)        | Practitioner (2-3)      | Architect (4)
---------------+-------------------+-------------------------+---------------------------
Data modeling  | reads a star      | builds a correct mart   | SETS the modeling standard
Governance     | knows PII exists  | applies the tags        | DESIGNS the classification model
Integration    | knows batch       | builds one pattern well | CHOOSES the pattern on trade-offs
Cost           | sees the bill     | reports own spend       | OWNS the cost model & tiering
Cloud          | uses the console  | ships on one service    | REASONS about lock-in & portability

The line between practitioner and architect is always the same verb:
  practitioner APPLIES the standard;  architect SETS the standard.

Step-by-step explanation.

The rubric's power is that the practitioner-to-architect jump is the same verb on every pillar: the practitioner applies a standard someone else set; the architect sets the standard. Once you see that pattern, you can self-place instantly.
On modeling, a practitioner builds a correct star schema; the architect decides when a star schema is wrong (write-heavy source → 3NF; regulated multi-source → Vault). The choice, not the construction, is the architect skill.
On governance, the gap is starkest: applying tags is a practitioner task; designing the classification taxonomy, the contract schema, and the lineage model is the architect task. This is why DEs who "do governance" still screen as practitioners.
On integration, the architect chooses among batch/streaming/CDC/event-driven based on the requirement; the practitioner builds whichever one was chosen for them. Defaulting to your favourite pattern is a practitioner tell.
On cost and cloud, the jump is from reporting/using to owning/reasoning. An architect attaches a dollar figure and a lock-in analysis to every design; a practitioner ships and lets someone else worry about the bill.

Output.

Pillar	The one question that reveals architect level
Data modeling	"When is a star schema the wrong choice?"
Governance	"Show me your classification taxonomy and contract schema."
Integration	"Why batch here and streaming there?"
Cost	"What does this design cost per month, and why that tier?"
Cloud	"What's your lock-in exposure and exit cost?"

Rule of thumb. Score yourself with the verb test: on each pillar, do you apply the standard or set it? You need "set" on modeling plus governance and "apply-to-set in progress" on the rest before you interview as an architect.

Worked example — a data-modeling worked example (star vs 3NF)

Detailed explanation. Modeling is the pillar you can prove in code, so architect interviews always include a modeling exercise. The canonical one: take an operational (3NF) order schema and design the dimensional (star) model an analytics team should query, explaining every denormalization choice. Do it end to end.

The source (3NF). Normalized OLTP: orders, customers, products, order_items — each fact once, joined by keys, optimised for writes.
The target (star). A fact_sales grain of one row per order line, surrounded by conformed dimensions dim_customer, dim_product, dim_date — optimised for analytical reads.
The judgement. What to denormalize, what to keep as a surrogate key, and how to handle a changing dimension.

Question. Given a 3NF order schema, design the star schema for sales analytics, and justify the grain and the surrogate keys.

Input.

Source table (3NF)	Star role	Notes
orders	fact (header)	dates, status → fact + dim_date
order_items	fact (grain)	one fact row per line item
customers	dim_customer	denormalize region, segment
products	dim_product	denormalize category, brand

Code.

-- SOURCE: normalized 3NF (operational) — every fact once, write-optimised
CREATE TABLE customers (
    customer_id BIGINT PRIMARY KEY,
    name        TEXT, region_id INT REFERENCES regions(region_id),
    segment_id  INT REFERENCES segments(segment_id)
);
CREATE TABLE products (
    product_id  BIGINT PRIMARY KEY,
    name        TEXT, category_id INT REFERENCES categories(category_id)
);
CREATE TABLE orders (
    order_id    BIGINT PRIMARY KEY,
    customer_id BIGINT REFERENCES customers(customer_id),
    order_ts    TIMESTAMPTZ, status TEXT
);
CREATE TABLE order_items (
    order_id    BIGINT REFERENCES orders(order_id),
    product_id  BIGINT REFERENCES products(product_id),
    qty         INT, unit_price_cents BIGINT,
    PRIMARY KEY (order_id, product_id)
);

-- TARGET: dimensional star (analytics) — read-optimised, controlled redundancy
CREATE TABLE dim_date (
    date_key     INT PRIMARY KEY,        -- surrogate: YYYYMMDD
    full_date    DATE, year INT, quarter INT, month INT, day_of_week INT
);
CREATE TABLE dim_customer (
    customer_key BIGINT PRIMARY KEY,     -- SURROGATE key (not the source id)
    customer_id  BIGINT,                 -- natural/business key retained
    name         TEXT,
    region       TEXT,                   -- denormalized from regions
    segment      TEXT                    -- denormalized from segments
);
CREATE TABLE dim_product (
    product_key  BIGINT PRIMARY KEY,     -- surrogate key
    product_id   BIGINT,
    name         TEXT,
    category     TEXT,                   -- denormalized from categories
    brand        TEXT
);
CREATE TABLE fact_sales (
    sales_key     BIGSERIAL PRIMARY KEY,
    date_key      INT    REFERENCES dim_date(date_key),
    customer_key  BIGINT REFERENCES dim_customer(customer_key),
    product_key   BIGINT REFERENCES dim_product(product_key),
    order_id      BIGINT,                -- degenerate dimension
    qty           INT,
    revenue_cents BIGINT                 -- additive measure = qty * unit_price
);

Step-by-step explanation.

The grain is chosen first and drives everything: one fact_sales row per order_item (line item). Grain is the single most important modeling decision — the wrong grain (per-order instead of per-line) makes product-level analysis impossible.
Dimensions denormalize the 3NF lookups: region and segment collapse into dim_customer; category and brand collapse into dim_product. This trades controlled redundancy for join elimination — a BI query touches one fact and three small dimensions instead of eight normalized tables.
Every dimension gets a surrogate key (customer_key) distinct from the natural key (customer_id). Surrogate keys decouple the warehouse from source-system key churn and — crucially — enable slowly-changing-dimension history (a customer who changes segment gets a new surrogate, preserving the old fact rows' link to the old segment).
order_id rides along as a degenerate dimension — a dimension attribute with no dimension table — so you can still group by order without a dim_order. revenue_cents is a fully additive measure (safe to SUM across any dimension); storing it precomputed avoids per-query multiplication.
dim_date is a conformed dimension: pre-built with a YYYYMMDD integer surrogate, shared by every fact table in the warehouse. Conformed dimensions are what let "sales by month" and "returns by month" line up — the architect owns the conformed set.

Output.

Query pattern	On 3NF source	On star schema
Revenue by category by quarter	6-table join	1 fact + 2 dims
Top customers by segment	4-table join	1 fact + 1 dim
Write a new order	1 clean insert	N/A (analytics only)
Track customer segment change over time	not modeled	SCD Type 2 on dim_customer

Rule of thumb. Choose the grain before anything else, give every dimension a surrogate key, denormalize lookups into dimensions, and keep measures additive. State out loud that the 3NF source is correct for writes and the star is correct for reads — the architect signal is knowing both are right for their job.

Worked example — an SCD Type 2 dimension for change history

Detailed explanation. The follow-up to any modeling question is "how do you track a dimension attribute that changes over time?" — the slowly-changing-dimension (SCD) problem. Type 2 (keep full history via new rows) is the architect default, and knowing the surrogate-key/effective-date mechanics cold is a strong modeling signal.

The scenario. A customer moves from segment "SMB" to "Enterprise". Historical facts must still attribute to "SMB"; new facts to "Enterprise".
The mechanism. On change, expire the current dim row (valid_to, is_current = false) and insert a new row with a new surrogate key and is_current = true.
The payoff. Facts join to the surrogate key that was current at fact time, so history is preserved automatically.

Question. Implement SCD Type 2 on dim_customer and show the state after a segment change.

Input.

Field	Purpose
customer_key	surrogate PK, new per version
customer_id	natural/business key, stable
valid_from / valid_to	version validity window
is_current	fast filter for the live row

Code.

-- SCD Type 2 dimension: history via versioned rows
CREATE TABLE dim_customer (
    customer_key BIGSERIAL PRIMARY KEY,   -- surrogate, changes per version
    customer_id  BIGINT      NOT NULL,    -- natural key, stable across versions
    name         TEXT,
    region       TEXT,
    segment      TEXT,
    valid_from   TIMESTAMPTZ NOT NULL,
    valid_to     TIMESTAMPTZ,             -- NULL = still current
    is_current   BOOLEAN     NOT NULL DEFAULT true
);

-- Apply a change: customer 7 moves SMB -> Enterprise
-- 1. Expire the current version
UPDATE dim_customer
SET    valid_to = now(), is_current = false
WHERE  customer_id = 7 AND is_current;

-- 2. Insert the new version (new surrogate key)
INSERT INTO dim_customer (customer_id, name, region, segment, valid_from, is_current)
VALUES (7, 'Acme Corp', 'EMEA', 'Enterprise', now(), true);

-- Facts inserted before the change keep pointing at the OLD customer_key;
-- facts after the change resolve to the NEW customer_key via is_current.

Step-by-step explanation.

The natural key customer_id = 7 is stable; the surrogate customer_key changes with every version. This split is what makes Type 2 work — facts reference the surrogate, so they are frozen to the version that was current when they occurred.
Step 1 expires the old row by stamping valid_to = now() and is_current = false. The old row is never deleted — that is the whole point of Type 2, full history.
Step 2 inserts a new row with a fresh surrogate key, the new segment, valid_from = now(), and is_current = true. There are now two rows for customer 7, distinguishable by their validity windows.
A fact-loading job resolves customer_id to customer_key by joining on customer_id AND is_current (for streaming loads) or on customer_id AND fact_ts BETWEEN valid_from AND COALESCE(valid_to, 'infinity') (for backfills) — the latter attributes each historical fact to the correct version.
Analytical queries "revenue by segment over time" now correctly show the customer's revenue under SMB before the change and Enterprise after — because the facts carry the surrogate that encodes the segment-at-the-time. Type 1 (overwrite) would have rewritten history; Type 2 preserves it.

Output.

customer_key	customer_id	segment	valid_from	valid_to	is_current
101	7	SMB	2024-01-01	2026-08-04	false
245	7	Enterprise	2026-08-04	(null)	true

Rule of thumb. For any dimension whose attribute changes and whose history matters, use SCD Type 2: stable natural key, versioned surrogate key, validity window, is_current flag. Reach for Type 1 (overwrite) only when history genuinely does not matter — and say why out loud.

Senior interview question on modeling a domain end-to-end

A senior interviewer might ask: "Model a subscription-billing domain end to end. Give me the operational schema, the warehouse star schema for revenue analytics, how you'd handle a customer changing plans mid-cycle, and the one governance control you'd attach to the model."

Solution Using a 3NF source, a conformed star, SCD Type 2 plans, and a PII-classification control

-- 1. OPERATIONAL (3NF) — write-optimised source of truth
CREATE TABLE customers (
    customer_id BIGINT PRIMARY KEY, email TEXT, country TEXT, created_at TIMESTAMPTZ
);
CREATE TABLE plans (
    plan_id INT PRIMARY KEY, plan_name TEXT, monthly_price_cents BIGINT
);
CREATE TABLE subscriptions (
    subscription_id BIGINT PRIMARY KEY,
    customer_id BIGINT REFERENCES customers(customer_id),
    plan_id     INT    REFERENCES plans(plan_id),
    started_at  TIMESTAMPTZ, ended_at TIMESTAMPTZ, status TEXT
);
CREATE TABLE invoices (
    invoice_id BIGINT PRIMARY KEY,
    subscription_id BIGINT REFERENCES subscriptions(subscription_id),
    period_start DATE, period_end DATE, amount_cents BIGINT, paid_at TIMESTAMPTZ
);

-- 2. WAREHOUSE (star) — read-optimised revenue analytics
CREATE TABLE dim_date     (date_key INT PRIMARY KEY, full_date DATE, year INT, month INT);
CREATE TABLE dim_customer (                     -- SCD Type 2 for plan/segment history
    customer_key BIGSERIAL PRIMARY KEY, customer_id BIGINT, country TEXT,
    plan_name TEXT, valid_from TIMESTAMPTZ, valid_to TIMESTAMPTZ, is_current BOOLEAN
);
CREATE TABLE fact_revenue (
    revenue_key   BIGSERIAL PRIMARY KEY,
    date_key      INT    REFERENCES dim_date(date_key),
    customer_key  BIGINT REFERENCES dim_customer(customer_key),
    invoice_id    BIGINT,                        -- degenerate dimension
    mrr_cents     BIGINT,                         -- monthly recurring revenue (additive)
    is_new        BOOLEAN, is_churn BOOLEAN       -- movement flags for MRR waterfall
);

-- 3. Plan change mid-cycle = SCD Type 2 version bump on dim_customer,
--    plus proration handled in fact_revenue as two partial-period rows.
UPDATE dim_customer SET valid_to = now(), is_current = false
WHERE  customer_id = 7 AND is_current;
INSERT INTO dim_customer (customer_id, country, plan_name, valid_from, is_current)
VALUES (7, 'DE', 'Enterprise', now(), true);

-- 4. Governance control: classify + mask PII (email) with a policy tied to a tag.
COMMENT ON COLUMN customers.email IS 'classification=PII';
CREATE OR REPLACE VIEW analytics.dim_customer_masked AS
SELECT customer_key, customer_id, country, plan_name, valid_from, valid_to, is_current
FROM   analytics.dim_customer;   -- email deliberately excluded from the analytics view

Step-by-step trace.

Layer	Choice	Reasoning
Source	3NF	write integrity for the billing OLTP
Warehouse	star + conformed dim_date	fast MRR analytics across time
Grain	one fact_revenue row per invoice period	supports the MRR waterfall
Plan change	SCD Type 2 on dim_customer	preserves revenue-by-plan history
Movement flags	is_new / is_churn	enables new/expansion/churn breakdown
Governance	email tagged PII, excluded from analytics view	least-privilege by construction

The design carries revenue at the invoice-period grain, attributes each period to the plan that was current when it was billed (via the SCD Type 2 surrogate), flags new and churned revenue for the MRR waterfall, and — critically — attaches a governance control at modeling time by classifying email as PII and building the analytics dimension without it. The architect signal is that governance is in the model, not bolted on afterward.

Output:

Analytics question	Answered by
MRR by month	SUM(mrr_cents) over dim_date
Revenue by plan over time	fact_revenue × SCD Type 2 dim_customer
New vs churned MRR	is_new / is_churn flags
Revenue by country	dim_customer.country
Can analysts see customer email?	no — excluded by the masked view

Why this works — concept by concept:

Grain-first modeling — choosing "one fact per invoice period" before drawing anything makes the MRR waterfall and proration natural. The wrong grain would make plan-level revenue analysis impossible to reconstruct later.
Conformed dimensions — a shared dim_date (and a shared dim_customer) lets every fact table line up on time and customer, which is what makes cross-domain analytics coherent. The architect owns the conformed set for the whole platform.
SCD Type 2 — versioned surrogate keys preserve revenue-by-plan history through a mid-cycle change. Overwriting (Type 1) would silently rewrite last quarter's numbers — a correctness bug that surfaces at board-reporting time.
Governance in the model — tagging email as PII and shaping the analytics view to exclude it is least-privilege by construction. Governance attached at modeling time is cheaper and safer than a masking policy retrofitted after an audit finding.
Cost — the star adds controlled storage redundancy (denormalized dimensions) in exchange for order-of-magnitude fewer joins per analytical query — O(1 fact + few dims) versus O(many normalized joins). The SCD Type 2 history grows the dimension slowly; partitioning fact_revenue by date_key keeps scan cost proportional to the queried window, not the table.

Dimensional modeling
Topic — dimensional-modeling
Star-schema and SCD dimensional-modeling problems

Practice →

SQL Topic — sql SQL problems on joins, grain, and aggregation

Practice →

3. Certifications that actually move the needle

Which data architect certification pays off — mapped to the pillar it proves and the role it targets

The mental model in one line: a data architect certification is a signal, not a skill — it opens a door and proves a floor, but it never proves the judgement the role is actually graded on, so the right strategy is to earn the one or two certs that map to your weakest hire-relevant pillar and your target role, then spend the rest of your effort on a portfolio that proves the judgement a cert cannot. The mistake is collecting certs as trophies; the discipline is choosing the cert that closes a specific gap for a specific role.

The three certification families and what each signals.

Cloud data certifications. The major-cloud data/analytics/architect tracks (for example the AWS, Google Cloud, and Azure data-engineer and data/solutions-architect certifications). They prove you can wire a cloud data stack and reason about its managed services — the integration, cost, and cloud pillars. Highest ROI for platform/lakehouse architect roles at cloud-native companies.
Enterprise-architecture certifications. TOGAF and comparable enterprise-architecture frameworks. They prove you speak the language of business capability, application, and technology layers and can produce architecture artifacts a large enterprise expects. Highest ROI for enterprise-architect and data-architect roles in large, governance-heavy organisations (banks, insurers, healthcare, government).
Platform certifications. Vendor tracks for the data platform itself (for example Snowflake, Databricks, and dbt certifications). They prove depth on a specific platform's modeling, performance, and governance features. Highest ROI when the target role names that platform in the job description.

How to prioritise by target role.

Cloud-lakehouse architect (startup / scale-up). One cloud data/architect cert on the company's primary cloud + one platform cert (Databricks or Snowflake). Skip TOGAF — it reads as enterprise ceremony in a fast-moving shop.
Enterprise data architect (large regulated org). TOGAF (or equivalent) first — it is often a literal checkbox in the JD — plus one cloud architect cert for credibility. The enterprise-architecture framework is the differentiator here.
Platform / data-platform architect. The platform cert that matches the stack (Snowflake/Databricks/dbt) + one cloud cert. Depth on the named platform beats breadth.
Any role, weak on cost. A cloud architect cert doubles as a cost-and-performance credential because the exams heavily test right-sizing and storage tiering.

The cert-vs-portfolio trade-off — what each actually proves.

A cert proves a floor. It says "this person knows the services and the vocabulary." It gets your résumé past the screen and gives a non-technical recruiter a checkbox. It does not prove you can make a trade-off.
A portfolio proves judgement. Reference designs, ADRs, a governed data model, a cost model — these show you choosing, which is the entire architect job. A portfolio survives the technical interview; a cert only survives the recruiter screen.
The verdict. Certs are necessary-but-not-sufficient for many roles and pure-optional for others; a portfolio is close to universally decisive. Spend the minimum cert effort to clear the screen, then over-invest in the portfolio that wins the panel.

Worked example — a certification comparison matrix

Detailed explanation. The comparison matrix maps each cert family to the pillar it proves, the role it targets, and a rough ROI, so you can pick deliberately instead of collecting. ROI here is signal-per-unit-effort for your target role, not the cert's absolute prestige.

The columns. Cert family → pillar proven → best-fit role → effort → ROI.
The read. Your highest-ROI cert is the one whose pillar matches your weakest hire-relevant gap and whose role matches your target.
The trap. A high-prestige cert with low ROI for your target role (e.g. TOGAF for a startup lakehouse role) is a waste of a quarter.

Question. Build the cert comparison matrix and pick the highest-ROI cert for two candidates: one targeting a scale-up lakehouse role, one targeting an enterprise bank.

Input.

Cert family	Pillar proven	Best-fit role	Effort
Cloud data/architect	integration, cost, cloud	cloud-lakehouse architect	medium
Enterprise-architecture (TOGAF)	governance, blueprint vocabulary	enterprise data architect	medium-high
Platform (Snowflake/Databricks/dbt)	modeling, performance	platform architect	low-medium

Code.

Certification comparison matrix  (ROI = signal per unit effort, by target role)
===============================================================================

Cert family            | Proves pillar          | Best-fit role          | Effort | ROI*
-----------------------+------------------------+------------------------+--------+-----
Cloud data / architect | integration, cost,cloud| cloud-lakehouse arch.  | med    | HIGH  (scale-up)
Enterprise-arch (TOGAF)| governance, blueprint  | enterprise data arch.  | med-hi | HIGH  (bank/gov)
Platform (Snowflake/   | modeling, performance  | platform architect     | low-med| HIGH  (named stack)
  Databricks/dbt)      |                        |                        |        |
Extra cloud cert #2    | portability literacy   | any                    | med    | LOW   (diminishing)
Second EA framework    | vocabulary (again)     | none new               | high   | LOW   (redundant)

* ROI is role-dependent. TOGAF is HIGH for a bank, LOW for a 40-person startup.

Candidate A (scale-up lakehouse):  Cloud data/architect + one platform cert. Skip TOGAF.
Candidate B (enterprise bank):     TOGAF + one cloud architect cert. Platform cert optional.

Step-by-step explanation.

The matrix's key column is ROI, and ROI is explicitly role-dependent — the same TOGAF cert is HIGH for a bank and LOW for a startup. Reading the cert in isolation of the target role is the classic mistake.
Candidate A (scale-up) picks the cloud data/architect cert (proves integration + cost + cloud, the pillars a lakehouse role tests) plus a platform cert on the stack the company runs. TOGAF is skipped because enterprise ceremony is a negative signal in a fast shop.
Candidate B (bank) inverts it: TOGAF first because it is frequently a literal JD requirement and proves the blueprint vocabulary a governance-heavy org expects, plus one cloud architect cert for technical credibility.
The two LOW-ROI rows are the traps: a second cloud cert (diminishing returns once you can reason about one cloud) and a second EA framework (redundant vocabulary). Both consume a quarter and add no new signal.
The whole matrix reduces to a two-factor pick: match the pillar to your weakest hire-relevant gap, match the role to your target. Everything else is trophy-hunting.

Output.

Candidate	Highest-ROI cert(s)	What to skip	Why
Scale-up lakehouse	cloud data/architect + platform	TOGAF	ceremony reads negative
Enterprise bank	TOGAF + cloud architect	second platform cert	JD checkbox + credibility
Weak-on-cost DE	cloud architect	platform-only cert	exam tests right-sizing
Strong generalist	one cert, then portfolio	any second cert	diminishing returns

Rule of thumb. Pick certs by two factors only — the pillar they close and the role they target — and cap yourself at two before pivoting to portfolio. A third cert almost always has lower ROI than the first ADR or reference design you could have written instead.

Worked example — a 6-month cert + portfolio plan

Detailed explanation. The highest-leverage plan interleaves one cert with portfolio artifacts so the cert's study reinforces a real deliverable. Six months, roughly one artifact a month, one cert in the middle — the cert proves the floor, the artifacts prove the judgement.

The structure. Months 1–2 close the weakest pillar with a real project; month 3 earn the cert that certifies it; months 4–6 build the portfolio that proves judgement.
The principle. Never study a cert in the abstract — pair every study block with an artifact so the knowledge lands in something you can show.

Question. Write the six-month plan for a DE weak on governance and cost, targeting a cloud-lakehouse architect role.

Input.

Month	Focus	Artifact
1–2	governance + cost	data-contract standard + cost model
3	cloud architect cert	the certification
4–5	reference designs	governed lakehouse + streaming design
6	ADRs + presentation	ADR set + a design talk

Code.

# 6-Month DE -> Data Architect Plan (cloud-lakehouse target)

## Month 1-2 — close the weakest pillars (governance + cost)
- [ ] Data-contract + PII-classification standard for one domain (shippable)
- [ ] Cost model for one warehouse workload: storage tiering + compute right-sizing
- [ ] Wire a CI contract check that fails PRs on breaking schema changes
  OUTPUT: two portfolio artifacts + a measurable governance/cost win

## Month 3 — the cert (proves the floor)
- [ ] Cloud data/architect certification on the company's primary cloud
- [ ] Pair each study module with a note mapping the service to a design decision
  OUTPUT: the cert + a services-to-decisions cheat sheet

## Month 4-5 — reference designs (prove judgement)
- [ ] Governed lakehouse reference design (bronze/silver/gold + governance overlay)
- [ ] Streaming vs batch trade-off design for one real use case
  OUTPUT: two reference designs with NFRs and cost envelopes

## Month 6 — communication (prove stewardship)
- [ ] 5 ADRs documenting real decisions (with alternatives + trade-offs)
- [ ] One internal design talk / RFC walkthrough
  OUTPUT: an ADR set + evidence you can lead a design conversation

## Success criteria
- Six-pillar self-score now 3+ across the board, with a 4 in modeling or governance
- A portfolio you can screen-share in the panel: contracts, cost model, 2 designs, ADRs

Step-by-step explanation.

Months 1–2 attack the weakest pillars first (governance and cost) with shippable artifacts, not courses — a contract standard and a cost model that also become portfolio pieces. This front-loads the gap that gets DEs rejected.
Month 3 places the cert after the hands-on work, so the exam reinforces knowledge you already applied. Pairing each study module with a "service → decision" note converts rote memorisation into architect reasoning.
Months 4–5 build the artifacts that actually win the panel: two reference designs, each carrying NFRs and a cost envelope. These prove the judgement no cert can certify.
Month 6 closes on communication — five ADRs and a design talk — because stewardship is the sixth pillar and the one that makes the other five visible to the org.
The success criteria are explicit and measurable: a 3+ across all six pillars, one 4, and a screen-shareable portfolio. The plan is done when you can show the judgement, not just claim it.

Output.

Month	Deliverable	Pillar advanced
1–2	contract standard + cost model	governance, cost
3	cloud architect cert	cloud, integration
4–5	two reference designs	integration, cost, modeling
6	ADR set + design talk	communication

Rule of thumb. Interleave the cert between portfolio artifacts, never in isolation — study months bracketed by shipping months. The cert clears the recruiter screen; the artifacts win the panel, and the panel is where the offer is decided.

Worked example — an ROI scoring rubric for the next cert

Detailed explanation. When you are tempted by a third or fourth cert, run it through an ROI rubric that scores it on gap-closed, role-fit, effort, and shelf-life. A cert that scores low on any dimension is a trophy, and trophies do not win panels.

The dimensions. Gap closed (does it fix a weak pillar?), role fit (does the JD want it?), effort (weeks), shelf-life (how long the signal lasts).
The threshold. Only pursue a cert scoring high on gap-closed and role-fit. Prestige alone is not on the rubric.

Question. Score three candidate certs for a DE who already holds one cloud cert, and decide which (if any) to pursue.

Input.

Candidate cert	Gap closed?	Role fit?	Effort
Second cloud cert	no (already literate)	weak	medium
Platform cert (target stack)	yes (modeling depth)	strong	low-med
TOGAF	partial (governance vocab)	depends on target	med-high

Code.

# ROI rubric for the next cert. Pursue only if roi >= threshold.
def cert_roi(gap_closed: int, role_fit: int, effort_weeks: int, shelf_life_yrs: int) -> float:
    # gap_closed, role_fit on 0..3; higher is better. Effort in the denominator.
    return round((gap_closed + role_fit + shelf_life_yrs) / max(effort_weeks, 1), 2)

candidates = {
    "second_cloud_cert": dict(gap_closed=0, role_fit=1, effort_weeks=8,  shelf_life_yrs=3),
    "platform_cert":     dict(gap_closed=3, role_fit=3, effort_weeks=4,  shelf_life_yrs=3),
    "togaf":             dict(gap_closed=2, role_fit=1, effort_weeks=10, shelf_life_yrs=5),
}
THRESHOLD = 1.0
for name, s in candidates.items():
    roi = cert_roi(**s)
    verdict = "PURSUE" if roi >= THRESHOLD and s["gap_closed"] >= 2 else "SKIP"
    print(f"{name:18s} roi={roi:<5} -> {verdict}")
# second_cloud_cert  roi=0.5   -> SKIP
# platform_cert      roi=2.25  -> PURSUE
# togaf              roi=0.8   -> SKIP  (unless the target JD demands it)

Step-by-step explanation.

The rubric puts effort in the denominator, so a cheap cert that closes a real gap beats an expensive prestige cert. This is the correct optimisation for a transition on a clock.
The second cloud cert scores gap-closed = 0 (you are already cloud-literate) and thus fails the gap_closed >= 2 gate regardless of ROI. Redundant certs are the most common wasted quarter.
The platform cert scores highest: it closes a real modeling/performance gap on the target stack, matches the role strongly, and is low effort. Clear PURSUE.
TOGAF is the nuanced case: it scores PURSUE-worthy only if the target JD demands it (role_fit jumps to 3 for a bank). The rubric correctly makes it target-dependent rather than universally good or bad.
The gate gap_closed >= 2 is deliberate: no cert is worth pursuing if it does not close a genuine pillar gap, no matter how prestigious. Prestige is not a pillar.

Output.

Cert	ROI	Verdict	Reason
Second cloud cert	0.5	SKIP	closes no gap; redundant
Platform cert	2.25	PURSUE	closes modeling gap; strong role fit
TOGAF	0.8	SKIP unless JD demands	target-dependent

Rule of thumb. Run every candidate cert through the ROI rubric with a hard gate on gap-closed — if a cert does not fix a weak, hire-relevant pillar, skip it however prestigious it is. Two well-chosen certs plus a portfolio beats five trophies every time.

Senior interview question on certifications for architects

A senior interviewer might ask: "Do certifications actually matter for a data architect, or is it all portfolio? Walk me through how you'd decide which certs to invest in, what a cert proves versus what it doesn't, and how you'd spend a fixed six months if you were making the jump."

Solution Using a signal-vs-judgement framing plus a role-targeted, ROI-gated plan

Answer structure — "do certs matter?" in four moves
===================================================

Move 1 — what a cert IS (a signal, a floor)
  "A cert proves a floor and clears the recruiter screen. It says I know
   the services and the vocabulary. It does not prove I can make a
   trade-off, which is the actual architect job."

Move 2 — what a portfolio IS (judgement, decisive)
  "A portfolio proves judgement: reference designs, ADRs, a governed
   model, a cost model. That survives the technical panel. So I treat
   certs as necessary-not-sufficient and portfolio as decisive."

Move 3 — how I'd pick (two factors + a gate)
  "I pick certs by two factors: the pillar they close and the role they
   target, with a hard gate that they must fix a weak, hire-relevant
   pillar. A second cloud cert fails the gate; a platform cert on the
   target stack passes it."

Move 4 — the fixed six months (interleaved)
  "I'd interleave: months 1-2 close governance and cost with shippable
   artifacts, month 3 earn the one cloud/architect cert, months 4-6
   build two reference designs and an ADR set. Cert clears the screen;
   portfolio wins the panel."

# The decision rule I'd state out loud, made concrete.
def pick_certs(target_role: str, weak_pillars: set[str]) -> list[str]:
    picks = []
    if "cloud" in weak_pillars or "cost" in weak_pillars:
        picks.append("cloud data/architect cert on the primary cloud")
    if target_role == "enterprise" and "governance" in weak_pillars:
        picks.append("TOGAF / enterprise-architecture cert")
    if "modeling" in weak_pillars or target_role == "platform":
        picks.append("platform cert on the named stack (Snowflake/Databricks/dbt)")
    return picks[:2]   # cap at two; the rest of the effort is portfolio

print(pick_certs("scale-up", {"governance", "cost", "cloud"}))
# -> ['cloud data/architect cert on the primary cloud']  (+ portfolio for governance)
print(pick_certs("enterprise", {"governance", "cost"}))
# -> ['cloud data/architect cert on the primary cloud', 'TOGAF / enterprise-architecture cert']

Step-by-step trace.

Move	Claim	Evidence it signals
1	cert = floor/signal	you understand what a cert can't prove
2	portfolio = judgement	you know what the panel actually tests
3	pick by pillar + role, gated	disciplined, not trophy-hunting
4	interleave over six months	delivery-oriented, measurable
pick_certs()	cap at two	you stop before diminishing returns

The answer refuses the false binary ("certs vs portfolio") and instead sequences them: certs clear the screen, the portfolio wins the panel, and the picking rule is a disciplined two-factor gate rather than trophy accumulation. The pick_certs function makes the reasoning executable — the same target role and weak-pillar set always yield the same defensible pick, capped at two so the remaining effort flows to the portfolio.

Output:

Question probed	Weak answer	This answer
Do certs matter?	"yes, get all of them" / "no, useless"	"signal yes, judgement no"
How to pick	"the prestigious ones"	pillar + role, gated on gap
Cert vs portfolio	"certs" or "portfolio"	sequence both; portfolio decisive
Six-month spend	"study for exams"	interleave cert + shippable artifacts

Why this works — concept by concept:

Signal vs judgement — the core reframe. A cert is a signal (floor, screen-clearing); judgement is what the panel grades. Stating the distinction proves you understand what each instrument can and cannot do.
Two-factor pick with a gate — choosing by pillar-closed and role-fit, gated on fixing a weak hire-relevant pillar, is the discipline that separates a strategist from a collector. The gate is what kills the redundant second cloud cert.
Interleaving — pairing cert study with shippable artifacts means the knowledge lands in something demonstrable, and the timeline produces both the screen-clearing floor and the panel-winning portfolio.
Cap at two — diminishing returns are real; the third cert almost always has lower ROI than the first reference design. Naming the cap signals you optimise effort, an architect trait.
Cost — a cert is roughly 4–10 focused weeks; a portfolio artifact is 2–4 weeks each. The optimal allocation for a six-month transition is one cert (~one month of the six) and four-to-five artifacts (the rest) — because the panel outcome correlates with judgement evidence, not certificate count.

Design
Topic — design
Design problems that build a real portfolio

Practice →

ETL Topic — etl ETL problems for pipeline and integration depth

Practice →

4. Architecture interviews: blueprints & trade-offs

The architect design interview — scope, NFRs, a reference design you can defend, and the trade-offs behind it

The mental model in one line: the data architect interview is not "build the pipeline" — it is "scope the problem, attach non-functional requirements with real numbers, draw a reference design, and defend it on cost, latency, governance, and reliability trade-offs" — and the candidates who score highest lead with governance and cost, not with the cleverest data flow. The DE design interview asks can you build it?; the architect design interview asks can you decide it, defend it, and would twelve engineers succeed building against your blueprint?

The five-phase structure of every architect design interview.

Phase 1 — scope. Clarify the problem before drawing anything: who consumes this, what's the data volume, what's the freshness requirement, what compliance regime applies. Weak candidates start drawing; strong candidates start asking. "What's the SLA?" and "Is any of this PII?" are the first two questions.
Phase 2 — NFRs. Attach numbers: cost budget, latency target (p50/p99), availability (nines), governance requirements (retention, residency, access), scale (rows/day, growth rate). NFRs are what turn a doodle into an architecture.
Phase 3 — reference design. Draw the blueprint: sources → ingestion → storage layers → transformation → serving → consumers, with a governance overlay across the whole thing. Name the pattern (lakehouse, streaming, batch, medallion) and the concrete services.
Phase 4 — trade-offs. For every major choice, name the alternative and why you rejected it: batch vs streaming, managed vs self-hosted, one big warehouse vs domain marts. The trade-off narrative is the interview.
Phase 5 — risks & evolution. What breaks at 10× scale, what the migration path is, what the failure modes are, how you'd roll it out. Architects think in year-three terms.

The reference-architecture menu — the four you must be able to draw cold.

Batch warehouse (medallion / lakehouse). Sources → batch ingest → bronze (raw) → silver (cleaned/conformed) → gold (dimensional marts) → BI. The default for analytics; cheap, simple, minutes-to-hours freshness.
Streaming. Sources → event bus (Kafka/Kinesis/PubSub) → stream processor (Flink/Spark Structured Streaming) → serving store + lakehouse. For sub-minute freshness; more expensive and operationally heavier.
Lambda / Kappa hybrid. A speed layer (streaming) plus a batch layer (reprocessing) — or Kappa's single streaming path that replays for reprocessing. For use cases that need both real-time and correct-eventually.
Governed enterprise lakehouse. Any of the above with a mandatory governance overlay: catalog, column-level lineage, tag-based access control, data contracts between domains. The enterprise default; governance is a first-class layer, not an afterthought.

The NFR numbers to attach — memorise these ranges.

Latency. Batch = minutes to hours; micro-batch = seconds to minutes; streaming = sub-second to seconds. State p50 and p99, not just "fast."
Cost. Storage tiering (hot/warm/cold), compute right-sizing, and the biggest lever — scan reduction via partitioning and clustering. Always attach a rough monthly figure.
Availability. Three nines (99.9% ≈ 8.8h/yr down) is typical for analytics; four nines for serving. Do not over-engineer analytics to five nines.
Governance. Retention (e.g. 7 years for financial), residency (data stays in-region), access (least-privilege, tag-based), lineage (column-level, auditor-ready).

Worked example — a reference-architecture trade-off matrix

Detailed explanation. The single most useful interview artifact is a trade-off matrix comparing the reference architectures on the NFR axes. It proves you can choose rather than default, which is the architect skill under test.

The axes. Latency, cost, operational complexity, governance fit, best use case.
The move. When asked to design, put this matrix on the whiteboard, then justify which row you pick for the stated requirement.
The signal. Naming the axis you optimise for (and the one you sacrifice) is the trade-off fluency interviewers grade.

Question. Build the reference-architecture trade-off matrix and pick the right architecture for two requirements: hourly executive dashboards, and real-time fraud scoring.

Input.

Architecture	Latency	Cost	Ops complexity	Governance fit
Batch lakehouse	minutes–hours	low	low	high (easy to govern)
Streaming	sub-second–seconds	high	high	medium (harder lineage)
Lambda/Kappa hybrid	mixed	high	very high	medium
Governed enterprise lakehouse	minutes–hours	medium	medium	very high

Code.

Reference-architecture trade-off matrix
=======================================

Architecture             | Latency          | Cost | Ops  | Governance | Best for
-------------------------+------------------+------+------+------------+---------------------------
Batch lakehouse (medall.)| minutes-hours    | low  | low  | high       | analytics, BI, reporting
Streaming                | sub-second-secs  | high | high | medium     | real-time scoring/alerts
Lambda / Kappa hybrid    | mixed            | high | v.hi | medium     | real-time + reprocessing
Governed enterprise LH   | minutes-hours    | med  | med  | VERY HIGH  | regulated, multi-BU

Requirement A: hourly exec dashboards
  -> Batch lakehouse. Latency (hourly) is met by batch; optimise cost + governance,
     sacrifice real-time you don't need.

Requirement B: real-time fraud scoring
  -> Streaming. Latency (sub-second) is the hard NFR; accept high cost + ops,
     add a governance overlay for the PII in transaction data.

Step-by-step explanation.

The matrix makes the choice mechanical: read the requirement's hardest NFR, find the row that meets it at lowest cost/complexity, and name what you sacrifice. That sentence — "I optimise X and sacrifice Y" — is the trade-off fluency signal.
Requirement A (hourly dashboards) has a soft latency NFR (hourly), so streaming is over-engineering. Batch lakehouse meets the latency for a fraction of the cost and is the easiest to govern — the right call is the cheaper one, and saying so is senior.
Requirement B (fraud scoring) has a hard latency NFR (sub-second), which eliminates batch entirely. Streaming is mandatory; the architect accepts its high cost and ops burden because the requirement demands it, then adds governance for the PII.
The Lambda/Kappa row is deliberately rarely picked — it is the highest-complexity option and only justified when you genuinely need both real-time serving and periodic reprocessing. Reaching for it by default is an over-engineering tell.
The governed enterprise lakehouse row wins whenever the context is regulated or multi-business-unit, because governance jumps from a nice-to-have to the hardest NFR. Recognising when governance is the binding constraint is the top architect signal.

Output.

Requirement	Chosen architecture	Optimised for	Sacrificed
Hourly exec dashboards	batch lakehouse	cost, governance	real-time (not needed)
Real-time fraud scoring	streaming	latency	cost, ops simplicity
Regulated multi-BU platform	governed enterprise lakehouse	governance	some latency/cost
Real-time + reprocessing	Lambda/Kappa	both	operational simplicity

Rule of thumb. Put the trade-off matrix on the whiteboard, read the requirement's hardest NFR, pick the cheapest row that meets it, and say out loud what you optimise and what you sacrifice. Choosing the cheaper architecture when the requirement allows it is a stronger signal than reaching for the fanciest one.

Worked example — a whiteboard governed-lakehouse walkthrough

Detailed explanation. The most common architect prompt is "design a governed lakehouse." The winning approach is a repeatable skeleton you walk left-to-right: sources → ingest → medallion layers → serving → consumers, with a governance overlay across all of it, narrating NFRs and trade-offs as you go.

The skeleton. Five columns (sources, ingest, storage/medallion, transform/serve, consumers) plus a governance band on top.
The narration. At each column, state the choice, the alternative, and the NFR it satisfies.
The finish. End on governance and cost — the two pillars weakest candidates forget.

Question. Walk the governed-lakehouse skeleton for a mid-size company's analytics platform, naming the pattern and one trade-off per layer.

Input.

Layer	Choice	Alternative rejected
Ingest	CDC + batch files	full-refresh nightly (too slow, no deletes)
Storage	object store + open table format	proprietary warehouse only (lock-in)
Medallion	bronze/silver/gold	one flat layer (no reprocessing)
Serve	warehouse + BI + feature store	BI only (blocks ML)
Governance	catalog + lineage + tag access	ad-hoc grants (audit fails)

Code.

Governed lakehouse — whiteboard skeleton (left -> right)
=======================================================

[ SOURCES ]      [ INGEST ]        [ MEDALLION STORAGE ]         [ SERVE ]         [ CONSUMERS ]
 OLTP DBs   -->   CDC (log-based)   bronze (raw, immutable)  -->  warehouse   -->   BI / dashboards
 SaaS APIs  -->   API pulls    -->  silver (clean, conformed)-->  (gold marts)      ML / feature store
 events     -->   stream (Kafka)    gold (dimensional marts) -->  serving DB        data science

   ======================  GOVERNANCE OVERLAY (spans all layers)  ======================
   catalog + discovery | column-level lineage | tag-based access (PII/confidential) | data contracts | retention

NFRs stated per layer:
  ingest:  freshness -> CDC for sub-minute source sync; batch for slow SaaS
  storage: cost -> object store + open table format (Iceberg/Delta) avoids lock-in
  serve:   latency -> gold marts pre-aggregated for BI p99 < 2s
  govern:  audit -> column lineage + 7-yr retention + residency in-region

Step-by-step explanation.

Start at sources and move strictly left-to-right so the interviewer can follow — never jump around the diagram. Name the ingest pattern per source: CDC (log-based) for OLTP freshness, API pulls for SaaS, a stream for events. Stating why CDC over nightly full-refresh (deletes + freshness) is the trade-off signal.
Storage uses an object store plus an open table format (Iceberg/Delta) rather than a proprietary warehouse-only design — the trade-off is a little more setup for dramatically less vendor lock-in, an architect-level cost/portability call.
The medallion layers (bronze raw/immutable, silver cleaned/conformed, gold dimensional marts) exist so you can reprocess from bronze when logic changes — the alternative (one flat layer) makes reprocessing impossible. This is where the modeling pillar meets the architecture pillar.
Serving splits into a warehouse for BI and a feature store for ML, because BI-only serving blocks the data-science consumers — an architect designs for all consumers, not just dashboards.
The governance overlay is drawn last but emphasised most: catalog, column-level lineage, tag-based access for PII/confidential, data contracts between domains, and retention/residency. Ending on governance and cost is the deliberate move that separates the architect answer from the DE answer.

Output.

Layer	Pattern	NFR satisfied
Ingest	CDC + batch + stream	freshness per source
Storage	object store + open table format	cost, no lock-in
Medallion	bronze/silver/gold	reprocessing, quality
Serve	warehouse + feature store	latency, all consumers
Governance	catalog + lineage + tag access	audit, compliance

Rule of thumb. Walk the skeleton left-to-right, state one trade-off and one NFR per layer, and finish on the governance overlay and the cost envelope. If you run out of time, cut the middle detail — never cut governance and cost, because those are the two pillars the panel is specifically checking you don't forget.

Worked example — a partitioning + row-access governance snippet

Detailed explanation. Architect design answers land better with one concrete artifact that proves you can go from blueprint to enforceable policy. The best two-for-one is a partitioning scheme (the biggest cost lever) plus a row-access policy (governance made executable) on the same table.

Partitioning. Partition the fact table by date so queries scan only the window they need — the single biggest cost reduction in a warehouse.
Row-access policy. Restrict rows by the querying user's business unit — governance enforced at query time, not by hoping analysts filter correctly.
The payoff. One snippet demonstrates both the cost and governance pillars concretely.

Question. Write the partitioning DDL and a row-access policy that limits each business unit to its own rows.

Input.

Control	Purpose	Pillar
Partition by event_date	scan reduction / cost	cost & performance
Cluster by customer_id	co-locate hot filters	performance
Row-access policy by BU	least-privilege at query time	governance

Code.

-- 1. Partitioning: the biggest cost lever. Scan only the queried window.
CREATE TABLE fact_events (
    event_key    BIGSERIAL,
    event_date   DATE        NOT NULL,   -- partition key
    business_unit TEXT       NOT NULL,   -- governance key
    customer_id  BIGINT,
    amount_cents BIGINT
) PARTITION BY RANGE (event_date);

CREATE TABLE fact_events_2026_08 PARTITION OF fact_events
  FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');
-- A query with WHERE event_date >= '2026-08-01' prunes to one partition.

-- 2. Row-access governance policy (Snowflake-style syntax shown for clarity):
--    each user only sees rows for their own business unit.
CREATE ROW ACCESS POLICY bu_isolation AS (bu TEXT) RETURNS BOOLEAN ->
    bu = CURRENT_ROLE()                    -- role name == business unit
    OR IS_ROLE_IN_SESSION('PLATFORM_ADMIN'); -- admins see everything

ALTER TABLE fact_events
    ADD ROW ACCESS POLICY bu_isolation ON (business_unit);

-- 3. The cost proof: a governed, partitioned query scans one month, one BU.
SELECT business_unit, SUM(amount_cents) AS revenue
FROM   fact_events
WHERE  event_date >= '2026-08-01' AND event_date < '2026-09-01'  -- partition prune
GROUP  BY business_unit;
-- Analyst in role FINANCE_BU sees only FINANCE_BU rows (row-access policy);
-- scans ~1/N of the table (partition pruning). Cost + governance in one query.

Step-by-step explanation.

Partitioning by event_date is the biggest cost lever in the warehouse: a query filtered to August scans one monthly partition instead of the whole table, cutting scan cost (and bill) by the ratio of window to table.
The row-access policy bu_isolation enforces least-privilege at query time — an analyst in the FINANCE_BU role physically cannot read other business units' rows, regardless of how they write the query. Governance you can bypass by forgetting a WHERE clause is not governance.
The PLATFORM_ADMIN escape hatch is deliberate and auditable: admins see everything, but the policy names exactly who and why. Architects design the exception explicitly rather than leaving a silent backdoor.
Steps 1 and 2 compose: the final query prunes to one partition (cost) and filters to one business unit (governance) automatically. One artifact demonstrates both pillars, which is exactly what an architect answer should do.
Stating the numbers out loud — "this prunes scan cost to roughly 1/N and enforces BU isolation the analyst cannot bypass" — is the difference between drawing a box labelled "governance" and proving the control works.

Output.

Query context	Rows scanned	Rows visible
FINANCE_BU analyst, August filter	one partition	FINANCE_BU only
MARKETING_BU analyst, August filter	one partition	MARKETING_BU only
PLATFORM_ADMIN, full scan	all partitions	all rows
No date filter (anti-pattern)	all partitions	still BU-scoped

Rule of thumb. Back every architecture answer with one enforceable artifact — partitioning for cost, a row-access or masking policy for governance — on the same table. A blueprint with a concrete, bypass-proof control reads as an architect; a blueprint of unlabelled boxes reads as a diagram.

Senior interview question on designing a governed enterprise lakehouse

A senior interviewer might ask: "Design a governed enterprise lakehouse for five business units sharing one platform. Cover the scope questions you'd ask, the NFRs you'd attach, the reference design, how you'd isolate and govern the five BUs, the cost model, and the one trade-off you'd flag as the biggest risk."

Solution Using a medallion lakehouse with tag-based governance, BU isolation, and a cost envelope

Phase 1 — SCOPE (ask before drawing)
====================================
  - Who consumes: BI, ML, and each BU's analysts? -> multi-consumer serving
  - Data volume + growth: rows/day, YoY growth? -> sizing + partitioning
  - Freshness per source: which need sub-minute, which nightly? -> ingest mix
  - Compliance: PII? residency? retention? audit? -> governance overlay
  - Isolation: must BUs be prevented from seeing each other's data? -> row-access

Phase 2 — NFRs (attach numbers)
===============================
  - Latency:   BI gold marts p99 < 2s; ingest freshness CDC < 1 min, SaaS hourly
  - Cost:      partition + tier to keep scan cost ~1/N; rough $X/month envelope
  - Availability: 99.9% for analytics serving
  - Governance: column lineage, 7-yr retention, in-region residency, tag-based PII

Phase 3 — REFERENCE DESIGN (medallion + governance overlay)
===========================================================
 sources -> ingest(CDC + API + stream) -> bronze -> silver(conformed) -> gold(BU marts) -> serve(BI+ML)
 GOVERNANCE OVERLAY: catalog | column lineage | tag-based access (PII/confidential) | data contracts | retention

Phase 4 — BU ISOLATION + GOVERNANCE (the crux)
==============================================
  - One physical platform, logical isolation via a business_unit column + row-access policy
  - Shared conformed dimensions (dim_date, dim_customer) so cross-BU reporting is possible
  - Per-BU gold marts; a platform-admin role for cross-BU governance

-- Phase 4 made concrete: tag-based classification + row-access + partitioning
COMMENT ON COLUMN silver.customers.email IS 'classification=PII';   -- tag
CREATE ROW ACCESS POLICY bu_isolation AS (bu TEXT) RETURNS BOOLEAN ->
    bu = CURRENT_ROLE() OR IS_ROLE_IN_SESSION('PLATFORM_ADMIN');
ALTER TABLE gold.fact_revenue ADD ROW ACCESS POLICY bu_isolation ON (business_unit);

-- Phase 2 cost lever made concrete: partition gold facts by month
ALTER TABLE gold.fact_revenue SET PARTITION BY RANGE (date_key);   -- scan ~1/N

Step-by-step trace.

Phase	Decision	Reasoning
Scope	ask 5 questions first	never draw before you know the SLA and PII
NFRs	attach numbers	p99 < 2s, 99.9%, 7-yr retention, in-region
Reference design	medallion + governance overlay	reprocessing + audit-ready
BU isolation	one platform, row-access policy	shared cost, logical isolation
Conformed dims	shared dim_date/dim_customer	cross-BU reporting stays possible
Cost	partition + tier	scan cost ~1/N of table
Biggest risk flagged	governance drift across BUs	needs contracts + a platform-admin owner

The answer scopes before drawing, attaches concrete NFR numbers, draws a medallion lakehouse with a first-class governance overlay, and solves the crux — five BUs on one platform — with logical isolation (a business_unit column plus a bypass-proof row-access policy) over shared physical infrastructure and shared conformed dimensions. It ends by naming the single biggest risk (governance drift across autonomous BUs) and its mitigation (enforced data contracts and a named platform-admin owner), which is the year-three thinking architects are graded on.

Output:

Requirement	How the design meets it
5 BUs, one platform	shared infra + row-access isolation
Cross-BU reporting	shared conformed dimensions
PII compliance	tag-based classification + masking
Cost efficiency	partitioning + storage tiering (scan ~1/N)
Audit readiness	column lineage + 7-yr retention
Biggest risk	governance drift → contracts + admin owner

Why this works — concept by concept:

Scope before design — asking the five scope questions (consumers, volume, freshness, compliance, isolation) before drawing is the phase weak candidates skip. The answer's quality is bounded by the questions you ask first.
NFRs with numbers — p99 < 2s, 99.9%, 7-year retention, in-region residency. Numbers turn a doodle into an architecture and give the panel something concrete to probe, which is what you want.
Logical isolation over physical — one platform with a row-access policy and a business_unit column shares cost across BUs while enforcing least-privilege at query time. Separate physical platforms per BU would multiply cost and fragment governance.
Conformed dimensions — shared dim_date/dim_customer keep cross-BU reporting coherent, so isolation does not preclude enterprise-wide analytics. Balancing isolation with conformance is the enterprise-architecture skill.
Cost — partitioning cuts scan cost to roughly O(window/table) instead of O(table); storage tiering pushes cold data to cheap storage; one shared platform amortises infra across five BUs. The design is defensible on a monthly dollar figure, which is what gets it funded — and naming the governance-drift risk shows you cost the operational future, not just the build.

Design
Topic — design
Design problems on lakehouse and platform architecture

Practice →

Indexing Topic — indexing Indexing and partitioning problems for performance

Practice →

5. Signals architects are graded on & the transition plan

The four signals, the portfolio that proves them, and the 18-month roadmap from DE to architect

The mental model in one line: an architect interview grades four signals — breadth across the pillars, trade-off fluency, governance-first thinking, and communication — and you make those signals legible with a portfolio (reference designs, ADRs, a governed model, a cost model) and a deliberate 18-month transition roadmap, because the role is won by demonstrated judgement, not claimed seniority. The previous four sections taught the pillars; this one is about proving you have them.

The four signals interviewers actually score.

Breadth. Can you reason across all five pillars in one answer, or do you retreat to the one you know? The tell is an answer that touches modeling and governance and cost in a single design, versus one that goes deep on ingestion and forgets the rest.
Trade-off fluency. Do you name the alternative and why you rejected it, in numbers? "I chose batch because the SLA is hourly and it's a third the cost of streaming" scores; "I chose batch because it's simpler" does not.
Governance-first thinking. Do you raise PII, access, lineage, and retention before being asked? The single strongest differentiator, because it is the pillar DEs forget and the one that ends up in a regulator's inbox.
Communication. Can you explain a trade-off to a mixed audience and defend it without defensiveness? The blueprint is worthless if the people funding and building it don't understand it.

The portfolio that makes the signals legible — five artifacts.

Reference designs (2–3). Governed lakehouse, streaming, one hybrid — each with NFRs, a trade-off matrix, and a cost envelope. Proves breadth and trade-off fluency.
ADRs (5+). Architecture decision records for real decisions, each with context, options, decision, and consequences. Proves communication and judgement over time.
A governed data model. One domain modeled end-to-end (3NF source → star + SCD Type 2) with classification, contracts, and lineage. Proves modeling and governance together.
A cost model. A spreadsheet or doc costing one real workload, with tiering and right-sizing recommendations. Proves the cost pillar concretely.
A design talk or RFC. Evidence you can lead a design conversation and get a decision funded. Proves communication and stewardship.

The transition roadmap — 18 months, four phases.

Months 0–3 — close the gap. Fix the two weakest pillars (usually governance and cost) with shippable artifacts. Score yourself; attack the biggest gap first.
Months 3–9 — build the portfolio. Ship reference designs and ADRs on real decisions at work. Volunteer for the cross-team design nobody owns.
Months 9–15 — credential + broaden. Earn the one role-targeted cert; broaden the weakest remaining pillar; get a governed model and cost model into the portfolio.
Months 15–18 — lead + interview. Lead a real design end-to-end, present it, then interview with a portfolio that proves all four signals.

Worked example — the architect signal scoring rubric

Detailed explanation. The signal rubric is what an interviewer actually fills in, so knowing it lets you engineer your answers to hit every box. Each signal is scored on whether it appeared unprompted, when prompted, or not at all.

The scale. 2 = raised unprompted, 1 = present when asked, 0 = absent.
The bar. An offer typically needs a 2 on governance-first and trade-off fluency, and at least a 1 on breadth and communication.
The lever. You can move governance and trade-offs from 1 to 2 simply by raising them before being asked — a free point.

Question. Fill in the signal rubric with the behaviour that earns each score, and name the two free points most candidates leave on the table.

Input.

Signal	Score 0	Score 1	Score 2
Breadth	one pillar only	covers most when asked	integrates all in one answer
Trade-off fluency	"it's simpler"	names alternative when asked	names alternative + numbers unprompted
Governance-first	never mentions	mentions when asked	raises PII/access/lineage first
Communication	jargon, defensive	clear when pushed	leads the room, invites challenge

Code.

Architect signal scoring rubric  (interviewer's actual sheet)
============================================================

Signal            | 0 (absent)        | 1 (when asked)          | 2 (unprompted)
------------------+-------------------+-------------------------+---------------------------
Breadth           | one pillar        | most pillars if probed  | all pillars in one design
Trade-off fluency | "it's simpler"    | alt named when asked    | alt + numbers, unprompted
Governance-first  | never raised      | raised when asked       | PII/access/lineage FIRST
Communication     | jargon/defensive  | clear under pushback    | leads + invites challenge

Offer bar: governance-first = 2, trade-off = 2, breadth >= 1, communication >= 1

The two FREE points most candidates leave on the table:
  1. Governance-first: raise PII/retention/access BEFORE the interviewer asks -> +1
  2. Trade-off fluency: attach a NUMBER to every choice -> +1

Step-by-step explanation.

The rubric scores when a signal appears, not just whether — unprompted (2) beats prompted (1). This is the mechanism behind "lead with governance": you are literally converting a 1 into a 2 by raising it first.
Governance-first is the highest-leverage signal because most candidates score 0–1 (they wait to be asked or never mention it), so raising PII/access/lineage unprompted is an immediate differentiator and a free point.
Trade-off fluency's jump from 1 to 2 is "attach a number." "Batch because it's simpler" is a 1; "batch because the SLA is hourly and it's ~1/3 the cost of streaming" is a 2. The number is the whole difference.
Breadth is scored by integration: touching all five pillars in one design answer (2) versus going deep on one (0). You engineer this by consciously running the pillar checklist — modeling, governance, integration, cost, cloud — through every design answer.
Communication is scored on leadership and openness: inviting challenge ("push back on this — where would you deviate?") reads as a 2, defensiveness reads as a 0. Architects lead rooms; they do not defend turf.

Output.

Signal	Free-point move	Score delta
Governance-first	raise PII/access/lineage unprompted	1 → 2
Trade-off fluency	attach a number to every choice	1 → 2
Breadth	run the 5-pillar checklist per answer	0 → 2
Communication	invite challenge on your own design	1 → 2

Rule of thumb. Engineer your answers against the rubric: raise governance first, attach a number to every trade-off, run the five-pillar checklist through every design, and invite challenge. Two of those are free points that most candidates simply forget to take.

Worked example — a transition-roadmap doc

Detailed explanation. The transition roadmap turns "I want to be an architect" into a dated, milestoned plan with portfolio checkpoints. Eighteen months, four phases, a deliverable per phase — reviewable by a mentor and updatable each quarter.

The phases. Close-the-gap → build-the-portfolio → credential-and-broaden → lead-and-interview.
The checkpoints. Each phase ends with a portfolio artifact and a re-score of the six pillars.
The discipline. No phase starts before the previous phase's artifact ships.

Question. Write the 18-month DE-to-architect roadmap for a strong DE weak on governance and cost.

Input.

Phase	Months	Deliverable	Signal advanced
Close the gap	0–3	contract standard + cost model	governance, cost
Build portfolio	3–9	2 reference designs + 5 ADRs	breadth, trade-offs, comms
Credential + broaden	9–15	cloud cert + governed model	cloud, modeling
Lead + interview	15–18	lead a design + interview	communication, all

Code.

# DE -> Data Architect Transition Roadmap (18 months)

## Phase 1 (months 0-3) — CLOSE THE GAP
- [ ] Self-score the six pillars; identify the two weakest (here: governance, cost)
- [ ] Ship a data-contract + PII-classification standard for one domain
- [ ] Ship a cost model for one workload (tiering + right-sizing)
- CHECKPOINT: governance & cost self-score 1 -> 3; two portfolio artifacts done

## Phase 2 (months 3-9) — BUILD THE PORTFOLIO
- [ ] Volunteer for the cross-team design nobody owns
- [ ] Ship 2 reference designs (governed lakehouse + streaming) with NFRs + cost
- [ ] Write 5 ADRs on real decisions (context / options / decision / consequences)
- CHECKPOINT: breadth + trade-off signals demonstrable in artifacts

## Phase 3 (months 9-15) — CREDENTIAL + BROADEN
- [ ] Earn one role-targeted cert (cloud/architect on the primary cloud)
- [ ] Model one domain end-to-end (3NF -> star + SCD2 + classification)
- [ ] Broaden the weakest remaining pillar to a 3
- CHECKPOINT: six-pillar score 3+ across the board, one 4

## Phase 4 (months 15-18) — LEAD + INTERVIEW
- [ ] Lead one real design end-to-end and present it (design talk / RFC)
- [ ] Assemble the portfolio: designs, ADRs, governed model, cost model
- [ ] Interview with a portfolio that proves all four signals
- CHECKPOINT: offer, or a concrete gap list for the next cycle

Step-by-step explanation.

Phase 1 attacks the two weakest pillars first with shippable artifacts, because the transition is bottlenecked by the gap, not by the strengths. The checkpoint is a re-score plus two portfolio pieces — measurable, not vague.
Phase 2 is the longest (six months) because portfolio depth is what wins panels. Volunteering for the unowned cross-team design is the single highest-leverage move — it produces a real reference design and the cross-team scope that reads as architect work.
Phase 3 slots the cert in after the portfolio has momentum, so it certifies real knowledge, and adds the two proof-of-modeling-and-governance artifacts (governed model). The checkpoint is the 3+ across all six pillars.
Phase 4 is about demonstrated leadership: leading and presenting a real design is the capstone that proves the communication signal, then interviewing with the full portfolio. Even a "no" ends with a concrete gap list, so the cycle compounds.
The gating discipline — no phase starts before the prior artifact ships — is what stops the roadmap from becoming a wish list. Architects finish things; the roadmap enforces that on the transition itself.

Output.

Phase	Exit checkpoint	Portfolio state
Close the gap	governance/cost → 3	contract standard + cost model
Build portfolio	breadth demonstrable	2 designs + 5 ADRs
Credential + broaden	3+ across all pillars	+ cert + governed model
Lead + interview	offer or gap list	full portfolio, presented

Rule of thumb. Make the transition a dated, gated roadmap with a portfolio artifact at every checkpoint, and re-score the six pillars each quarter. A transition without milestones drifts for years; a transition with them typically completes in twelve-to-eighteen months.

Worked example — an ADR (architecture decision record) template

Detailed explanation. ADRs are the highest-ROI portfolio artifact because they prove judgement over time and cost almost nothing to write. Each records one decision with its context, the options considered, the decision, and the consequences — the exact shape of architect thinking made durable.

The shape. Context → options (with trade-offs) → decision → consequences (including the downside you accepted).
The value. A stack of ADRs is a portable proof that you make reasoned trade-offs, not just technical choices.
The tell. An ADR that omits the rejected options or the accepted downside is a decision log, not an ADR.

Question. Write an ADR for the decision to use log-based CDC over nightly full-refresh, in the standard template.

Input.

ADR field	Content
Context	need sub-minute freshness + deletes to warehouse
Options	full-refresh, timestamp CDC, log-based CDC
Decision	log-based CDC (Debezium)
Consequences	+freshness/deletes; −ops (slot monitoring)

Code.

# ADR-014: Use log-based CDC for warehouse ingestion

## Status
Accepted (2026-08-04)

## Context
The warehouse needs sub-minute freshness and must capture physical deletes for
audit. The current nightly full-refresh is 24h stale and silently drops deletes.
Data volume rules out re-scanning the source table frequently.

## Options considered
1. Nightly full-refresh (status quo) — simple; 24h stale; blind to deletes. REJECTED.
2. Timestamp CDC (poll updated_at) — cheap, portable; blind to physical deletes
   without a reconcile; latency floored by poll interval. REJECTED for the delete gap.
3. Log-based CDC (Debezium + pgoutput) — sub-second; captures deletes; ~0 source
   load; needs wal_level=logical + slot monitoring. SELECTED.

## Decision
Adopt log-based CDC via Debezium on a logical replication slot, publishing to Kafka.

## Consequences
+ Sub-second freshness; native delete capture; near-zero source-table load.
- New operational burden: replication-slot lag monitoring + disk-full defense
  (max_slot_wal_keep_size). Requires DBA to grant wal_level=logical.
- Accepted downside: on-call must own slot-lag alerts from day one.

Step-by-step explanation.

The Status + Context fields frame why the decision was needed — sub-minute freshness plus delete capture — so a future reader (or an interviewer) understands the constraint before the choice.
The Options section lists all three candidates with their trade-offs and an explicit REJECTED/SELECTED. This is the heart of the ADR and the proof of trade-off fluency — an ADR that lists only the chosen option proves nothing.
Each rejected option names why it lost: full-refresh is stale and delete-blind; timestamp CDC is delete-blind without a reconcile. Naming the specific disqualifier is the architect signal.
The Consequences section records both the upside and the accepted downside — new slot-monitoring burden, a DBA grant dependency. Owning the downside in writing is what separates an ADR from marketing.
A stack of ten such ADRs is a portable, dated proof of reasoned judgement — the single most convincing portfolio artifact because it shows how you think, decision after decision, not just what you built.

Output.

ADR section	Proves
Context	you scope before deciding
Options + trade-offs	trade-off fluency, breadth
Decision	you commit to a choice
Consequences (incl. downside)	honesty, year-three thinking

Rule of thumb. Write an ADR for every non-trivial decision, always including the rejected options and the downside you accepted. Ten ADRs are worth more in a panel than any certificate, because they prove the one thing a cert cannot: how you make trade-offs.

Senior interview question on critiquing and improving an architecture

A senior interviewer might ask: "Here's an existing architecture: a nightly full-refresh from Postgres to a single flat warehouse table, analysts querying it directly with ad-hoc grants, no lineage. Critique it and propose a better one. I'm grading how you find the weaknesses, prioritise them, and defend your redesign."

Solution Using a signal-ordered critique (governance-first) plus a medallion redesign with a migration path

Critique — ordered by the signals I'm graded on
===============================================

1. GOVERNANCE (raise first) — ad-hoc grants + no lineage = audit failure waiting
   to happen; no PII classification; no retention policy. HIGHEST priority.
2. CORRECTNESS — nightly full-refresh is 24h stale AND blind to deletes; the flat
   table has no history (any overwrite loses the past).
3. COST — full-refresh re-scans the whole source nightly; the flat table scans
   everything on every query (no partitioning). Expensive and slow.
4. MODELING — one flat table means no conformed dimensions, no grain discipline,
   duplicated logic in every analyst query.

Redesign — medallion lakehouse with governance overlay
======================================================
 Postgres --(log-based CDC)--> bronze(raw) --> silver(conformed) --> gold(star marts) --> BI
 GOVERNANCE OVERLAY: catalog | column lineage | tag-based access (PII) | data contracts | 7-yr retention

Key changes vs the old design:
  - full-refresh  -> log-based CDC        (freshness + deletes)
  - flat table    -> medallion + star     (history, grain, reprocessing)
  - ad-hoc grants -> tag-based row access  (governance, least-privilege)
  - no lineage    -> column-level lineage  (audit-ready)

-- One concrete governance + cost improvement in the redesign:
-- classify PII, enforce row access, and partition the gold fact.
COMMENT ON COLUMN silver.customers.email IS 'classification=PII';
CREATE ROW ACCESS POLICY bu_isolation AS (bu TEXT) RETURNS BOOLEAN ->
    bu = CURRENT_ROLE() OR IS_ROLE_IN_SESSION('PLATFORM_ADMIN');
ALTER TABLE gold.fact_sales ADD ROW ACCESS POLICY bu_isolation ON (business_unit);
-- gold.fact_sales PARTITION BY RANGE (date_key)  -> scan ~1/N instead of full table

Step-by-step trace.

Step	Move	Signal demonstrated
Critique order	governance first, then correctness/cost/modeling	governance-first thinking
Prioritisation	rank by risk (audit) not by ease	judgement
Redesign	medallion + CDC + governance overlay	breadth
Migration	run both in parallel, cut over per consumer	year-three thinking
Concrete artifact	classification + row access + partition	trade-off made executable
Downside owned	CDC adds slot-monitoring ops	honesty

The critique leads with governance (the highest-risk weakness and the highest-scoring signal), prioritises by business risk rather than by what is easiest to fix, and proposes a medallion lakehouse with log-based CDC and a first-class governance overlay. It grounds the redesign in one concrete, bypass-proof artifact (PII classification + row-access policy + partitioning) and names the migration path (parallel-run, per-consumer cutover) plus the accepted downside (new slot-monitoring burden) — the full architect signal set in one answer.

Output:

Weakness	Old design	Redesign
Governance	ad-hoc grants, no lineage	tag-based access + column lineage
Freshness / deletes	24h stale, delete-blind	log-based CDC, sub-minute
History	flat overwrite	medallion + SCD Type 2
Cost	full scan per query	partitioned, scan ~1/N
Modeling	one flat table	conformed star marts

Why this works — concept by concept:

Governance-first critique order — leading with the ad-hoc-grants/no-lineage weakness raises the highest-risk issue first and scores the top signal unprompted. Critiquing correctness before governance would leave the free point on the table.
Risk-based prioritisation — ranking weaknesses by business risk (audit failure) rather than by fix difficulty is the judgement signal. Architects triage by consequence, not by convenience.
Medallion redesign — bronze/silver/gold plus a governance overlay fixes history, reprocessing, quality, and audit in one coherent pattern, demonstrating breadth across the pillars.
Migration path — proposing a parallel-run with per-consumer cutover proves you think about how the change lands, not just the target state — the year-three thinking that separates architects from designers.
Cost — the redesign trades a new operational burden (slot monitoring) for large wins: partitioning cuts query scan to O(window) instead of O(table), CDC eliminates the nightly full-scan, and governance moves from O(manual audit) to O(automated lineage). Owning the downside in the same breath as the upside is the honesty the rubric rewards.

Design
Topic — design
Design-critique and redesign problems

Practice →

Dimensional modeling
Topic — dimensional-modeling
Modeling problems for the architect portfolio

Practice →

Cheat sheet — DE → architect recipes

The lane change in one line. The data engineer owns the implementation; the data architect owns the blueprint. You move from the R (responsible, does the work) column to the A (accountable, owns the outcome) column — and inherit the cost envelope and cross-team consistency that nobody else is positioned to hold. If you cannot name three responsibilities that flip from you do it to you own it, you are interviewing for senior-DE with an architect label.
The six-pillar skill ladder. Data modeling, governance & security, integration patterns, cost & performance, cloud platforms, and communication/stewardship. The hire bar is 3+ across all six with at least one 4 — a spike in modeling with a hole in governance reads as specialist, not architect. Score yourself, sort by gap, fix the biggest gap first.
Modeling signals. Star schema (dimensional) for read-heavy analytics; 3NF for write-heavy operational and silver-layer integrity; Data Vault for auditable, multi-source, regulated environments. Choose the grain first, give every dimension a surrogate key, denormalize lookups into dimensions, keep measures additive, and use SCD Type 2 (versioned surrogate + validity window + is_current) whenever dimension history matters.
Governance-first signals. Catalog + column-level lineage + tag-based access (PII/confidential) + data contracts (enforced in CI) + retention/residency. Governance is the pillar DEs score lowest on and interviewers weight highest — raise PII, access, lineage, and retention before being asked. It is a free point on the rubric.
Certifications — the two-factor pick. Choose certs by the pillar they close and the role they target, with a hard gate that they fix a weak, hire-relevant pillar. Cloud data/architect cert for lakehouse roles; TOGAF/enterprise-architecture for large regulated orgs; platform cert (Snowflake/Databricks/dbt) when the JD names the stack. Cap at two, then pivot to portfolio.
Cert vs portfolio verdict. A cert proves a floor and clears the recruiter screen; a portfolio proves judgement and wins the panel. Certs are necessary-not-sufficient for some roles and optional for others; a portfolio is close to universally decisive. Spend the minimum cert effort to clear the screen, then over-invest in the portfolio.
The architecture interview — five phases. Scope (ask before drawing) → NFRs (attach numbers: p99 latency, cost/month, 99.9% availability, retention/residency) → reference design (medallion + governance overlay) → trade-offs (name the alternative + why rejected, in numbers) → risks & evolution (what breaks at 10×, the migration path). Weak candidates start drawing; strong candidates start asking.
The reference-architecture menu. Batch lakehouse (cheap, minutes-hours, easy to govern); streaming (sub-second, expensive, harder lineage); Lambda/Kappa hybrid (both, highest complexity — rarely the right default); governed enterprise lakehouse (any of the above with a mandatory governance overlay — the enterprise default). Pick the cheapest row that meets the hardest NFR.
NFR numbers to memorize. Latency: batch = minutes-hours, micro-batch = seconds-minutes, streaming = sub-second-seconds. Availability: 99.9% for analytics, 99.99% for serving — don't over-engineer analytics to five nines. Cost: partitioning is the biggest scan-reduction lever (~1/N); tier hot/warm/cold; always attach a monthly figure.
Enforceable-artifact reflex. Back every architecture answer with one concrete control: partitioning by date (cost — scan ~1/N) plus a row-access or masking policy tied to a classification tag (governance — bypass-proof at query time). A blueprint of unlabelled boxes reads as a diagram; a blueprint with an enforceable control reads as an architect.
The four graded signals. Breadth (all pillars in one answer), trade-off fluency (alternative + a number), governance-first (raise it unprompted), communication (lead the room, invite challenge). Offer bar: 2 on governance-first and trade-offs, 1+ on breadth and communication. The two free points most candidates forget: raise governance first, and attach a number to every trade-off.
The portfolio — five artifacts. 2–3 reference designs (with NFRs + cost), 5+ ADRs (context/options/decision/consequences — always include the rejected options and the accepted downside), one governed data model (3NF → star + SCD2 + classification), one cost model, and one design talk/RFC. Ten ADRs beat any certificate in a panel because they prove how you decide.
The 18-month roadmap. Months 0–3 close the two weakest pillars (usually governance + cost) with shippable artifacts; 3–9 build the portfolio (volunteer for the unowned cross-team design); 9–15 credential + broaden to 3+ across all pillars; 15–18 lead a real design, present it, and interview. Gate each phase on shipping the prior artifact — a transition without milestones drifts for years.

Frequently asked questions

What does a data architect do?

A data architect owns the blueprint of an organisation's data systems — the reference architectures, the modeling and governance standards, the integration patterns, and the cost envelope that engineering teams build against — while being deliberately one level removed from the day-to-day pipeline code. Where a data engineer is accountable for a component (a pipeline, a table), the architect is accountable for how components fit into a coherent system that survives scaling, multiple business units, and governance audits. The role spans six competencies: data modeling, governance and security, integration patterns, cost and performance, cloud platforms, and the communication that makes those decisions legible to engineers, product, finance, and compliance. In practice the architect spends their time on reference designs, data-contract and modeling standards, build-vs-buy decisions, and defending trade-offs to a mixed audience — not on being the on-call hero for any single pipeline.

Data engineer vs data architect — what's the difference?

The core difference is accountability versus responsibility: the data engineer is responsible (does the work — builds the DAG, writes the table) while the data architect is accountable (owns the outcome — sets the standard the DAG conforms to, decides whether the data belongs in the warehouse at all). A data engineer optimises for the sprint and the incident on a stack they know deeply; a data architect optimises for the three-year horizon across a breadth of domains — modeling, governance, integration, cost, cloud. The engineer talks to other engineers and a manager; the architect talks to engineers, product, finance, and often the C-suite, because the blueprint only matters if the people funding and building it both understand it. The most common failed transition is the engineer who takes the architect title but keeps doing the engineer's job, leaving the pure-architect responsibilities (cost envelope, cross-team consistency) unowned.

Which certifications should a data architect get?

The certifications that move the needle fall into three families, and the right pick depends on your target role, not on prestige. Cloud data/architect certifications (the major-cloud data-engineer and solutions/data-architect tracks) prove the integration, cost, and cloud pillars and have the highest ROI for cloud-lakehouse and platform roles. Enterprise-architecture certifications (TOGAF and equivalents) prove the blueprint vocabulary large regulated organisations expect and are often a literal checkbox in enterprise-architect job descriptions. Platform certifications (Snowflake, Databricks, dbt) prove modeling and performance depth and are worth it when the job description names that stack. Pick by two factors only — the pillar the cert closes and the role it targets — with a hard gate that it must fix a weak, hire-relevant pillar, and cap yourself at two before pivoting your remaining effort to a portfolio.

Do you need certifications or a portfolio to become a data architect?

You need both, but they do different jobs and a portfolio is the decisive one. A certification proves a floor — it says you know the services and the vocabulary, and it clears the recruiter screen and gives a non-technical gatekeeper a checkbox — but it never proves you can make a trade-off, which is the actual architect job. A portfolio proves judgement: reference designs with NFRs and cost envelopes, ADRs documenting real decisions with their rejected alternatives, a governed data model, and a cost model. The portfolio is what survives the technical panel, where the offer is decided. The optimal strategy is to spend the minimum certification effort to clear the screen (one, maybe two role-targeted certs) and then over-invest in the portfolio artifacts that prove the judgement no exam can certify.

How do you prepare for the data architect interview?

Prepare for the architecture design interview as a five-phase structure rather than a build exercise: scope the problem (ask about consumers, volume, freshness, and compliance before drawing anything), attach NFRs with real numbers (p99 latency, monthly cost, 99.9% availability, retention and residency), draw a reference design (medallion lakehouse with a governance overlay is the safe default), defend every major choice with its rejected alternative and a number, and close on risks and the migration path. The interview grades four signals — breadth across the pillars, trade-off fluency, governance-first thinking, and communication — so engineer your answers to hit each: run the five-pillar checklist through every design, attach a number to every trade-off, and raise PII, access, and lineage before being asked. Rehearse a governed-lakehouse whiteboard skeleton and a reference-architecture trade-off matrix until you can draw them cold, and bring a portfolio you can screen-share.

How long does it take to go from data engineer to data architect?

For a strong mid-to-senior data engineer, the realistic transition is roughly twelve to eighteen months of deliberate, project-based effort — not calendar time coasting in the current role. The bottleneck is almost never the pillars you already have (modeling, integration) but the ones DEs are furthest from: governance and cost, each of which takes about a quarter of focused work to move from "applied" to "standard-setting." A workable roadmap is months 0–3 to close the two weakest pillars with shippable artifacts, months 3–9 to build a portfolio of reference designs and ADRs (volunteer for the cross-team design nobody owns), months 9–15 to earn one role-targeted certification and broaden to a 3+ across all six pillars, and months 15–18 to lead a real design end-to-end and interview. Gate each phase on shipping the prior artifact — a transition without milestones drifts for years, while a milestoned one typically completes inside eighteen months.

Practice on PipeCode

Drill the design practice library → for the system-design, reference-architecture, and design-critique problems every architect interview is built on.
Build modeling muscle on the dimensional-modeling practice library → for the star-schema, grain, conformed-dimension, and SCD Type 2 problems architects are graded on.
Keep your foundations sharp with the SQL practice library → and the indexing practice library → for the query, partitioning, and performance reps behind every cost trade-off.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the six-pillar skill ladder and the architecture trade-off matrix against real graded inputs.

Build the judgement architects are hired for

Certifications prove a floor. PipeCode drills prove the judgement — when a star schema beats 3NF, when governance is the binding NFR, when the cheaper architecture is the right call, and how to defend a trade-off in numbers. Pipecode.ai is Leetcode for Data Engineering — design-first practice tuned for the modeling, governance, and trade-off decisions data architects are actually graded on.

Practice design problems →
Practice modeling problems →

Data Platform Team Structure: Central Platform + Embedded DEs + Analytics Engineers

Gowtham Potureddi — Sat, 01 Aug 2026 13:32:01 +0000

data platform team structure is the decision that quietly determines whether your data org ships trustworthy pipelines in weeks or drowns in a two-quarter backlog of ticket-driven requests — and it is the one architectural choice leaders get wrong most often because they treat it as an HR reporting-line question instead of a systems-design question. The shape of the data platform team — a single central data platform everyone queues behind, embedded data engineers sitting inside product squads, or a hybrid where a platform team paves the road and embedded engineers drive on it — decides how fast a new data product ships, who is accountable when a dashboard is wrong, whether governance is a bottleneck or a guardrail, and how the org scales from ten engineers to a hundred without collapsing into either chaos or gridlock. The trade-off does not live in "should we have a data team"; every company past its first analyst needs one. It lives in which topology you pick and what that topology costs you in latency, ownership, and standards drift.

This guide is the leader-and-staff-engineer walkthrough you wished existed the first time someone asked "how should we structure the data org as we scale," or "central platform or embedded engineers — where do the DEs report," or "when do we adopt data mesh, and what does an analytics engineer actually own." It walks through the framing (Conway's Law for data and the three canonical archetypes), the central platform team run as an internal product, embedded data engineers and the dotted-line reporting problem, analytics engineers and the dbt-era boundary between engineering and analysis, and finally scaling with team topologies, data mesh, headcount ratios, and the anti-patterns that stall a data team org. Each section pairs a teaching block with a Solution-Tail interview answer — a concrete artifact (a charter, a RACI, an org map), a step-by-step trace, an output table, then a concept-by-concept breakdown of why the structure works.

When you want hands-on reps immediately after reading, drill the system design practice library →, rehearse the modelling foundations on the dimensional-modeling practice library →, and sharpen the pipeline axis with the ETL practice library →.

On this page

Why org structure is a data-architecture decision
The central platform team: paved roads and self-serve
Embedded data engineers and the dotted-line problem
Analytics engineers and the dbt-era boundary
Scaling the org: topologies, data mesh and anti-patterns
Cheat sheet — data-team-structure recipes
Frequently asked questions
Practice on PipeCode

1. Why org structure is a data-architecture decision

The org chart becomes the data architecture — Conway's Law is not a metaphor for data teams, it is the mechanism

The one-sentence invariant: your data platform team structure is not a management detail bolted onto the architecture after the fact — it is the architecture, because Conway's Law guarantees the pipelines, the ownership boundaries, the schemas, and the SLAs will mirror the communication structure of whoever builds them, and the three archetypes (centralized, embedded/decentralized, hybrid platform + embedded) each hard-code a different set of trade-offs between delivery speed, domain accuracy, and governance that cannot be undone by tooling alone. A company that centralizes every data engineer into one team ships a warehouse that looks like a single monolith with a request queue in front of it. A company that scatters engineers into product squads with no coordinating function ships fifteen incompatible definitions of "active user." The structure you pick in year one becomes the migration you fight in year three, because every domain, dashboard, and downstream consumer hard-codes assumptions about who owns what.

The three archetypes every data leader must be able to name.

Centralized. All data engineers, analytics engineers, and analysts report into one data organization that serves the whole company. Strong standards, one warehouse, one set of tools — but the central team becomes a bottleneck the moment demand outpaces headcount, and it sits far from the domain knowledge locked inside product teams.
Decentralized / embedded. Data engineers are distributed into product or business squads, reporting to those squads. Fast iteration, deep domain context, clear product accountability — but standards drift, tools fragment, and there is nobody who owns the shared platform, so every squad rebuilds ingestion from scratch.
Hybrid (platform + embedded). A central platform team owns the shared substrate (ingestion, orchestration, catalog, governance) and runs it as an internal product; embedded data engineers and analytics engineers sit in the domains and build on the paved road. This is the topology most scaling companies converge on, and the one senior interviewers expect you to defend.

Conway's Law for data — the mechanism, stated precisely.

The law. Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure. For software this is famous; for data it is under-appreciated and more brutal, because data systems are integration systems — their whole job is to join across domains.
The consequence. If two teams do not talk, their data will not join cleanly. Two squads that own "orders" and "payments" separately, with no shared platform team forcing a common customer_id contract, will produce two customer dimensions that cannot be reconciled without a heroic quarter of remediation.
The lever. Because the law is a mechanism and not a curse, you can use it deliberately. Want a single source of truth for the customer? Give one team clear ownership of the customer data product and a contract everyone consumes. Want fast domain iteration? Embed engineers where the domain lives. The org chart is the API surface you are actually designing.

What breaks at each stage of growth.

Early (centralized, 3–10 people). Works well. One team, one warehouse, tight standards. Breaks when the request queue grows faster than the team — every stakeholder is waiting behind a ticket, and the central team is a hostage to its own backlog.
Mid (embedding starts, 10–40 people). Companies push engineers into domains for speed. Works until nobody owns the shared platform — five squads maintain five Airflow instances, three metric definitions for revenue, and no shared lineage. This is the "silo stage."
Scale (hybrid or mesh, 40+ people). The platform team is carved out to own the substrate; domains own their data products on top. Breaks only when governance is under-invested (ungoverned mesh = a thousand data products nobody can find) or the platform team drifts back into being a bottleneck by owning too much.

What interviewers and leaders actually probe.

Do you name all three archetypes and their failure modes without prompting? — senior signal.
Do you invoke Conway's Law as the reason the structure matters, not as a buzzword? — senior signal.
Do you tie structure to ownership and accountability ("who is paged when this dashboard is wrong") rather than to headcount? — required answer.
Do you describe the hybrid platform-plus-embedded model as "platform paves the road, domains drive on it" rather than as "some central, some embedded"? — senior signal.
Do you connect the org decision to a concrete failure the current structure is causing rather than reorganizing for its own sake? — required answer.

Worked example — the three-archetype comparison table

Detailed explanation. The single most useful artifact for a data-org-structure discussion is a memorised 3×N comparison table. Every serious conversation about how to structure a data team converges on it within the first ten minutes; having it in your head is what separates a fluent leader from someone reciting org-chart clichés. Walk through building the table for a hypothetical mid-size company with three product lines (checkout, growth, logistics) that currently has one central data team of eight people and a two-quarter backlog.

Delivery speed. How fast can a squad get a new data product or metric shipped without waiting on another team.
Domain accuracy. How correct the semantics are — does "active user" mean the same thing everywhere, and does whoever built the pipeline understand the business.
Governance and standards. Whether there is one warehouse, one set of tools, consistent lineage, and enforced contracts.
Failure mode. The specific way each archetype breaks under load.

Question. Build the three-archetype comparison for this company and pick the archetype it should move toward given a two-quarter backlog and three domains.

Input.

Archetype	Delivery speed	Domain accuracy	Governance	Primary failure mode
Centralized	slow (queue-bound)	weak (far from domain)	strong (one team)	central bottleneck
Decentralized / embedded	fast (in-squad)	strong (in the domain)	weak (fragmentation)	silos + drift
Hybrid (platform + embedded)	fast on the paved road	strong (embedded)	strong (platform owns substrate)	platform re-centralizes if it owns too much

Code.

Data-org archetype decision memo (1 page)
=========================================

Context
  - 8-person central data team, 2-quarter backlog.
  - 3 product lines: checkout, growth, logistics.
  - Symptom: every squad waits behind one ticket queue;
    domain teams complain the central team "does not
    understand our data."

Diagnosis (Conway's Law)
  - Centralized structure => one integration point => one
    queue => bottleneck. Architecture mirrors org: a monolith
    warehouse with a request desk in front.

Options
  A. Stay centralized, add headcount.      -> queue shrinks
                                              but returns.
  B. Fully embed 1 DE per squad, no center. -> fast now,
                                              silos in 2 quarters.
  C. Hybrid: carve a 3-person platform team
     (ingestion, orchestration, catalog),
     embed 1 DE + 1 AE per domain.          -> paved road +
                                              domain speed.

Recommendation
  - Move to (C). Platform team owns the substrate as a product;
    domains own their data products on the road.
  - 90-day plan: publish paved-road ingestion + a single
    semantic layer contract; embed engineers into checkout and
    growth first (highest backlog), logistics next quarter.

Step-by-step explanation.

The company is centralized, so by Conway's Law its architecture is a single integration point with a queue in front — exactly the bottleneck symptom the backlog describes. The structure is causing the delivery problem; adding headcount (Option A) treats the symptom, not the mechanism.
Full embedding (Option B) fixes speed immediately but removes the only thing keeping standards coherent. In two quarters the three domains will have three ingestion stacks and three definitions of revenue — the silo failure mode. Fast today, expensive to reconcile later.
The hybrid (Option C) splits the problem: extract a small platform team to own the shared substrate as an internal product, and embed engineers into the domains for speed. This is the only option that improves delivery speed and preserves governance.
The sequencing matters. You do not embed everyone at once — you pave the road first (a self-serve ingestion path and a single semantic contract), then embed into the highest-backlog domains, so embedded engineers have a road to drive on instead of paving their own.
The recommendation is written as a one-page memo, not an org chart. Leaders decide on the trade-off, not the boxes-and-lines; the boxes fall out of the trade-off once the platform-versus-domain boundary is agreed.

Output.

Domain	Structure after move	Owns
Platform team (new, 3 people)	central, product-run	ingestion, orchestration, catalog, contracts
Checkout squad	embedded DE + AE	checkout data products on the paved road
Growth squad	embedded DE + AE	growth metrics, experimentation tables
Logistics squad (next quarter)	embedded DE + AE	logistics data products

Rule of thumb. Never reorganize a data team by copying another company's org chart. Diagnose the failure mode first (bottleneck, silo, or ungoverned), map it to the archetype that is causing it via Conway's Law, then pick the structure whose failure mode you can afford. Write the trade-off as a one-page memo before you draw a single box.

Worked example — a Conway's-Law mapping exercise

Detailed explanation. The most senior move in a data-org discussion is to run the mapping in reverse: given the data architecture you want, derive the org structure that will naturally produce it. Codifying this makes the reorg defensible — you are not moving people for taste, you are aligning the communication structure with the target architecture. Walk through the mapping for a company that wants a single trusted "customer 360" data product consumed by every domain.

Target architecture. One canonical customer data product with a published contract; every domain (orders, support, marketing) consumes it rather than rebuilding it.
Conway's implication. A single, clearly-owned artifact requires a single, clearly-owned team with a stable contract and a communication channel to every consumer.
The anti-pattern to avoid. Three domains each maintaining their own customer table means three teams, three schemas, no single owner — Conway's Law guarantees fragmentation.

Question. Map the desired "customer 360" architecture to the team structure and contract that will produce it, and name the coordination mechanism.

Input.

Target architecture property	Org structure it requires
One canonical customer data product	One team with clear, sole ownership
Stable schema every domain trusts	A published data contract + versioning
Domains consume, do not fork	A consumer channel (guild / contract review)
Changes do not silently break consumers	Contract tests in CI + deprecation policy

Code.

## Customer-360 ownership contract

**Owner:** Platform / Customer-data pod (single team, on-call rotation)
**Consumers:** orders, support, marketing, finance

### The contract
- `customer` data product exposes: `customer_id` (stable, never reused),
  `email_hash`, `first_seen_at`, `status`, `lifetime_value_cents`.
- Schema is versioned (`v1`, `v2`); breaking changes ship a new version
  and a 2-sprint deprecation window on the old one.
- SLA: freshness < 1h, availability 99.9%, published on the catalog.

### Coordination mechanism
- Any consumer needing a new field files a change request in the
  data-guild channel; the owning pod triages weekly.
- Contract tests run in the owner's CI: a PR that breaks `customer_id`
  stability or drops a field fails the build.

### Anti-pattern this prevents
- No domain maintains its own `customer` table. Forking the customer
  entity is a governance violation, not a shortcut.

Step-by-step explanation.

The target is a single canonical artifact, so Conway's Law says you need a single owning team. Spreading customer ownership across three domains would, by the same law, produce three customer definitions — the exact fragmentation you are trying to avoid.
A stable schema that other teams trust requires a published contract with versioning. Without it, the owning team ships a breaking change on a Tuesday and four dashboards break on Wednesday. The contract is the API between the owning team and its consumers.
The coordination mechanism — a guild channel plus a weekly triage — is the "communication structure" Conway's Law is really about. It is the deliberate channel that lets consumers influence the product without forking it.
Contract tests in the owner's CI turn the social agreement into an enforced one. A PR that breaks customer_id stability fails the build; enforcement at the pipeline level is far more reliable than enforcement in a wiki page nobody reads.
The explicit anti-pattern clause ("no domain maintains its own customer table") is what keeps the architecture from degrading. Without a stated prohibition, the first team under deadline pressure forks the entity and the single-source-of-truth is gone.

Output.

Property	Without the mapping	With the mapping
Number of customer definitions	3+ (one per domain)	1 (canonical)
Breaking-change blast radius	silent, discovered in prod	caught by contract tests
Consumer influence	forking	change request + triage
Ownership when it's wrong	ambiguous	one on-call pod

Rule of thumb. To get the architecture you want, design the org that Conway's Law will translate into it: one canonical artifact needs one owning team plus a published, versioned, test-enforced contract and a real communication channel for consumers. If you cannot name the single owner, you do not have a data product — you have a fork waiting to happen.

Worked example — diagnose this failing data team

Detailed explanation. Leaders and interviewers love a "here is a broken data team, tell me what is wrong and how you would fix it" scenario, because it tests whether you diagnose structure or just add tools. The skill is to map the symptoms to the archetype failure mode and prescribe a structural change, not a technology. Walk through a realistic broken team.

Symptom set. A 25-person data org, fully embedded (each of six squads has 3–4 data people, no central team). Six Airflow deployments. Four definitions of "monthly active user." Nobody can produce a company-wide revenue number without a week of reconciliation. New hires take two months to find where anything lives.
The trap. The naive fix is "buy a data catalog" or "standardize on one orchestrator." Tools do not fix a structural gap; they need an owner.
The real diagnosis. This is the classic embedded-silo failure mode: fast local delivery, zero shared substrate, no owner of cross-domain semantics.

Question. Diagnose the failing team by archetype and prescribe the minimal structural change that fixes the root cause.

Input.

Symptom	Surface reading	Structural root cause
6 Airflow deployments	"tooling sprawl"	no platform owner
4 definitions of MAU	"data quality issue"	no owner of shared metrics
Week-long revenue reconciliation	"reporting is slow"	no canonical finance data product
2-month onboarding	"docs are bad"	no catalog, no single substrate

Code.

Failing-team diagnosis + remediation
====================================

Archetype: fully decentralized / embedded (no center).
Failure mode: silos + standards drift (textbook).

Root cause (NOT tooling):
  There is no team that OWNS the shared substrate or the
  cross-domain semantic contracts. Six squads each optimize
  locally; nobody is accountable for the whole.

Minimal structural fix (do not re-centralize everything):
  1. Carve a 4-person platform + enabling team from the
     existing 25 (do not hire first; reallocate).
       - Owns: one orchestration standard, one catalog,
         the semantic layer for shared entities (user,
         revenue), and the paved-road ingestion path.
  2. Keep DEs/AEs embedded in the six squads, but add a
     dotted line to the platform team (a guild) for standards.
  3. Declare 2 cross-domain data products canonical:
     `active_user` and `revenue`; one squad owns each with
     a contract; other squads consume, do not fork.

What NOT to do:
  - Do not fully centralize (kills the delivery speed the
    embedding bought).
  - Do not "buy a catalog" and hope; a catalog with no owner
    is an empty catalog in 3 months.

Step-by-step explanation.

The symptoms all point at one root: there is no owner of the shared substrate. Six Airflow deployments, four MAU definitions, and week-long reconciliation are not four separate problems — they are one structural gap (no platform + no cross-domain semantic owner) expressed four ways.
The trap is to prescribe tools. A catalog, a single orchestrator, or a metrics layer are all necessary — but each needs a team that owns it. Buying tooling without assigning ownership recreates the silo with nicer logos.
The fix is additive and minimal: carve a small platform + enabling team from existing headcount rather than hiring, so it lands this quarter. Reallocation signals the change is structural, not a growth bet.
Crucially, you keep the embedding. The embedded model bought real delivery speed; re-centralizing everything would trade one failure mode (silos) for another (bottleneck). The dotted-line guild preserves speed while restoring standards.
Declaring exactly two entities canonical (active_user, revenue) is deliberate scope control. You fix the highest-pain cross-domain semantics first with contracts, rather than trying to boil the ocean of governance on day one.

Output.

Before	After
6 orchestrators, no owner	1 paved-road standard, platform-owned
4 MAU definitions	1 canonical `active_user` contract
Week-long revenue reconcile	1 canonical `revenue` data product
No coordination	dotted-line guild + weekly triage

Rule of thumb. When a data team is failing, diagnose the archetype and its failure mode before you prescribe anything. Silos need a platform owner and shared contracts, not more tools; bottlenecks need embedding and self-serve, not more headcount. The fix is almost always structural, and the smallest structural change that removes the root cause beats the biggest tooling purchase.

Senior interview question on data org structure

A senior interviewer often opens with: "You join as head of data at a 120-person company. The data function is a single centralized team of twelve with a two-quarter backlog; product teams route every request through a ticket queue and complain the data team does not understand their domains. Walk me through how you would restructure the data org, what you would sequence in the first 90 days, and how you would prevent the new structure from breaking as the company doubles."

Solution Using a hybrid platform-plus-embedded reorg with a paved road and a 90-day sequence

Target structure (hybrid platform + embedded)
=============================================

  Central Platform team (4)          <- product-run substrate
    - paved-road ingestion, orchestration standard, catalog,
      semantic layer for shared entities, governance guardrails
    - runs as an internal product with an SLA and a roadmap

  Enabling team (2, time-boxed)      <- upskills domains, then dissolves
    - embeds temporarily to bootstrap each domain onto the road

  Embedded pods (1 DE + 1 AE per domain, ~6 pods)
    - solid line to the domain (delivery + accountability)
    - dotted line to platform via a data guild (standards, craft)

  Analytics engineers
    - own transformation + semantic models per domain, on the
      platform's shared semantic layer

## 90-day reorg sequence

### Days 0-30 — stabilize + pave
- Freeze the ticket queue triage into 2 lanes: platform work vs
  domain work. Stop being a request desk.
- Carve the 4-person platform team from the existing 12
  (reallocate, do not hire yet). Publish the paved-road
  ingestion path and one orchestration standard.
- Stand up a catalog + a single semantic layer skeleton.

### Days 30-60 — embed where the pain is
- Embed 1 DE + 1 AE into the top-2 backlog domains (checkout,
  growth). Solid line to the domain; dotted line to the guild.
- Migrate those 2 domains onto the paved road; delete their
  bespoke ingestion.
- Declare `active_user` and `revenue` canonical data products
  with contracts + contract tests in CI.

### Days 60-90 — scale + govern
- Embed into remaining domains. Enabling team rotates through
  each to bootstrap, then steps back.
- Publish SLAs/SLOs for the platform (freshness, availability).
- Establish the data guild: weekly standards triage, shared
  on-call for the platform, career ladder that values embedded
  and platform paths equally.

Step-by-step trace.

Step	Before (centralized)	After (hybrid)
Delivery model	ticket queue, one team	self-serve paved road + embedded pods
Domain accuracy	weak (team far from domain)	strong (DE/AE in the domain)
Governance	strong but bottlenecked	platform owns substrate + contracts
Ownership when wrong	ambiguous ("data team")	embedded pod (delivery) + platform (substrate)
Bottleneck	one queue for everything	platform work vs domain work split
Scaling behavior	linear headcount to keep up	road scales, pods add per domain

After the reorg, product squads stop queuing behind a central desk: they build on a self-serve paved road with embedded engineers who understand their domain, while a small platform team keeps the substrate and the shared contracts coherent. The two-quarter backlog drains because most requests are now self-serve; the platform team's roadmap is the queue for genuinely shared work, and cross-domain semantics stay consistent because active_user and revenue are owned data products, not per-squad forks.

Output:

Metric	Before	After
Median request lead time	weeks (queue-bound)	days (self-serve + embedded)
Cross-domain metric definitions	many, inconsistent	canonical, contract-enforced
Domain context in pipelines	low	high (embedded)
Governance owner	overloaded central team	dedicated platform team
Scaling response	add headcount to the queue	pave once, add pods per domain

Why this works — concept by concept:

Conway's Law alignment — the reorg deliberately shapes the communication structure to match the target architecture: a shared substrate needs one owning platform team, domain speed needs embedded pods, and cross-domain truth needs owned data products with contracts. Structure and architecture stop fighting.
Platform-as-product paved road — extracting the substrate into a self-serve product converts most of the backlog from "queue behind the central team" into "self-serve on the road," which is the only way delivery speed stops being headcount-bound.
Embedded pods with a dotted line — solid line to the domain buys accountability and domain accuracy; the dotted-line guild buys standards without re-centralizing, so you get speed and coherence instead of choosing one.
Canonical data products — declaring active_user and revenue as owned, contract-tested products stops the metric fragmentation at its source; consumers consume instead of forking.
Cost — one small platform team (4) + an enabling team (2, time-boxed) reallocated from existing headcount, plus a guild and contract-test CI. The eliminated cost is the compounding O(domains²) reconciliation of divergent definitions and the perpetual central-queue tax. Net: near-constant coordination cost per new domain instead of linear-or-worse.

Design
Topic — design
System design problems on data org and ownership

Practice →

SQL Topic — sql SQL problems on canonical metrics and reconciliation

Practice →

2. The central platform team

Run the central data platform as an internal product — paved roads, golden paths, and self-serve, not a ticket queue

The mental model in one line: the central data platform team is the group that owns the shared data substrate — ingestion, orchestration, storage, catalog, governance, and the semantic layer — and its defining move is to run that substrate as an internal product with real users (the domain engineers), a paved-road golden path, published SLAs, and a funding model, rather than as a services desk that other teams file tickets against. The difference between a platform team that scales the org and one that becomes the new bottleneck is entirely whether it thinks "product" (build the road once, let everyone self-serve) or "service" (do the work for everyone, one ticket at a time). Every senior data leader has watched a well-intentioned central team collapse under its own request queue because it never made the product shift.

Platform-as-product — the mindset shift that defines the role.

Users are engineers. The platform team's customers are the embedded DEs and AEs in the domains. Their "product" is measured by adoption and self-serve success, not by tickets closed. A capability nobody adopts is a failed feature, not a completed request.
Golden path over bespoke. The team publishes an opinionated, well-supported "golden path" for the 80% case (ingest a source, schedule a job, register a dataset). Teams that stay on the golden path get support, upgrades, and SLAs for free; teams that go off-road own their own maintenance.
Roadmap, not backlog. A product-run platform has a roadmap driven by user research (what are domains struggling with), not a backlog of one-off asks. This is the single most important cultural signal that a team has made the shift.
Self-serve is the goal. Every recurring request should become a self-serve capability. "I need a new source ingested" should be a config change a domain engineer makes, not a ticket the platform team fulfils.

Paved-road capabilities — what the platform actually owns.

Ingestion. A self-serve path to land a new source into the warehouse/lake with schema detection, incremental loads, and monitoring — so domains do not each rebuild ingestion.
Orchestration. One blessed scheduler and a standard way to define, deploy, and observe pipelines. Domains write DAGs; the platform runs the runtime.
Catalog + lineage. A single discoverable catalog with lineage, so a new hire can find the canonical revenue table in minutes, not weeks.
Governance guardrails. Access control, PII tagging, data-contract enforcement, and quality checks provided as guardrails on the paved road, so doing the right thing is the easy path.
Semantic / metrics layer. The shared definitions of the entities and metrics everyone uses, exposed so domains build on them instead of forking.

SLAs, SLOs, and the reliability contract.

Publish them. A platform without published SLOs is a platform whose users cannot plan. Freshness (data is < 1h old), availability (the warehouse is up 99.9%), and support response are the three every platform commits to.
Measure and report. SLOs mean nothing without dashboards and error budgets. When the platform burns its freshness budget, domain roadmaps slow — which is exactly the feedback loop that justifies platform investment.
On-call. The platform team runs an on-call rotation for the substrate. If the shared ingestion path breaks at 2 a.m., the platform team is paged, not each domain individually.

The funding model — the question that decides the team's fate.

Central cost center. The platform is funded centrally as shared infrastructure. Simple, avoids per-team friction, but risks the platform being under-invested because "nobody's budget line."
Chargeback / showback. Domains are charged (or shown) the cost of their platform usage. Aligns incentives and surfaces waste, but adds accounting overhead and can push domains off the paved road to save money.
The senior take. Most scaling orgs run showback (visibility without hard billing) plus central funding for the core substrate, so the platform is properly resourced and domains see the cost of their footprint.

Common interview probes on the central platform team.

"What does a central data platform team own?" — required answer: the shared substrate (ingestion, orchestration, catalog, governance, semantic layer) run as a product.
"How do you keep a platform team from becoming a bottleneck?" — self-serve golden paths, roadmap over backlog, adoption metrics.
"How do you fund a platform team?" — central funding for the core + showback for usage visibility.
"How do you measure platform success?" — adoption, self-serve rate, SLO attainment, domain lead time — not tickets closed.

Worked example — a platform-team charter and scope doc

Detailed explanation. The first artifact a new platform team ships is not code — it is a charter that says what the team owns, what it explicitly does not own, and how domains engage with it. A crisp charter is what prevents the team from silently sliding back into a services desk. Walk through writing one for a newly-carved 4-person platform team.

In scope. The shared substrate and the golden paths on top of it.
Out of scope. Domain-specific pipelines and metrics — those belong to embedded engineers.
Engagement model. Self-serve first; guild for standards; roadmap for shared asks.

Question. Write the platform-team charter that defines ownership boundaries and the engagement model.

Input.

Charter section	Content
Mission	make domains productive and data trustworthy via a self-serve platform
In scope	ingestion, orchestration, catalog, governance, semantic layer
Out of scope	domain pipelines, domain metrics, dashboards
Engagement	self-serve golden path; guild for standards; roadmap for shared work
Success metrics	adoption, self-serve rate, SLO attainment, domain lead time

Code.

# Data Platform Team — Charter (v1)

## Mission
Make every domain team fast and every dataset trustworthy by
providing a self-serve data platform run as an internal product.

## What we own (in scope)
- **Ingestion**: self-serve source onboarding (config-driven),
  incremental loads, schema detection, ingest monitoring.
- **Orchestration**: one blessed scheduler + deploy/observe tooling.
- **Catalog + lineage**: single discoverable catalog for all datasets.
- **Governance guardrails**: access control, PII tagging, data
  contracts, quality checks — provided ON the paved road.
- **Semantic layer**: shared entity + metric definitions.

## What we do NOT own (out of scope)
- Domain pipelines, domain-specific transformations, dashboards,
  and domain metric logic. Those are owned by embedded DEs/AEs.
- We build the road; domains drive on it.

## How to engage with us
1. **Self-serve first**: golden-path docs cover the 80% case.
2. **Data guild**: standards, RFCs, and craft questions — weekly.
3. **Roadmap**: genuinely shared, cross-domain asks are triaged
   into the platform roadmap, not a ticket queue.

## Success metrics (reviewed monthly)
- % of domains on the golden path (adoption)
- self-serve rate (requests resolved without platform toil)
- SLO attainment (freshness, availability)
- domain lead time (idea -> shipped data product)

## Explicit anti-goal
- We are NOT a request desk. If a request recurs, we turn it into
  a self-serve capability rather than fulfilling it repeatedly.

Step-by-step explanation.

The mission is framed around domain productivity and trust, not around the platform's own outputs. This keeps the team honest: the platform exists to make others fast, so its success is measured by their speed, not its ticket throughput.
The in-scope list is the shared substrate — the things that are wasteful to rebuild per domain. Everything here has the property that centralizing it removes duplication without removing domain autonomy.
The out-of-scope list is the most important section. By explicitly disowning domain pipelines and metrics, the charter prevents the team from being pulled into doing domain work, which is the road back to being a bottleneck.
The engagement model puts self-serve first and demotes ticketed work to a roadmap for genuinely shared asks. This is the structural mechanism that keeps the team a product team rather than a services desk.
The success metrics and the explicit anti-goal ("we are NOT a request desk") turn the culture into something measurable. When a request recurs, the charter obliges the team to productize it — that is the flywheel that drains the backlog over time.

Output.

Question	Charter answer
Who ingests a new source?	domain self-serves on the golden path
Who owns the checkout revenue metric?	the embedded AE, not the platform
Who is paged if the scheduler is down?	platform on-call
Who decides a new shared capability?	platform roadmap via the guild
What if a request recurs?	platform productizes it (anti-goal)

Rule of thumb. A platform team's charter must state what it does not own as clearly as what it does. The out-of-scope list and the "not a request desk" anti-goal are what keep the team productized. If your charter only lists responsibilities, you have written a services-desk job description, not a platform charter.

Worked example — a paved-road capability map with SLOs

Detailed explanation. Once the charter defines the boundary, the platform team publishes a capability map: for each capability, the golden-path way to use it, the self-serve status, and the SLO. This map is the platform's product catalog — it tells domains what they can rely on and what they still have to build themselves. Walk through building it.

Capability. Ingestion, orchestration, catalog, quality, semantic layer.
Golden path. The blessed, supported way to use each.
SLO. The reliability commitment for each.

Question. Build the paved-road capability map and attach an SLO to each capability.

Input.

Capability	Golden path	Self-serve?	SLO
Ingestion	config-driven source onboarding	yes	new source live < 1 day; freshness < 1h
Orchestration	standard DAG template + deploy CLI	yes	99.9% scheduler availability
Catalog	auto-registered on ingest	yes	100% of golden-path datasets cataloged
Data quality	declarative checks in the DAG template	partial	check failures alert < 5 min
Semantic layer	shared metric definitions repo	partial	metric change reviewed < 2 days

Code.

# paved-road ingestion — a domain engineer self-serves by adding
# a source config; the platform runtime does the rest.
source:
  name: checkout_events
  domain: checkout
  connector: postgres_cdc          # from the platform connector catalog
  destination: warehouse.raw.checkout_events
  load: incremental                # platform handles watermarking
  freshness_slo: 1h                # platform monitors + alerts
  pii:
    - column: email
      classification: pii          # governance guardrail: auto-masked
  owner: checkout-pod              # who is paged for domain logic
  contract: schemas/checkout_events.v1.yaml

-- Platform-provided SLO query: freshness for every golden-path source.
-- Runs on a schedule; breaches page the platform on-call.
SELECT
    source_name,
    domain,
    max(loaded_at)                                   AS last_load,
    now() - max(loaded_at)                           AS staleness,
    freshness_slo,
    (now() - max(loaded_at)) > freshness_slo         AS breaching
FROM   platform.ingest_audit
GROUP  BY source_name, domain, freshness_slo
HAVING (now() - max(loaded_at)) > freshness_slo
ORDER  BY staleness DESC;

Step-by-step explanation.

The capability map is a product catalog: each row tells a domain engineer exactly how to use a capability, whether they can self-serve, and what reliability they can plan around. This is what makes the platform legible to its users.
The ingestion golden path is config-driven — a domain engineer adds a source YAML and the platform runtime handles watermarking, freshness monitoring, and PII masking. The domain writes ~12 lines of config instead of a bespoke pipeline, which is the entire point of the paved road.
The pii block is a governance guardrail delivered on the road: because the golden path auto-masks tagged columns, doing the compliant thing is the default, not an extra step. Guardrails-on-the-path is how a platform enforces governance without policing.
The freshness SLO is not just a promise — the platform ships the query that measures it and wires breaches to on-call. An SLO the platform cannot measure is marketing; an SLO with a query and a pager is a contract.
The owner field encodes the split cleanly: the platform is paged if the ingestion runtime breaks, but the domain pod owns the source's business logic and contract. The capability map makes the DE-versus-platform boundary operational.

Output.

Scenario	Who acts	Why
New checkout source needed	checkout DE adds YAML	self-serve golden path
Ingestion runtime down	platform on-call	platform owns the runtime
checkout_events schema wrong	checkout pod	domain owns the contract
PII column added	auto-masked by guardrail	governance on the road
Freshness SLO breached	platform on-call, then domain if source-side	SLO query pages

Rule of thumb. Publish the platform as a capability map with a golden path and an SLO per capability. If a capability has no golden path, domains will build their own and you have a silo; if it has no SLO, domains cannot plan around it. The map is the platform's product; keep it honest by shipping the query that measures every SLO.

Worked example — a data contract as the platform's enforcement primitive

Detailed explanation. The platform's most powerful governance tool is the data contract: a machine-checkable declaration of a dataset's schema, semantics, and quality expectations, enforced in CI so a producer cannot ship a breaking change silently. Contracts are how a platform team enforces standards without becoming a review bottleneck. Walk through defining and enforcing one.

Declare. The producing team writes a contract for the dataset it owns.
Enforce. The platform provides a CI check that fails a PR violating the contract.
Consume. Downstream teams read the contract to know what they can rely on.

Question. Write a data contract for the canonical active_user product and the check that enforces it in CI.

Input.

Contract element	Value
Dataset	analytics.active_user
Owner	growth-pod
Guaranteed columns	user_id, active_date, channel
Stability rule	user_id never null, never reused
Quality rule	no duplicate (user_id, active_date)
Freshness	< 3h

Code.

# contracts/active_user.v1.yaml — the canonical active-user product
dataset: analytics.active_user
owner: growth-pod
version: v1
description: One row per user per active day. Canonical MAU/DAU source.

columns:
  - name: user_id
    type: bigint
    nullable: false            # stability guarantee
    tests: [not_null, positive]
  - name: active_date
    type: date
    nullable: false
  - name: channel
    type: string
    accepted_values: [web, ios, android]

checks:
  - unique: [user_id, active_date]   # no double counting
  - freshness:
      column: loaded_at
      max_lag: 3h

policy:
  breaking_change: new_version_required   # v1 -> v2, 2-sprint deprecation

# Platform-provided CI gate: fail the PR if a change violates the contract.
import sys, yaml

def check_contract(contract_path: str, table_profile: dict) -> list[str]:
    """Return a list of violations; empty list == pass."""
    contract = yaml.safe_load(open(contract_path))
    violations = []

    declared = {c["name"] for c in contract["columns"]}
    present  = set(table_profile["columns"])

    # 1. No guaranteed column may disappear (breaking change).
    for col in contract["columns"]:
        if not col.get("nullable", True) and col["name"] not in present:
            violations.append(f"missing guaranteed column: {col['name']}")

    # 2. Uniqueness check must hold.
    for chk in contract.get("checks", []):
        if "unique" in chk and table_profile["dup_count"] > 0:
            violations.append(f"duplicate rows on {chk['unique']}")

    return violations

if __name__ == "__main__":
    issues = check_contract(sys.argv[1], load_profile(sys.argv[2]))
    if issues:
        print("CONTRACT VIOLATIONS:\n  " + "\n  ".join(issues))
        sys.exit(1)          # fails the build -> breaking change blocked
    print("contract OK")

Step-by-step explanation.

The contract declares not just the schema but the guarantees consumers rely on: user_id is non-null and never reused, and (user_id, active_date) is unique so nobody double-counts. These are the semantic promises that make the dataset a product rather than a table.
The accepted_values on channel is a semantic guardrail — a pipeline that starts emitting channel = "amp" fails the contract, catching a data-quality regression at the producer instead of in a downstream dashboard.
The CI gate is what turns the contract from documentation into enforcement. A PR that drops user_id or introduces duplicates fails the build; the producer cannot ship the breaking change silently. The platform provides the gate once; every contracted dataset reuses it.
The breaking_change: new_version_required policy encodes how change happens: you do not mutate v1 in place, you ship v2 with a deprecation window. This is how the platform lets producers evolve without breaking consumers overnight.
This is the platform enforcing standards without being a review bottleneck. The platform team does not manually review every dataset change — it ships the contract mechanism and the CI gate, and enforcement scales automatically across every domain.

Output.

Producer action	Contract gate result
Add a nullable column	pass (additive)
Drop `user_id`	fail (guaranteed column missing)
Introduce duplicate (user_id, active_date)	fail (uniqueness)
Emit unexpected `channel` value	fail (accepted_values)
Ship v2 with deprecation window	pass (policy-compliant)

Rule of thumb. Give the platform team a data-contract primitive and a CI gate, and it enforces standards at the speed of every domain's pipeline instead of the speed of a central review queue. Contracts are how governance scales: the platform builds the guardrail once, and correct behavior becomes the default path for every producer.

Senior interview question on the central platform team

A senior interviewer might ask: "Your central data platform team of five is drowning — it has a 40-ticket backlog, domains complain it is slow, and leadership is asking whether to just disband it and embed everyone. Make the case for the platform team, then describe exactly how you would transform it from a services desk into a product team, including what you would build, what you would stop doing, and how you would measure the change."

Solution Using a platform-as-product transformation with golden paths, showback funding, and adoption metrics

## The case for the platform team
Disbanding it re-creates the silo failure mode: 6 domains each
rebuild ingestion, orchestration, and governance -> O(domains)
duplication + divergent standards. The platform team is not the
problem; being a *services desk* is the problem. Fix the operating
model, not the existence of the team.

## Transformation plan (services desk -> product)

### Stop doing
- Stop fulfilling one-off pipeline requests. Every recurring
  request becomes a self-serve capability instead.
- Stop being the single approver on domain dataset changes;
  replace manual review with contract-test CI gates.

### Start building (golden paths)
1. Self-serve ingestion (config-driven source onboarding).
2. A standard DAG template + deploy CLI (self-serve orchestration).
3. Auto-cataloging on ingest + lineage.
4. Data-contract CI gate (governance without a review queue).
5. Shared semantic layer for `user` and `revenue`.

### Funding
- Central funding for the core substrate + showback so domains
  SEE their platform footprint (aligns incentives, no hard billing
  friction that pushes teams off-road).

-- The metric that proves the transformation: self-serve rate.
-- Requests resolved WITHOUT platform-team toil / total requests.
WITH reqs AS (
  SELECT
    date_trunc('month', created_at)                      AS month,
    count(*)                                             AS total,
    count(*) FILTER (WHERE resolution = 'self_serve')    AS self_serve
  FROM platform.requests
  GROUP BY 1
)
SELECT
  month,
  total,
  self_serve,
  round(100.0 * self_serve / nullif(total, 0), 1)        AS self_serve_pct
FROM reqs
ORDER BY month;

Step-by-step trace.

Layer	Before (services desk)	After (product)
Work intake	40-ticket backlog	self-serve + roadmap for shared work
Ingestion	platform builds each pipeline	domains self-serve via config
Governance	manual review bottleneck	contract-test CI gate
Funding	central, "nobody's line"	central core + showback
Success metric	tickets closed	self-serve rate + adoption + SLOs
Team identity	request desk	internal product team

After the transformation, the backlog stops growing because most requests become self-serve config changes domains make themselves; the platform team's roadmap becomes the queue only for genuinely shared work; governance is enforced by CI gates rather than a human review queue; and showback makes every domain's platform footprint visible so investment is justified by usage. The self-serve-rate metric climbs quarter over quarter, which is the objective proof that the team became a product.

Output:

Metric	Before	After
Open backlog	40 tickets, growing	shrinking; shared-only roadmap
Self-serve rate	~10%	70%+ target
Governance throughput	review-queue bound	CI-gate, per-pipeline
Domain lead time	weeks	days
Platform investment case	"cost center"	showback-justified

Why this works — concept by concept:

Platform-as-product operating model — the transformation changes how the team works (product, roadmap, self-serve) rather than whether it exists. Disbanding treats the symptom; productizing removes the root cause (services-desk intake) while keeping the anti-duplication benefit.
Golden-path self-serve — converting recurring requests into self-serve capabilities is the only mechanism that breaks the headcount-bound backlog; each productized capability permanently removes a class of tickets.
Contract-test CI gate — governance moves from a human review queue (O(changes) toil) to an automated gate (O(1) platform build, enforced per pipeline), so standards scale with the org instead of bottlenecking it.
Showback funding — visibility of usage aligns incentives and justifies investment without the hard-billing friction that drives teams off the paved road.
Cost — five platform engineers building golden paths once, plus a showback pipeline and CI gates. The eliminated cost is O(domains) duplicated substrate plus a perpetually growing ticket queue. Net: the platform's cost is amortized across every domain that self-serves on it, and the self-serve-rate metric makes that leverage measurable.

Design
Topic — design
Design problems on self-serve data platforms

Practice →

ETL Topic — etl ETL problems on ingestion and orchestration paved roads

Practice →

3. Embedded data engineers

Embed data engineers in the domains for proximity and ownership — then solve the dotted-line problem before the silos form

The mental model in one line: embedded data engineers are DEs who sit inside product or business squads — reporting to the squad for delivery and accountability, working shoulder-to-shoulder with the engineers and PMs who own the domain — which buys deep domain context and fast iteration, but which creates the dotted-line problem: without a coordinating guild and a platform to build on, embedded engineers drift into silos, reinvent the same infrastructure, and diverge on standards until the org has as many data stacks as it has squads. Embedding is the highest-leverage structure for delivery speed and the highest-risk structure for coherence, and the entire art is capturing the speed while paying down the silo risk with a deliberate reporting and guild model.

Why embed — the benefits that make it worth the risk.

Proximity to the domain. An embedded DE hears the product discussions, understands why an order can be in five states, and models the data correctly the first time. Domain context is the single biggest determinant of pipeline correctness, and embedding maximizes it.
Fast iteration. No cross-team ticket to change a schema — the DE and the product engineer are in the same standup. Lead time from "we need this metric" to "it is in the dashboard" collapses.
Clear product accountability. When the checkout dashboard is wrong, the checkout pod is accountable, not a distant central team. Ownership is unambiguous because the owner sits in the domain.
Better prioritization. Embedded engineers work on what the domain actually needs, ranked by domain value, rather than on whatever bubbled to the top of a shared queue.

The dotted-line problem — the core tension of embedding.

Solid line vs dotted line. The classic model: embedded DEs report solid line to their squad (day-to-day work, delivery, performance) and dotted line to a central data function or guild (standards, craft, career development, tooling). The solid line owns what they build; the dotted line owns how well they build it.
Why the dotted line matters. Without it, an embedded DE is a data team of one, disconnected from peers, with no standards pressure and no craft community. They will make locally-reasonable choices that are globally incoherent — a different orchestrator, a different testing approach, a fork of the customer table.
The failure of pure solid-line. If the DE reports only to the squad, the squad optimizes for its own delivery and treats shared standards as overhead. Multiply by six squads and you have six incompatible stacks — the silo failure mode.
The failure of pure dotted-line. If the "embedded" DE actually reports to the central team and is merely seconded to the squad, they get treated as an outsider, deprioritized, and never gain real domain context — the worst of both worlds.

The guild model — how to make the dotted line real.

What a guild is. A cross-cutting community of practice: all the embedded DEs (and AEs) across squads meet regularly, own shared standards, review each other's designs, and collectively steward the craft. It is the dotted line made concrete.
What the guild owns. Coding standards, the golden-path adoption push, shared RFCs for cross-domain decisions, onboarding, and the career ladder for embedded engineers.
Guild + platform together. The guild sets standards; the platform team provides the paved road that makes those standards the easy path. Guild without platform is a book club; platform without guild is a road nobody agrees to drive on.

Standards enforcement without central control.

Contracts over review. Enforce shared standards through the platform's data-contract CI gates and shared DAG templates, not through a central team reviewing every PR. This scales; central review does not.
Golden-path defaults. Make the standard the default by shipping templates and scaffolding. An embedded DE who scaffolds a new pipeline from the golden-path template inherits the standards for free.
Career-path parity. Ensure embedded and platform tracks have equal promotion ceilings. If the only way to get promoted is to join the central team, your best embedded engineers leave the domains and the model collapses.

Common interview probes on embedded DEs.

"Central vs embedded — where do DEs report?" — required answer: solid line to the domain, dotted line to a guild/platform for standards.
"How do you stop embedded engineers from siloing?" — guild + platform paved road + contract-enforced standards + career parity.
"What is the risk of embedding?" — standards drift, infra duplication, isolated engineers, divergent metrics.
"When would you NOT embed?" — very early stage (too few people) or a capability that is genuinely shared and better centralized.

Worked example — the embedding RACI matrix

Detailed explanation. The cleanest way to settle "who owns what" between an embedded DE, the squad, the platform, and the guild is a RACI matrix (Responsible, Accountable, Consulted, Informed). It makes the solid-line/dotted-line split concrete for each activity and pre-empts the turf ambiguity that kills embedded models. Walk through building one.

Actors. Embedded DE, squad lead, platform team, data guild.
Activities. Domain pipeline delivery, tooling choice, standards, career development, incident response.

Question. Build the RACI for an embedded DE across the five activities that most often cause ownership disputes.

Input.

Activity	Embedded DE	Squad lead	Platform team	Data guild
Domain pipeline delivery	R	A	C	I
Tooling / infra choice	R	C	A	C
Coding + data standards	R	I	C	A
Career development / promo	C	R	I	A
Domain data incident	R	A	C	I

Code.

Reading the embedded-DE RACI
============================
R = Responsible (does the work)
A = Accountable (answerable; one per row)
C = Consulted   (two-way input)
I = Informed    (one-way notified)

Key resolutions this RACI encodes:
  - Pipeline delivery: DE does it (R), squad lead owns the
    outcome (A). This is the SOLID line -> the domain.
  - Tooling choice: platform is Accountable (A) so the org does
    not fragment; DE is Responsible for using it well. Domains
    do not each pick their own orchestrator.
  - Standards: the guild is Accountable (A). This is the DOTTED
    line -> craft coherence across squads.
  - Career/promo: shared A between squad lead (day-to-day) and
    guild (craft bar). Prevents "you must join the central team
    to get promoted."

Step-by-step explanation.

Pipeline delivery has the DE as Responsible and the squad lead as Accountable — this is the solid line encoded precisely. The domain owns the outcome; the embedded engineer does the work inside the domain. Nobody outside the squad is on the hook for domain delivery.
Tooling and infra choice flips the accountability to the platform team. This is the anti-silo control: individual squads do not get to pick their own orchestrator or warehouse, because that is exactly how six incompatible stacks arise. The DE is Responsible for using the shared tooling well, not for choosing new tooling.
Standards are the guild's accountability — the dotted line made operational. The DE follows and contributes to standards; the squad lead is only Informed, which stops a delivery-focused squad lead from overriding craft standards for short-term speed.
Career development is deliberately a shared accountability between the squad lead and the guild. The squad lead knows the day-to-day performance; the guild guards the craft bar and ensures the embedded path has the same ceiling as the platform path. This is the single most important row for retention.
Incident response mirrors delivery: the DE responds, the squad lead is accountable for the domain's data being right. The platform is Consulted because the incident might be in the shared substrate, in which case it escalates to platform on-call.

Output.

Dispute	RACI resolution
"Can our squad use its own scheduler?"	No — platform is Accountable for tooling
"Who fixes the wrong checkout metric?"	Embedded DE (R), squad lead (A)
"Who sets the testing standard?"	The guild (A)
"Can I get promoted staying embedded?"	Yes — guild + squad lead share promo accountability
"Substrate is down mid-incident"	Escalate to platform (Consulted -> on-call)

Rule of thumb. Settle embedded-team ownership with a RACI before the first turf dispute, not after. The two rows that matter most are tooling (platform Accountable, to stop fragmentation) and career development (guild + squad shared, to stop your best embedded engineers from leaving). If those two are ambiguous, the embedded model degrades into silos or attrition.

Worked example — a data-guild charter that operationalizes the dotted line

Detailed explanation. The guild is the mechanism that makes the dotted line real, but a guild without a charter is just a recurring meeting that decays into a status update. A guild charter defines its purpose, membership, decision rights, and rituals so it actually stewards standards. Walk through writing one.

Purpose. Steward the craft and standards across all embedded data engineers.
Decision rights. What the guild can decide vs recommend.
Rituals. The concrete recurring activities that keep it alive.

Question. Write the data-guild charter that makes the dotted-line reporting relationship operational.

Input.

Charter element	Content
Purpose	shared standards, craft, and career coherence for embedded DEs/AEs
Membership	all embedded DEs + AEs; platform sends a liaison
Decision rights	owns standards + golden-path adoption; recommends on tooling
Rituals	weekly design review, RFC process, quarterly standards refresh
Career role	maintains the leveling rubric for embedded engineers

Code.

# Data Guild — Charter

## Purpose
Give every embedded data engineer a craft home: shared standards,
peer design review, and a career ladder that does not require
leaving the domain. This IS the dotted line.

## Membership
- All embedded DEs and analytics engineers across squads.
- A platform-team liaison (keeps the road and the standards aligned).
- Rotating facilitator (not a permanent "guild lead" -> shared ownership).

## Decision rights
- **Owns (can decide):** coding/data standards, testing conventions,
  the golden-path adoption bar, the RFC process, onboarding for new
  embedded engineers.
- **Recommends (advisory):** tooling/infra choices -> final call sits
  with the platform team (see RACI).
- **Escalates:** cross-domain data-product ownership disputes ->
  head of data.

## Rituals
- **Weekly design review**: any non-trivial pipeline or model design
  gets a 20-min peer review before it ships.
- **RFC process**: cross-domain decisions (a new shared entity, a
  standard change) go through a written RFC with guild sign-off.
- **Quarterly standards refresh**: prune, update, and re-publish
  standards so they do not rot.

## Career role
- Maintains the leveling rubric so an embedded DE can reach staff/
  principal without transferring to the platform team.

Step-by-step explanation.

The purpose statement names the guild as the dotted line explicitly. This is important: it tells embedded engineers that the guild is where their craft home and career coherence live, not a bureaucratic overhead meeting. The dotted line is a community, not a reporting box.
Membership includes a platform liaison, which keeps the guild's standards and the platform's paved road in sync. A guild that sets standards the platform does not support produces standards nobody can follow; the liaison closes that loop.
The decision rights are split precisely to match the RACI: the guild owns standards and craft (can decide) but only recommends on tooling (platform decides). This prevents the guild from becoming a shadow platform team while still giving it real authority over how work is done.
The rituals are what keep the guild from decaying. Weekly design review catches divergence before it ships; the RFC process gives cross-domain decisions a written, reviewable form; the quarterly refresh stops standards from rotting into ignored wiki pages.
The career-role clause is the retention mechanism. By maintaining a leveling rubric that lets embedded engineers reach staff/principal in place, the guild removes the incentive for the best engineers to abandon the domains — which is the failure mode that silently kills embedded models over 18 months.

Output.

Without a guild charter	With a guild charter
Embedded DEs isolated, no peers	craft community + peer review
Standards decided ad hoc per squad	guild-owned, refreshed quarterly
Cross-domain decisions litigated repeatedly	written RFC + sign-off
Promotion requires joining central team	in-place staff/principal ladder
Standards rot	quarterly refresh keeps them live

Rule of thumb. A guild without decision rights and rituals is a status meeting; give it clear ownership of standards, an RFC process, and a career ladder, and it becomes the dotted line that keeps embedded engineers coherent without a central command. The quarterly refresh and the in-place career ladder are the two clauses that decide whether the guild survives its first year.

Worked example — the reporting-line decision for a new domain

Detailed explanation. A concrete recurring decision: a new domain is spinning up and needs data engineering. Do you embed a DE, extend the platform, or start with the guild providing part-time support? The answer depends on domain maturity, data complexity, and shared-versus-specific needs. Walk through the decision for three new domains.

Signals. Domain data complexity, delivery cadence, how much is genuinely shared vs domain-specific.
Options. Full embed, platform-supported self-serve, or guild-supported part-time.
Anti-pattern. Embedding a lone DE into a domain with no paved road to build on.

Question. Decide the reporting/support model for three new domains and justify each.

Input.

New domain	Data complexity	Cadence	Shared vs specific	Decision
Payments	high	fast	mostly specific	full embed (DE + AE)
Internal tools	low	slow	mostly shared	platform self-serve
Experimentation	high	fast	shared framework, specific data	embed AE + platform framework

Code.

Reporting-line decision tree (new domain)
=========================================

Q1. Is the domain's data complex AND its cadence fast?
      yes -> embedding is justified (proximity pays off)
      no  -> go to Q2

Q2. Is the need mostly SHARED (not domain-specific)?
      yes -> platform self-serve (do NOT embed a lone DE)
      no  -> lightweight guild support until it grows

Q3. If embedding: is there a paved road to build on?
      yes -> embed DE (+ AE if modelling-heavy)
      no  -> pave the minimal road FIRST, then embed
             (never drop a lone DE into a greenfield with no
              platform -> that is how silos are born)

Applied:
  Payments        -> Q1 yes -> embed DE + AE (complex, fast, specific)
  Internal tools  -> Q1 no, Q2 yes -> platform self-serve
  Experimentation -> Q1 yes, but framework is shared ->
                     embed 1 AE for domain models +
                     consume the platform's experimentation framework

Step-by-step explanation.

Payments is complex, fast-moving, and mostly domain-specific — the textbook case for full embedding. The proximity of an embedded DE and AE to the payments domain pays off immediately in correctness and speed, and there is little shared logic to centralize.
Internal tools is low-complexity, slow-cadence, and mostly shared needs. Embedding a dedicated DE here would be waste — the DE would be underutilized and isolated. Platform self-serve is correct: the internal-tools team uses the golden path for its modest needs.
Experimentation is the nuanced case: the data is domain-specific but the framework (assignment, exposure logging, stats) is shared. The right answer is a hybrid — embed an AE for the domain-specific models, but have them consume a platform-owned experimentation framework rather than build one.
The decision tree's Q3 encodes the most important anti-pattern check: never embed a lone DE into a greenfield domain with no paved road. Without a platform to build on, that DE will pave their own road, and you have manufactured a silo. Pave the minimal road first, then embed.
Reporting lines follow the decision: embedded engineers (payments, experimentation) get solid line to the domain and dotted line to the guild; the self-serve domain (internal tools) has no embedded engineer and simply consumes the platform. The structure matches the need rather than applying one template everywhere.

Output.

Domain	Model	Reporting
Payments	embed DE + AE	solid: payments; dotted: guild
Internal tools	platform self-serve	none embedded; consumes platform
Experimentation	embed AE + shared framework	solid: experimentation; dotted: guild

Rule of thumb. Do not apply one structural template to every domain. Embed where data is complex, fast-moving, and domain-specific; use platform self-serve where needs are shared and light; and never embed a lone engineer into a domain with no paved road to build on. Match the structure to the domain's complexity and cadence, and always pave before you embed.

Senior interview question on embedded data engineers

A senior interviewer might ask: "Your company embedded data engineers into six product squads a year ago. Delivery got faster, but now there are six orchestration setups, three definitions of revenue, embedded engineers say they feel isolated and two just quit for 'a real data team,' and leadership wants to yank everyone back to central. Design the reporting and coordination model that keeps the delivery speed but fixes the silos, isolation, and attrition — and explain what you would tell leadership."

Solution Using solid-line/dotted-line reporting, a chartered guild, a paved road, and career parity

Model: embedded delivery + guild coordination + platform road
=============================================================

  Embedded DE/AE in each squad
    - SOLID line -> squad (delivery, prioritization, accountability)
    - DOTTED line -> data guild (standards, craft, career)

  Data guild (all embedded engineers + platform liaison)
    - owns standards, RFCs, design review, leveling rubric
    - weekly design review; quarterly standards refresh

  Platform team
    - owns the paved road the standards ride on (one orchestrator,
      contracts, catalog, semantic layer)

  Fixes mapped to symptoms:
    6 orchestrators  -> platform owns tooling (RACI); migrate to one
    3 revenue defs   -> declare canonical `revenue` product + contract
    isolation        -> guild = craft community + peer review
    attrition        -> in-place staff/principal ladder (career parity)

## What I would tell leadership

Do NOT re-centralize. Yanking everyone back trades the silo
problem for the bottleneck problem you left behind a year ago,
and throws away the domain speed you paid for.

The problem is not embedding; it is embedding WITHOUT the two
things that make it work:
  1. A coordinating dotted line (a chartered guild).
  2. A shared paved road (a platform team) so engineers build on
     common rails instead of each paving their own.

Add those two. Keep the embedding. Declare the 2-3 highest-pain
cross-domain entities canonical with contracts. Give embedded
engineers a real career ladder so the best ones stay.

Step-by-step trace.

Symptom	Root cause	Structural fix
6 orchestration setups	no tooling owner	platform Accountable for tooling (RACI); consolidate
3 revenue definitions	no cross-domain semantic owner	canonical `revenue` product + contract
engineers isolated	no craft community	chartered guild + weekly design review
2 resignations	no in-place career path	staff/principal ladder for embedded track
leadership wants re-centralize	conflating "embedding" with "embedding done wrong"	keep embedding, add guild + platform

After the change, the six squads keep their embedded engineers and their delivery speed, but now build on one orchestrator and one paved road; revenue has a single contract-enforced definition; the guild gives isolated engineers a peer community and a design-review ritual; and the in-place career ladder removes the reason the two engineers left. Leadership gets the coherence they wanted without paying the bottleneck tax of re-centralization.

Output:

Metric	Before	After
Orchestration stacks	6	1 (platform-owned)
Revenue definitions	3	1 (canonical, contracted)
Embedded-engineer attrition	rising	stabilized (career parity)
Delivery speed	fast (kept)	fast (kept)
Standards coherence	drifting	guild-owned + contract-enforced

Why this works — concept by concept:

Solid-line/dotted-line split — solid line to the squad preserves the delivery speed and accountability embedding bought; dotted line to the guild restores craft coherence. Keeping both is what distinguishes "embedding done right" from the silo failure mode.
Chartered guild — the guild converts isolated engineers into a community with design review and shared standards, directly fixing both the isolation-driven attrition and the standards drift, without a central command structure.
Platform paved road — one orchestrator and shared contracts give embedded engineers common rails, which is the only durable fix for infrastructure duplication; standards enforced by contract CI scale across all six squads.
Career parity — an in-place staff/principal ladder removes the "leave for a real data team" incentive; retention of senior embedded engineers is what keeps domain context from resetting every 18 months.
Cost — a guild (mostly time, not headcount), a platform team owning tooling, and a one-time orchestrator consolidation. The eliminated cost is O(squads) duplicated infrastructure, O(squads) divergent metric reconciliation, and the recurring cost of re-hiring and re-onboarding after attrition. Net: coordination cost is near-constant per squad instead of growing with fragmentation.

Design
Topic — design
Design problems on team ownership and reporting models

Practice →

SQL Topic — sql SQL problems on cross-domain metric consistency

Practice →

4. Analytics engineers and the dbt-era boundary

The analytics engineer is the dbt-era bridge — owning transformation and the semantic layer between the data engineer's pipelines and the analyst's dashboards

The mental model in one line: the analytics engineer is the role the modern data stack created to fill the gap between the data engineer (who builds ingestion and infrastructure) and the data analyst (who builds dashboards and answers questions) — the AE owns the transformation layer, turning raw ingested data into clean, tested, documented, business-ready models (typically in dbt) and often the semantic/metrics layer on top, applying software-engineering rigor (version control, testing, CI, modularity) to analytics work that used to live in untested SQL scripts and spreadsheet exports. The AE role exists because dbt and the ELT pattern moved transformation out of the pipeline and into the warehouse, and someone with one foot in engineering and one in analytics had to own that middle. Getting the DE↔AE↔analyst boundary clear is what stops the same model being built three times or falling through the cracks entirely.

What the analytics engineer actually does.

Owns transformation. Takes raw, ingested tables and builds the staging, intermediate, and mart models that the business consumes. In the modern stack this is dbt: modular SQL models, refs, sources, and a DAG the AE owns end to end.
Applies engineering rigor to analytics. Version control, code review, tests (uniqueness, not-null, referential integrity, freshness), CI, documentation, and modularity — the practices DEs take for granted, applied to the transformation layer that analysts previously wrote as one-off scripts.
Owns the semantic/metrics layer. Increasingly the AE owns the shared definitions of metrics (revenue, active users, retention) so every downstream tool computes them the same way. This is the "single source of truth for metrics" that the org relies on.
Bridges the vocabulary. The AE speaks SQL and dbt fluently (engineering) and understands the business questions (analytics), so they translate "we need to understand churn" into a tested, documented churn model the analyst can trust.

The DE ↔ AE ↔ analyst boundary — who owns which layer.

Data engineer. Owns ingestion, the pipeline runtime, infrastructure, orchestration, and the raw/source layer landing in the warehouse. The DE gets trustworthy raw data into the warehouse reliably and at scale.
Analytics engineer. Owns the transformation from raw to business-ready: staging, intermediate, and mart models; tests; documentation; and typically the semantic layer. The AE turns raw data into trusted, reusable models.
Data analyst. Owns the last mile: exploration, dashboards, ad-hoc analysis, and turning trusted models into insight and recommendations for the business. The analyst answers questions using the AE's models.
Data scientist. Owns modelling, experimentation analysis, and ML features — often consuming the AE's marts as clean inputs rather than wrangling raw data.

Where the boundary blurs (and how to settle it).

DE vs AE on transformation. In some orgs DEs write transformations; in the modern stack AEs do. The clean split: DEs own the pipeline and the raw layer; AEs own everything from staging onward. If your DEs are writing dbt marts, you either do not have AEs or you have mislabeled them.
AE vs analyst on modelling. Analysts sometimes build their own models when the AE is a bottleneck — which recreates the untested-SQL problem. The fix: AEs own reusable models; analysts do analysis on top. If analysts are maintaining production models, the AE layer is under-resourced.
Who owns the semantic layer. The most-contested question. The strongest answer: the AE owns the semantic layer as code, in the same repo and CI as the dbt models, so metrics are versioned, tested, and consistent across every consuming tool. It is neither a BI-tool setting nor a DE responsibility.

The modern-data-stack ownership map.

Ingestion (Fivetran/Airbyte/custom). DE / platform.
Warehouse (Snowflake/BigQuery/Databricks). DE / platform.
Transformation (dbt). Analytics engineer.
Semantic/metrics layer (dbt metrics / semantic layer / metric store). Analytics engineer.
BI / dashboards (Looker/Tableau/Mode). Data analyst, on the semantic layer.
Reverse ETL / activation. DE/AE depending on org.

Common interview probes on analytics engineers.

"What is an analytics engineer?" — required answer: the role that owns tested, documented transformation (dbt) between DE pipelines and analyst dashboards.
"Who owns the semantic layer?" — strong answer: the AE, as versioned tested code alongside the dbt models.
"DE vs AE vs analyst — where are the lines?" — DE owns raw + infra, AE owns transformation + semantics, analyst owns insight.
"Why did the AE role emerge?" — dbt/ELT moved transformation into the warehouse; someone with engineering rigor and business context had to own it.

Worked example — the role-boundary matrix (DE vs AE vs DA vs DS)

Detailed explanation. The single artifact that settles most "who does what" disputes in a modern data team is a role-boundary matrix mapping each stack layer and each responsibility to a role. It stops the same model being built twice and stops layers falling through the cracks. Walk through building it for a team with all four roles.

Layers. Ingestion, raw, transformation, semantic layer, dashboards, ML features.
Roles. Data engineer, analytics engineer, data analyst, data scientist.
Rule. Each layer has exactly one owning role; adjacent roles are consulted.

Question. Build the role-boundary matrix and identify the two layers most likely to be contested.

Input.

Layer / responsibility	Data engineer	Analytics engineer	Data analyst	Data scientist
Ingestion + pipeline runtime	Owner	—	—	—
Raw / source layer	Owner	Consulted	—	—
Transformation (staging → marts)	Consulted	Owner	Consulted	—
Semantic / metrics layer	—	Owner	Consulted	Consulted
Dashboards + ad-hoc analysis	—	Consulted	Owner	—
ML features + experimentation	Consulted	Consulted	—	Owner

Code.

Role-boundary rules (one owner per layer)
=========================================
DE   owns: ingestion, runtime, raw/source, infra, orchestration.
AE   owns: transformation (staging/intermediate/marts), tests,
           docs, and the semantic/metrics layer.
DA   owns: dashboards, exploration, ad-hoc analysis, insight.
DS   owns: ML features, model training, experiment analysis.

Contested layers (watch these):
  1. Transformation: DEs sometimes write marts. Rule -> if you have
     AEs, DEs stop at the raw layer; AEs own staging onward.
  2. Semantic layer: BI tool vs dbt vs DE. Rule -> AE owns it AS
     CODE (versioned, tested, in CI), not as a BI-tool setting.

Smell tests:
  - Analysts maintaining production models -> AE layer under-resourced.
  - DEs writing business marts -> AE role missing or mislabeled.
  - Metrics differ across dashboards -> no single semantic owner.

Step-by-step explanation.

The matrix assigns exactly one Owner per layer, which is the whole point — ambiguous ownership is what causes the same revenue model to be built by a DE, an AE, and an analyst independently. One owner per layer, adjacent roles Consulted.
The DE owns everything up to and including the raw/source layer: ingestion, runtime, infra, and landing trustworthy raw data. The AE is Consulted on the raw layer because they consume it, but they do not own the pipeline.
The AE owns transformation and the semantic layer — the two layers that define the modern AE role. The DE and analyst are Consulted (they produce inputs and consume outputs) but the AE is accountable for the models being tested, documented, and correct.
The two contested layers are transformation and the semantic layer. Transformation is contested because DEs historically wrote it; the rule is that with AEs present, DEs stop at raw. The semantic layer is contested because it can live in the BI tool, in dbt, or with DEs; the rule is AE-owned-as-code.
The smell tests turn the matrix into a diagnostic. Analysts maintaining production models means the AE layer is under-resourced; DEs writing business marts means the AE role is missing; metrics differing across dashboards means no single semantic owner. Each smell maps to a boundary the matrix would have prevented.

Output.

Symptom	Boundary violated	Fix
Same metric, 3 values	no single semantic owner	AE owns semantic layer as code
Analysts maintaining prod models	AE layer under-resourced	staff more AEs / clarify ownership
DEs writing marts	AE role missing/mislabeled	DEs stop at raw; hire/assign AEs
Marts untested	rigor not applied to transformation	AE brings tests + CI
ML features hand-built from raw	DS reinventing transformation	DS consumes AE marts

Rule of thumb. Draw a role-boundary matrix with exactly one owner per stack layer, and treat the two contested layers — transformation and the semantic layer — as AE-owned by default. When metrics disagree across dashboards, you almost always have no single semantic owner; assign it to the AE as versioned, tested code and the disagreement disappears.

Worked example — the layered-ownership map as text

Detailed explanation. A role-boundary matrix says who owns each layer; a layered-ownership map shows the flow — how data moves through the layers and where each role's responsibility begins and ends. Drawing it as an ASCII map makes the handoffs explicit and is a great whiteboard artifact. Walk through building it.

Flow. Source → raw → staging → intermediate → marts → semantic layer → dashboards.
Handoffs. The points where ownership passes from one role to the next.
Contracts. What each handoff guarantees.

Question. Draw the layered-ownership map from source system to dashboard and annotate each handoff.

Input.

Layer	Owner	Handoff guarantee to next layer
Source → raw	DE	data landed, fresh, schema-detected
raw → staging	AE	typed, cleaned, renamed
staging → marts	AE	business entities, tested
marts → semantic	AE	metrics defined once
semantic → dashboards	DA	consistent numbers everywhere

Code.

Layered ownership map: source -> dashboard
==========================================

  [Source systems]          Postgres, Stripe, events
        |  DE owns: ingestion, runtime, freshness
        v
  [ raw / source layer ]    warehouse.raw.*        <-- DE handoff:
        |                                              landed + fresh
        |  AE owns from here down -------------------------------
        v
  [ staging ]               stg_* : typed, renamed, cleaned
        |
        v
  [ intermediate ]          int_* : joins, reusable logic
        |
        v
  [ marts ]                 dim_* / fct_* : business entities, TESTED
        |
        v
  [ semantic / metrics ]    metrics defined ONCE (revenue, MAU, churn)
        |  AE handoff: one definition per metric
        |  DA owns from here -----------------------------------
        v
  [ dashboards / analysis ] Looker/Tableau/Mode -> insight

Handoff contracts:
  DE -> AE : raw is fresh (< SLO) and schema is registered.
  AE -> DA : marts are tested; metrics have one definition.
  Break a contract -> the downstream layer breaks; contracts
  make the blame boundary unambiguous.

Step-by-step explanation.

The map makes the DE→AE handoff explicit: the DE owns everything down to the raw layer and guarantees it is landed and fresh. Everything below raw is the AE's territory. This single horizontal line resolves most DE-vs-AE disputes.
The staging → intermediate → marts progression is the AE's dbt DAG. Staging types and renames; intermediate holds reusable join logic; marts are the tested business entities (dim_customer, fct_orders). The AE owns this entire chain and its tests.
The marts → semantic handoff is where metrics get defined once. Instead of each dashboard computing revenue its own way, the semantic layer defines revenue a single time and every consumer references it. This is the AE's highest-leverage responsibility.
The semantic → dashboards handoff passes ownership to the analyst, with the guarantee that the numbers are consistent everywhere because they come from one definition. The analyst builds insight on trusted models rather than re-deriving metrics.
The handoff contracts make the blame boundary unambiguous. If a dashboard is wrong, you walk the map: is raw stale (DE), is a mart failing tests (AE), or is the dashboard mis-querying (DA)? The layered map turns "the data is wrong" into a specific, ownable failure.

Output.

Failure	Where on the map	Owner
Dashboard shows stale data	raw layer not fresh	DE
`dim_customer` has duplicates	marts failing tests	AE
Two dashboards disagree on revenue	no single semantic definition	AE
Dashboard filters wrong	dashboard query	DA
ML model fed raw instead of marts	DS bypassing the map	DS

Rule of thumb. Draw the layered-ownership map as a flow with explicit handoff contracts, and every "the data is wrong" incident becomes a walk down the map to a single owner. The two handoffs that carry the most weight are DE→AE (raw is fresh and registered) and AE→DA (marts are tested and metrics are defined once); enforce those two and the middle takes care of itself.

Worked example — a tested dbt model with a semantic definition

Detailed explanation. The clearest way to show what an AE owns is the artifact itself: a dbt model that transforms raw data into a tested business entity, plus the semantic definition of a metric built on it. This is the AE's day job made concrete. Walk through a fct_orders model and a revenue metric.

Model. Transform raw.orders into a clean, tested fct_orders.
Tests. Uniqueness, not-null, referential integrity.
Metric. Define revenue once, on top of fct_orders.

Question. Write the dbt model, its tests, and the semantic metric definition an AE owns.

Input.

Artifact	Purpose
`fct_orders.sql`	clean, business-ready orders fact
`fct_orders.yml`	tests + docs (the contract)
`revenue` metric	single definition of revenue

Code.

-- models/marts/fct_orders.sql  (owned by the analytics engineer)
with source as (
    select * from {{ ref('stg_orders') }}      -- staging, not raw
),

cleaned as (
    select
        order_id,
        customer_id,
        status,
        total_cents / 100.0            as revenue_usd,   -- business unit
        ordered_at::date               as order_date
    from source
    where status != 'test'                              -- exclude test orders
)

select * from cleaned

# models/marts/fct_orders.yml — the tests ARE the contract
models:
  - name: fct_orders
    description: "One row per real customer order. Canonical orders fact."
    columns:
      - name: order_id
        tests: [unique, not_null]        # no duplicate orders
      - name: customer_id
        tests:
          - not_null
          - relationships:               # referential integrity
              to: ref('dim_customer')
              field: customer_id

# semantic layer: revenue defined ONCE for every consumer
metrics:
  - name: revenue
    label: "Revenue (USD)"
    model: ref('fct_orders')
    calculation_method: sum
    expression: revenue_usd
    dimensions: [order_date, status]

Step-by-step explanation.

The model reads from stg_orders (staging), not raw.orders. This encodes the layering: the DE lands raw, the AE's staging layer cleans and types it, and the mart builds business entities on staging. The AE never reaches into raw directly from a mart.
The transformation applies business logic the analyst should not have to re-derive: converting cents to dollars, excluding test orders, and casting to a date. Doing this once in the mart means every downstream consumer gets consistent revenue_usd, not fifteen slightly different conversions.
The .yml tests are the contract, enforced in dbt's CI. unique + not_null on order_id guarantees no double-counted orders; the relationships test guarantees every order references a real customer. A change that breaks these fails the build — the same contract-enforcement pattern the platform uses, applied at the model level.
The metrics block defines revenue exactly once, as sum(revenue_usd) over fct_orders. Every BI tool and every analyst references this metric rather than writing their own SUM, which is how the semantic layer guarantees consistent numbers everywhere.
This single pair of files is the AE role in miniature: engineering rigor (version control, tests, CI, modularity via ref) applied to analytics, producing a tested business entity and a single-definition metric that the analyst builds insight on and the DE never has to touch.

Output.

Consumer	What they get
Analyst in Looker	references `revenue` metric — consistent by construction
Another AE's model	`ref('fct_orders')` — reuses the tested fact
dbt CI	fails build if `order_id` dupes or FK breaks
Data scientist	clean `fct_orders` as an ML input, not raw
Auditor	docs + tests describe the contract

Rule of thumb. The AE's deliverable is a tested, documented, single-definition model — not a query. If your metrics live as ad-hoc SUMs in dashboards instead of as versioned semantic definitions on tested marts, you do not have an analytics-engineering layer; you have analysts re-deriving numbers and the inconsistency that follows. Define each metric once, test the model under it, and let CI enforce the contract.

Senior interview question on the analytics engineer boundary

A senior interviewer might ask: "Your company has data engineers and data analysts but no analytics engineers. Symptoms: DEs are swamped writing business transformations they do not have context for, analysts maintain a tangle of untested SQL that produces three different revenue numbers, and nobody owns the semantic layer. Make the case for introducing the analytics-engineering function, define its boundary with DEs and analysts, and describe how you would stand it up and who owns the semantic layer."

Solution Using a defined AE function, a role-boundary contract, and an AE-owned semantic layer

Where AE fits (the missing middle)
==================================

  Data Engineer      -> ingestion, runtime, raw/source layer, infra
        | handoff: raw is fresh + registered
  Analytics Engineer -> staging -> marts (dbt, TESTED) + semantic layer
        | handoff: marts tested; every metric defined once
  Data Analyst       -> dashboards, exploration, insight

Problem today: no AE -> the middle is done by whoever is free
  -> DEs write marts without domain context (swamped + wrong)
  -> analysts write untested SQL (3 revenue numbers)
  -> semantic layer is nobody's -> inconsistency by default

## Stand-up plan for the AE function

### Case
The transformation + semantic middle is currently orphaned.
DEs and analysts are both doing it badly because it is neither's
core role. An AE function owns it with engineering rigor.

### How to stand it up
1. Reskill/hire: promote SQL-strong analysts into AE roles or
   hire AEs; they need SQL + dbt + business context.
2. Move all business transformations OUT of DE pipelines and
   OUT of analyst scripts INTO a dbt project the AEs own.
3. Add tests (unique, not_null, relationships, freshness) to
   every mart -> the tests are the contract, enforced in CI.
4. Declare the semantic layer AE-owned, as code, in the dbt repo.
   Define `revenue` ONCE. All BI tools reference it.

### Boundary
- DE stops at raw/source. AE owns staging onward + semantics.
- Analysts do analysis on AE models; they do NOT maintain
  production models.

### Who owns the semantic layer
- The analytics engineer, as versioned + tested code alongside
  the dbt models. NOT a BI-tool setting, NOT the DE.

Step-by-step trace.

Concern	Before (no AE)	After (AE function)
Business transformation	DEs (no context) + analysts (no rigor)	AEs (context + rigor) in dbt
Testing	none on transformations	unique/not_null/relationships in CI
Revenue definition	3 different numbers	1 semantic definition
DE workload	swamped with marts	back to pipelines + infra
Analyst workload	maintaining SQL tangle	analysis on trusted models
Semantic layer owner	nobody	AE, as code

After introducing the AE function, business transformations move into a tested dbt project the AEs own; DEs return to pipelines and infrastructure where their skills fit; analysts stop maintaining production SQL and do analysis on trusted models; and the semantic layer gets a single owner, collapsing three revenue numbers into one contract-enforced definition. The boundary is unambiguous: DE owns raw and below, AE owns transformation and semantics, analyst owns insight.

Output:

Metric	Before	After
Revenue definitions	3	1 (semantic layer)
Tested transformations	0%	high (CI-enforced)
DE time on business logic	high	low (back to platform)
Analyst time maintaining models	high	low (analysis only)
Semantic-layer ownership	none	AE, versioned + tested

Why this works — concept by concept:

The missing middle — the AE fills the transformation + semantic gap that neither DEs (lack domain context) nor analysts (lack engineering rigor) own well. Naming the role assigns the orphaned middle to someone whose job it is.
Engineering rigor on analytics — version control, tests, CI, and modularity turn the untested-SQL tangle into a tested dbt project; the tests are the contract that stops silent regressions and the three-revenue-numbers problem.
AE-owned semantic layer as code — defining each metric once, in the repo, under CI, is what makes numbers consistent across every tool; putting it in a BI-tool setting or on the DE recreates the inconsistency.
Clean role boundary — DE stops at raw, AE owns the middle, analyst owns insight; the boundary frees DEs to return to infrastructure and analysts to return to analysis, so every role does what it is best at.
Cost — reskilling analysts or hiring AEs, plus standing up a dbt project with CI. The eliminated cost is O(dashboards) re-derived metrics, swamped DEs doing low-context transformation, and the recurring firefighting of inconsistent numbers. Net: transformation cost is paid once per model instead of re-paid per consumer.

Dimensional Modeling
Topic — dimensional-modeling
Dimensional modeling problems on facts, dims, and marts

Practice →

SQL Topic — sql SQL problems on transformation and semantic-layer logic

Practice →

5. Scaling the org: topologies, data mesh and anti-patterns

Scale a data org with Team-Topologies mapping and deliberate data-mesh adoption — and fix the anti-patterns before they calcify

The mental model in one line: scaling a data team org past ~40 people is a topology problem — you map the data function onto the four Team-Topologies team types (stream-aligned, platform, enabling, complicated-subsystem), decide deliberately whether and how far to adopt data mesh (domain-owned data products with federated governance), size the org with sane headcount ratios (platform : embedded : analytics engineers), and actively remediate the anti-patterns (central bottleneck, data silos, ungoverned mesh, hero dependency) before they harden into permanent structure. There is no single right topology — there is the topology that matches your scale, your domain complexity, and your governance maturity, plus a discipline of catching anti-patterns while they are still cheap to fix.

Team Topologies mapped onto data.

Stream-aligned teams. The domain/product squads with embedded DEs and AEs, aligned to a flow of business value (checkout, growth, logistics). They own their data products end to end and are the primary delivery teams.
Platform teams. The central data platform team providing the paved road as an internal product, so stream-aligned teams self-serve. Reduces the cognitive load on domain teams.
Enabling teams. A small, time-boxed team that helps stream-aligned teams adopt new capabilities (a new semantic layer, a migration to dbt), then steps back. Enabling teams coach; they do not do the work permanently.
Complicated-subsystem teams. A specialist team for a genuinely hard shared subsystem (a real-time streaming platform, an ML feature store, a complex entity-resolution service) that needs deep expertise most domains lack.
The interaction modes. Team Topologies also defines how teams interact: collaboration (temporary, close), X-as-a-Service (the platform model — clean interface, low coupling), and facilitating (enabling teams coaching). The platform-to-stream relationship should be X-as-a-Service; anything else means the platform is a bottleneck.

Data mesh — the four principles and the honest trade-offs.

Domain ownership. Each domain owns its data as a product, end to end. Powerful at scale; premature below ~40–50 data people because you lack the domains and the platform maturity to support it.
Data as a product. Domains publish discoverable, trustworthy, documented data products with SLAs — not raw dumps. This is the principle even non-mesh orgs should adopt.
Self-serve data platform. Domains cannot own products without a strong platform to build on — mesh requires a mature platform team first. Mesh without a platform is just silos with a rebrand.
Federated computational governance. Global standards (interoperability, security, canonical entities) enforced computationally (contracts, CI) while domains retain autonomy. This is the hardest principle and the one that separates a real mesh from an ungoverned sprawl.
The honest trade-off. Data mesh trades central control for domain autonomy and scalability. It pays off at high scale with many domains and a mature platform; it is an expensive mistake at small scale, where it fragments a team that centralization would serve better.

Headcount ratios — sane starting points, not laws.

Platform : embedded engineers. Roughly 1 platform engineer per 4–6 embedded engineers is a common starting ratio — enough platform to keep the road paved without over-centralizing.
DE : AE. Often 1:1 to 2:1 embedded DE to AE in analytics-heavy domains; DE-heavy in infrastructure-heavy domains. The AE count grows as the transformation/semantic surface grows.
Engineers : analysts. Highly variable; a company with many business stakeholders runs more analysts per engineer. The AE layer is what lets a few engineers support many analysts.
The caveat. Ratios are diagnostics, not targets. A platform team that is 1:20 is under-invested (bottleneck incoming); 1:2 is over-invested (re-centralizing). Use the ratio to spot drift, not to set headcount mechanically.

The anti-pattern catalogue — and the fix for each.

Central bottleneck. Everything queues behind the central team. Fix: self-serve paved road + embedding. (Section 2 and 3.)
Data silos. Embedded teams diverge, no shared substrate. Fix: platform + guild + contracts. (Section 3.)
Ungoverned mesh. Domain ownership without federated governance → a thousand undiscoverable data products. Fix: federated computational governance (catalog + contracts + interoperability standards) before scaling domains.
Hero dependency. One person holds critical undocumented knowledge; the org grinds when they are out. Fix: documentation, on-call rotation, pairing, and bus-factor > 1 as a standard.
Premature mesh. Adopting mesh at 20 people. Fix: centralize or hybrid until you have the domains and platform maturity mesh requires.
Platform ivory tower. Platform builds what it finds interesting, not what domains need. Fix: platform-as-product with user research and adoption metrics.

Common interview probes on scaling.

"How would you map a data org onto Team Topologies?" — stream-aligned domains, platform for the road, enabling teams to coach, complicated-subsystem for hard specialties.
"When should we adopt data mesh?" — when you have many domains, a mature platform, and can afford federated governance; not before.
"What are the ideal data-team ratios?" — ~1 platform per 4–6 embedded; DE:AE 1:1–2:1; use as diagnostics not targets.
"How do you fix a data team hitting a scaling wall?" — diagnose the anti-pattern, map to topology, remediate structurally.

Worked example — mapping a data org onto Team Topologies

Detailed explanation. The most senior scaling artifact is a Team-Topologies map of the data function: which teams are stream-aligned, which is the platform, where an enabling team is needed, and which specialist subsystem warrants its own team — plus the interaction modes between them. Walk through mapping a 60-person data org.

Teams. Six domain squads, one platform team, one streaming specialist team, plus a migration need.
Types. Assign each to a topology type.
Interactions. Define how they interact (X-as-a-Service, collaboration, facilitating).

Question. Map the 60-person data org onto Team Topologies and specify the interaction modes.

Input.

Team	Topology type	Interaction with others
6 domain squads (DE+AE embedded)	stream-aligned	consume platform X-as-a-Service
Data platform team	platform	X-as-a-Service to domains
Streaming/real-time team	complicated-subsystem	X-as-a-Service + collaboration when needed
dbt-migration team (temporary)	enabling	facilitating, then dissolves

Code.

Team-Topologies map: 60-person data org
========================================

  [ Enabling: dbt-migration squad ]  (time-boxed)
        : facilitating (coaches domains onto dbt, then dissolves)
        v
  [ Stream-aligned: 6 domain squads ]  <-- primary delivery
     checkout | growth | logistics | payments | support | catalog
        ^                     ^
        | X-as-a-Service      | X-as-a-Service (+collab when hard)
        |                     |
  [ Platform: data platform ] [ Complicated-subsystem: streaming ]
   paved road, catalog,        real-time / feature store —
   contracts, semantic layer   deep expertise domains lack

Interaction rules:
  - Platform -> domains: X-as-a-Service ONLY. If it turns into
    collaboration/ticketing, the platform is becoming a bottleneck.
  - Streaming team -> domains: X-as-a-Service, with short bursts of
    collaboration for a new hard use case.
  - Enabling team: facilitating, explicitly time-boxed, dissolves
    after the migration.

Step-by-step explanation.

The six domain squads are stream-aligned teams — the primary delivery units, each aligned to a flow of business value with embedded DEs and AEs. In Team Topologies, most teams should be stream-aligned; the others exist to support them.
The data platform team is a platform team whose entire job is to reduce the domains' cognitive load via X-as-a-Service. The critical rule: the interaction mode must be X-as-a-Service (clean self-serve interface), not collaboration. If domains have to collaborate closely with the platform for routine work, the platform has become a bottleneck.
The streaming/real-time team is a complicated-subsystem team — a genuinely hard specialty (exactly-once streaming, feature stores) that most domains cannot staff. It exists because forcing every domain to master streaming would be wasteful; centralizing the hard part is correct.
The dbt-migration team is an enabling team, explicitly time-boxed. It coaches domains onto dbt (facilitating interaction) and then dissolves. The anti-pattern is an enabling team that becomes permanent — that means it started doing the work instead of teaching it.
The interaction modes are the real content of the map. X-as-a-Service from platform to domains is the target state; collaboration should be temporary and intentional; facilitating is for enabling teams. Getting the interaction mode wrong (e.g. platform stuck in collaboration) is how a correctly-typed team still becomes a bottleneck.

Output.

Team	Type	Interaction	Health check
Domain squads	stream-aligned	consume platform as a service	shipping without waiting?
Platform	platform	X-as-a-Service	self-serve rate high?
Streaming	complicated-subsystem	X-as-a-Service + collab	domains not rebuilding streaming?
dbt-migration	enabling	facilitating (time-boxed)	on track to dissolve?

Rule of thumb. Map your data org onto the four Team-Topologies types, but spend your attention on the interaction modes, not the labels. The single most important rule is that platform-to-domain must be X-as-a-Service; the moment it becomes collaboration or ticketing, a correctly-structured platform team has silently become the bottleneck you built it to prevent.

Worked example — a headcount-ratio staffing model

Detailed explanation. When leadership asks "how many people do we need and in what mix," the answer is a staffing model built from ratios and domain count, not a gut number. It makes the ask defensible and surfaces where the org is under- or over-invested. Walk through building one for a company scaling from 6 to 10 domains.

Inputs. Number of domains, complexity per domain, target platform ratio.
Ratios. Platform : embedded, DE : AE, engineers : analysts.
Output. A headcount plan with a rationale per line.

Question. Build a staffing model for a 10-domain org using sane ratios and flag any imbalance.

Input.

Role	Rule of thumb	10-domain sizing
Embedded DE	1–2 per domain	~12 (heavier domains get 2)
Analytics engineer	~1 per domain (more if analytics-heavy)	~10
Platform engineer	1 per 4–6 embedded	~4
Analysts	domain-driven	~15 (business-facing)
Specialist (streaming/ML)	1 shared team	~4

Code.

# Staffing model from ratios + domain count (defensible headcount ask)
domains          = 10
avg_de_per_dom   = 1.2       # some domains need 2
ae_per_domain    = 1.0
platform_ratio   = 5         # 1 platform eng per 5 embedded eng
analysts         = 15        # business-driven, not ratio-driven
specialist_team  = 4         # streaming + ML feature store

embedded_de = round(domains * avg_de_per_dom)      # 12
embedded_ae = round(domains * ae_per_domain)       # 10
embedded    = embedded_de + embedded_ae            # 22
platform    = round(embedded / platform_ratio)     # ~4

print(f"embedded DE : {embedded_de}")
print(f"analytics eng: {embedded_ae}")
print(f"platform    : {platform}   (ratio 1:{platform_ratio})")
print(f"analysts    : {analysts}")
print(f"specialists : {specialist_team}")
print(f"TOTAL       : {embedded_de+embedded_ae+platform+analysts+specialist_team}")

# Imbalance check
if platform < embedded / 8:
    print("WARN: platform under-invested -> bottleneck risk")
if platform > embedded / 3:
    print("WARN: platform over-invested -> re-centralizing risk")

Step-by-step explanation.

The model starts from domain count and complexity, not a target headcount. Ten domains at ~1.2 DEs each gives ~12 embedded DEs; heavier domains (payments, streaming-adjacent) justify two while lighter ones share. Sizing from domains makes the number defensible.
Analytics engineers are sized ~1 per domain, scaled up where the domain is analytics-heavy (growth, finance). The AE count tracks the transformation and semantic surface area, which is what lets a modest analyst count serve many stakeholders.
The platform team is derived from the embedded count via the 1:5 ratio — ~4 platform engineers for ~22 embedded. This keeps the road paved without over-centralizing. The ratio is the control that prevents both under- and over-investment.
Analysts are sized by business demand, not by a ratio to engineers, because analyst need is driven by the number of business questions and stakeholders, not by pipeline count. The AE layer is what keeps this number from exploding.
The imbalance checks turn the model into a diagnostic. If the platform falls below 1:8 of embedded, the code warns of bottleneck risk; above 1:3, it warns of re-centralizing. This is how the ratio is used correctly — to detect drift, not to mechanically set headcount.

Output.

Role	Count	Rationale
Embedded DE	12	~1.2 per domain
Analytics engineer	10	~1 per domain
Platform engineer	4	1:5 of embedded
Analysts	15	business demand
Specialists	4	shared streaming/ML
Total	45	defensible from ratios

Rule of thumb. Build the staffing ask from domain count and ratios, then use the ratios as a drift alarm, not a target. A platform team below ~1:8 of embedded engineers is a bottleneck forming; above ~1:3 it is re-centralizing. Size analysts from business demand and let the analytics-engineering layer be the multiplier that keeps that number sane.

Worked example — anti-pattern remediation

Detailed explanation. The final scaling skill is recognizing an anti-pattern from its symptoms and applying the structural fix rather than a band-aid. Each anti-pattern has a signature and a known remedy. Walk through diagnosing and fixing four common ones in a scaling org.

Signatures. The observable symptoms of each anti-pattern.
Root cause. The structural gap behind each.
Fix. The remediation, plus what not to do.

Question. Diagnose four anti-patterns from their symptoms and prescribe the structural fix for each.

Input.

Anti-pattern	Signature symptom	Structural fix
Central bottleneck	everything behind one queue	self-serve road + embedding
Data silos	N stacks, divergent metrics	platform + guild + contracts
Ungoverned mesh	1000 undiscoverable products	federated computational governance
Hero dependency	org stalls when X is on leave	docs + rotation + bus factor > 1

Code.

Anti-pattern remediation runbook
================================

1) CENTRAL BOTTLENECK
   Symptom : lead time = weeks; central team backlog grows.
   Root    : services-desk platform, no self-serve.
   Fix     : paved-road self-serve + embed into domains.
   NOT     : add headcount to the queue (queue returns).

2) DATA SILOS
   Symptom : 6 orchestrators, 3 revenue definitions.
   Root    : embedding without platform/guild/contracts.
   Fix     : platform owns tooling; guild owns standards;
             canonical data products with contract-CI.
   NOT     : re-centralize everything (loses domain speed).

3) UNGOVERNED MESH
   Symptom : hundreds of data products, none discoverable,
             no interop, duplicated entities.
   Root    : domain ownership WITHOUT federated governance.
   Fix     : catalog + interop standards + contract-CI +
             canonical shared entities BEFORE more domains.
   NOT     : abandon mesh and re-centralize (throws away scale).

4) HERO DEPENDENCY
   Symptom : critical pipeline only one person understands.
   Root    : no docs, no rotation, bus factor = 1.
   Fix     : document, pair, on-call rotation, ownership in the
             catalog, bus factor > 1 as a standard.
   NOT     : give the hero a retention bonus and hope.

Step-by-step explanation.

The central bottleneck's signature is week-long lead times and a growing central backlog. The root is a services-desk operating model; the fix is a self-serve paved road plus embedding. The wrong move — adding headcount to the queue — treats the symptom, and the queue regrows to the new capacity.
Data silos show up as multiple orchestrators and divergent metric definitions. The root is embedding without the coordinating structures. The fix is platform-owned tooling, a guild for standards, and canonical data products with contract CI. The wrong move is full re-centralization, which sacrifices the domain speed embedding bought.
Ungoverned mesh is the failure specific to premature or under-governed mesh: hundreds of undiscoverable, non-interoperable data products. The root is domain ownership without federated computational governance. The fix is a catalog, interoperability standards, and contract CI before scaling more domains. The wrong move is abandoning mesh entirely, throwing away the scalability it offers.
Hero dependency is a bus-factor-of-one problem: a critical pipeline only one person understands. The root is missing documentation, rotation, and shared ownership. The fix is documentation, pairing, on-call rotation, and cataloged ownership so bus factor exceeds one. The wrong move is a retention bonus, which pays to preserve the risk instead of removing it.
Across all four, the pattern is identical: match the symptom to the anti-pattern, fix the structural root, and refuse the band-aid that treats the symptom. Anti-patterns caught early are cheap structural adjustments; caught late they are calcified reorgs.

Output.

Anti-pattern	Band-aid (wrong)	Structural fix (right)
Central bottleneck	more headcount	self-serve + embed
Data silos	re-centralize	platform + guild + contracts
Ungoverned mesh	abandon mesh	federated governance first
Hero dependency	retention bonus	docs + rotation + bus factor > 1

Rule of thumb. Every data-org anti-pattern has a signature symptom, a structural root, and a band-aid that will tempt you. Diagnose to the root, apply the structural fix, and refuse the band-aid — more headcount for a bottleneck, re-centralization for silos, or a retention bonus for a hero all treat symptoms while the root calcifies. Catch anti-patterns while they are still cheap structural adjustments.

Senior interview question on scaling a data team

A senior interviewer might ask: "Your data org has grown to 70 people organized as one big central team plus a few analysts scattered in business units. Delivery has slowed to a crawl, the central team is a bottleneck, business units are hiring shadow analysts and building their own pipelines, and the CTO is asking whether you should 'do data mesh.' Design the target topology, decide how far to go toward data mesh, give a staffing model, and sequence the transition without a big-bang reorg."

Solution Using a Team-Topologies target, deliberate partial mesh, a ratio-based staffing model, and an incremental transition

Target topology (Team Topologies)
=================================
  Stream-aligned : domain squads (embedded DE+AE), aligned to
                   business value; own their data products.
  Platform       : central data platform as X-as-a-Service (road).
  Enabling       : time-boxed team to coach domains onto the road.
  Complicated-   : streaming/ML feature-store specialists.
  subsystem

How far toward data mesh:
  - Adopt "data as a product" + "self-serve platform" NOW.
  - Adopt "domain ownership" INCREMENTALLY (domain by domain).
  - Adopt "federated computational governance" IN LOCKSTEP with
    domain ownership (catalog + contracts + interop standards).
  - This is a PARTIAL, deliberate mesh — not a big-bang.

## Staffing model (70 people, ~8 domains)
- Platform: 8   (1 : ~5 embedded)
- Embedded DE: 12, Analytics eng: 10   (~22 embedded)
- Specialists (streaming/ML): 5
- Analysts: 18 (business-driven, folded IN from shadow hires)
- Enabling: 3 (time-boxed) + leadership/leads: rest

## Transition (no big-bang; 3 quarters)
Q1 — Pave + legitimize
  - Carve platform team from central; ship self-serve road + catalog.
  - Bring shadow analysts into the org (fold, don't fight) with a
    dotted line to the guild.
Q2 — Embed + productize
  - Embed DE+AE into top-3 domains; migrate them onto the road.
  - Declare canonical entities (customer, revenue) with contract-CI.
  - Enabling team coaches each domain, then rotates on.
Q3 — Federate + govern
  - Extend domain ownership to remaining domains (partial mesh).
  - Turn on federated governance: interop standards, catalog
    coverage SLO, contract-CI everywhere. Enabling team dissolves.

Step-by-step trace.

Concern	Before	After
Structure	one central team + scattered analysts	stream-aligned domains + platform + enabling + specialists
Delivery	bottlenecked	self-serve road + embedded pods
Shadow IT	business units building pipelines	folded into domains on the road
Mesh stance	"should we do mesh?"	deliberate partial mesh, governance in lockstep
Governance	central + overloaded	federated computational (catalog + contracts)
Transition	risk of big-bang reorg	incremental, 3-quarter sequence

After the transition, the central bottleneck is replaced by stream-aligned domain squads building on a platform delivered X-as-a-Service; the shadow analysts are legitimized inside domains rather than fought; and data mesh is adopted deliberately and partially — "data as a product" and "self-serve platform" immediately, "domain ownership" and "federated governance" together, domain by domain, never as a big-bang. Governance scales computationally through the catalog and contract CI rather than through an overloaded central review.

Output:

Metric	Before	After
Delivery lead time	weeks	days
Shadow pipelines	proliferating	folded onto the road
Governance model	central, overloaded	federated + computational
Mesh adoption	undecided / risky	deliberate, incremental, governed
Reorg risk	big-bang	3-quarter incremental

Why this works — concept by concept:

Team-Topologies target — mapping to stream-aligned + platform + enabling + complicated-subsystem gives each team a clear type and interaction mode; platform-as-X-as-a-Service is what breaks the bottleneck structurally.
Deliberate partial mesh — adopting data-as-a-product and self-serve now, but domain ownership and federated governance in lockstep and incrementally, captures mesh's scalability while avoiding the ungoverned-mesh anti-pattern that premature adoption causes.
Fold the shadow IT — legitimizing shadow analysts inside domains on the paved road converts a governance problem into embedded capacity, instead of fighting a losing battle against local demand.
Ratio-based staffing — sizing platform at ~1:5 of embedded and analysts from business demand makes the headcount ask defensible and keeps the platform from becoming either a bottleneck or an ivory tower.
Incremental transition — a 3-quarter, domain-by-domain sequence with a time-boxed enabling team avoids the big-bang reorg that stalls delivery for two quarters; each domain is migrated and governed before the next starts.
Cost — a platform team, an enabling team (time-boxed), and the governance substrate (catalog + contract CI). The eliminated cost is the compounding O(domains²) reconciliation of an ungoverned mesh, the perpetual central-queue tax, and the shadow-IT duplication. Net: coordination and governance cost grow near-linearly with domains under federated computational governance, instead of exploding.

Design
Topic — design
Design problems on scaling data platforms and mesh

Practice →

Streaming
Topic — streaming
Streaming problems for the complicated-subsystem team

Practice →

Cheat sheet — data-team-structure recipes

Three archetypes, one decision rule. Centralized = strong standards, bottleneck failure mode. Decentralized/embedded = fast + domain-accurate, silo failure mode. Hybrid (platform + embedded) = paved road plus domain speed, fails only if the platform re-centralizes. Diagnose your current failure mode first, then pick the archetype whose failure mode you can afford. Never copy another company's org chart.
Conway's Law is the mechanism. Your pipelines, schemas, and ownership boundaries will mirror your communication structure. To get one canonical customer product, create one owning team plus a versioned, test-enforced contract and a real consumer channel. If you cannot name the single owner, you have a fork waiting to happen.
Platform-as-product checklist. Users are engineers; golden path over bespoke; roadmap not backlog; self-serve is the goal; publish SLAs/SLOs with the query that measures them; run on-call for the substrate; state an explicit anti-goal ("we are NOT a request desk"). Measure success by adoption and self-serve rate, never tickets closed.
Paved-road capability map. For each capability (ingestion, orchestration, catalog, quality, semantic layer) publish the golden path, the self-serve status, and an SLO. No golden path → domains silo; no SLO → domains cannot plan. Deliver governance as guardrails ON the road (auto-PII masking, contract CI), so the compliant path is the easy path.
Data contract + CI gate. Give the platform a data-contract primitive (schema + semantics + quality) enforced by a CI gate that fails breaking-change PRs. This scales governance at the speed of every pipeline instead of a central review queue. Breaking changes ship as a new version with a deprecation window, never in place.
Embedding: solid line + dotted line. Embedded DEs/AEs report solid line to the domain (delivery, accountability) and dotted line to a guild (standards, craft, career). Solid-line-only → silos; dotted-line-only → outsiders with no domain context. The two RACI rows that matter most: tooling (platform Accountable, stops fragmentation) and career (guild + squad shared, stops attrition).
Guild charter, not a status meeting. A real guild owns standards, an RFC process, weekly design review, a quarterly standards refresh, and an in-place staff/principal ladder. The career ladder is the retention mechanism; the refresh is the anti-rot mechanism. Never embed a lone engineer into a domain with no paved road to build on.
AE = the tested middle. The analytics engineer owns transformation (dbt staging → marts) and the semantic layer as versioned, tested code between DE pipelines and analyst dashboards. DE stops at raw; AE owns staging onward; analysts do analysis on AE models, not production SQL. If metrics disagree across dashboards, you have no single semantic owner.
Role-boundary matrix. One owner per stack layer: DE = ingestion/raw/infra, AE = transformation + semantic layer, DA = dashboards/insight, DS = ML features. Contested layers default to AE (transformation and semantic layer). Smell tests: analysts maintaining prod models = AE under-resourced; DEs writing marts = AE role missing; three revenue numbers = no semantic owner.
Team Topologies mapping. Stream-aligned domain squads (primary delivery), platform team (X-as-a-Service road), enabling team (time-boxed coaching, then dissolves), complicated-subsystem team (streaming/ML specialists). Spend your attention on interaction modes: platform → domain must be X-as-a-Service; the moment it becomes collaboration/ticketing, the platform is the bottleneck.
Data mesh readiness test. Adopt "data as a product" and "self-serve platform" early; adopt "domain ownership" and "federated computational governance" together, incrementally, only when you have many domains and a mature platform. Mesh without a platform is silos with a rebrand; domain ownership without federated governance is the ungoverned-mesh anti-pattern.
Headcount ratios (diagnostics, not targets). ~1 platform engineer per 4–6 embedded; DE:AE roughly 1:1 to 2:1; analysts sized by business demand with the AE layer as multiplier. Below ~1:8 platform-to-embedded = bottleneck forming; above ~1:3 = re-centralizing. Build the ask from domain count and ratios; use the ratio to spot drift.
Anti-pattern → structural fix. Central bottleneck → self-serve + embed (not more headcount). Silos → platform + guild + contracts (not re-centralize). Ungoverned mesh → federated governance first (not abandon mesh). Hero dependency → docs + rotation + bus factor > 1 (not a retention bonus). Premature mesh → hybrid until mature. Always fix the root, refuse the band-aid.

Frequently asked questions

What is a data platform team?

A data platform team is the central group that owns the shared data substrate — ingestion, orchestration, storage, catalog, governance, and often the semantic layer — and provides it to the rest of the organization as a self-serve internal product rather than as a services desk. Its users are the domain engineers (embedded DEs and analytics engineers), and its job is to build a paved-road golden path so those teams can self-serve common needs (onboarding a source, scheduling a pipeline, discovering a dataset) instead of queuing behind a central team. A well-run platform team publishes SLAs/SLOs, runs on-call for the substrate, and measures itself by adoption and self-serve rate, not tickets closed. The defining test of a healthy platform team is whether it thinks "product" (build the road once) or "service" (do the work per request) — the latter becomes the org's bottleneck.

Central platform vs embedded data engineers — which is better?

Neither is better in isolation; the strongest data platform team structure at scale is the hybrid where a central platform team owns the shared substrate as a product and embedded data engineers build domain data products on top of it. A purely central data platform gives you strong standards and one warehouse but becomes a bottleneck as demand outpaces headcount and sits far from domain knowledge. Purely embedded data engineers give you fast iteration and deep domain context but drift into silos — divergent tools, duplicated infrastructure, and inconsistent metrics — because nobody owns the shared platform. The hybrid captures both: the platform paves the road (self-serve ingestion, contracts, catalog) and embedded engineers drive on it, reporting solid line to their domain for delivery and dotted line to a guild for standards. The centralized vs decentralized data team debate is really a question of what stage you are at — centralize early, hybridize as you scale.

What is an analytics engineer?

An analytics engineer is the modern-data-stack role that owns the transformation layer between the data engineer's pipelines and the data analyst's dashboards — turning raw, ingested data into clean, tested, documented, business-ready models (typically in dbt) and often owning the semantic/metrics layer on top. The role emerged because dbt and the ELT pattern moved transformation out of the pipeline and into the warehouse, and someone with software-engineering rigor (version control, testing, CI, modularity) and business context had to own that middle. In practice the analytics engineer writes modular SQL models, adds tests (uniqueness, not-null, referential integrity, freshness), documents the models, and defines each metric once in the semantic layer so every downstream tool computes it the same way. The clean boundary: the DE owns raw and below, the AE owns transformation and semantics, and the analyst does analysis on the AE's trusted models.

When should we adopt data mesh?

Adopt data mesh when you have many domains, a mature self-serve platform team, and the ability to enforce federated computational governance — typically past ~40–50 data people, not before. Data mesh has four principles: domain ownership of data as a product, data-as-a-product thinking, a self-serve data platform, and federated computational governance. The mistake is adopting "domain ownership" without the other three: mesh without a strong platform is just silos with a rebrand, and domain ownership without federated governance produces the ungoverned-mesh anti-pattern — hundreds of undiscoverable, non-interoperable data products and duplicated entities. The pragmatic path is a partial, deliberate mesh: adopt data-as-a-product and self-serve platform early, then extend domain ownership incrementally, domain by domain, with federated governance (catalog, interoperability standards, contract CI) turned on in lockstep. Data mesh trades central control for domain autonomy and scalability, which pays off only when your scale and platform maturity can support it.

DE vs AE vs analyst — what's the difference?

The three roles own three consecutive layers of the stack. The data engineer owns ingestion, the pipeline runtime, infrastructure, orchestration, and the raw/source layer landing in the warehouse — their job is trustworthy raw data at scale. The analytics engineer owns the transformation from raw to business-ready (staging → intermediate → marts in dbt), the tests and documentation on those models, and typically the semantic/metrics layer — their job is turning raw data into tested, reusable, single-definition models. The data analyst owns the last mile: exploration, dashboards, ad-hoc analysis, and turning trusted models into insight and recommendations. The clean rule is one owner per layer: DEs stop at raw, AEs own the middle, analysts own insight. Warning signs that the boundary has blurred: DEs writing business marts (AE role missing), analysts maintaining production models (AE layer under-resourced), or three different revenue numbers across dashboards (no single semantic owner).

What are the ideal data-team ratios?

There is no universal ratio, but sane starting diagnostics are roughly 1 platform engineer per 4–6 embedded engineers, an embedded DE:AE ratio of about 1:1 to 2:1 (more AE-heavy in analytics-intensive domains, more DE-heavy in infrastructure-heavy ones), and an engineers-to-analysts ratio driven by business demand rather than by a fixed number — with the analytics-engineering layer acting as the multiplier that lets a modest engineering team support many analysts. Treat these as drift alarms, not targets: a platform team below ~1:8 of embedded engineers is a bottleneck forming, while above ~1:3 it is re-centralizing and over-investing. Build the actual headcount ask from your domain count and complexity — roughly one to two embedded DEs and about one AE per domain, plus a shared specialist team for genuinely hard subsystems (streaming, ML feature stores) — then use the ratios to check the mix is balanced. The right data team org size is derived from domains and ratios, defended as a staffing model, never guessed.

Practice on PipeCode

Drill the system design practice library → for the data-platform, ownership, and org-topology scenarios that leadership and senior interviews love.
Rehearse the modelling foundations on the dimensional-modeling practice library → for the facts, dimensions, and marts an analytics engineer owns.
Sharpen the pipeline axis with the ETL practice library → for the ingestion and orchestration paved-road patterns a central platform team provides.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the archetype, platform-as-product, and AE-boundary decisions against real graded inputs.

Lock in data-org design muscle memory

Docs describe org charts. PipeCode drills teach the decision — when centralization becomes a bottleneck, when embedding drifts into silos, where the analytics-engineer boundary sits, and when data mesh earns its place. Pipecode.ai is Leetcode for Data Engineering — design-first practice tuned for the org and architecture trade-offs senior data engineers and data leaders actually defend.

Practice design problems →
Practice SQL problems →

Data Engineering Manager Interview Prep: People, Roadmaps, Platform vs Product Trade-Offs

Gowtham Potureddi — Sat, 01 Aug 2026 13:27:04 +0000

The data engineering manager interview is the one loop where your ability to write a flawless window function stops being the thing that gets you hired — and the switch catches almost every strong individual contributor off guard. You have spent years being rewarded for shipping the pipeline, tuning the query, and closing the incident yourself; now a panel wants to know whether you can build a team that ships those things without you, sequence a year of work against a headcount you do not fully control, and defend a platform investment to a VP who only cares about the next product launch. That is a different muscle, and it is trained differently. The interviewer is not testing whether you can do the work — they assume you can — they are testing whether your judgment scales past your own two hands.

This guide is the senior walkthrough you wished existed the first time a hiring manager said "tell me about a time you managed someone out," or "you just lost two engineers mid-quarter — what do you cut?", or "why would you spend a whole quarter on an internal platform instead of shipping the feature the business asked for?" It covers the four tracks every panel probes — people management (hiring, growth, performance, conflict), roadmap planning (prioritization, capacity, dependencies), the platform vs product trade-off that defines data-team leadership, and the cross-functional behavioral loop where team leadership is judged through STAR stories and metrics. Each section pairs a teaching block with a Solution-Tail interview answer — a template or script, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps on the architecture and trade-off muscles the panel will probe, drill the design practice library →, keep the fundamentals sharp on the SQL practice library →, and rehearse the prioritization instinct on the optimization practice library →.

On this page

Why the DE manager interview tests judgment, not code
People management: hiring, growth, performance, and conflict
Roadmaps, planning, and prioritization under constraints
Platform vs product: the central data-team trade-off
Cross-functional influence, metrics, and the behavioral loop
Cheat sheet — EM interview recipes
Frequently asked questions
Practice on PipeCode

1. Why the DE manager interview tests judgment, not code

Four interview tracks, one thing being measured — can your judgment scale past your own hands

The one-sentence invariant: the engineering manager loop is not a harder version of the IC loop — it is a different exam, one where every question is secretly measuring whether you can turn ambiguous business pressure into a sequence of good decisions that other people execute, and where "I would just do it myself" is the fastest way to fail. A senior IC is hired for throughput and depth; a manager is hired for leverage — the multiplier you apply to a team of five or eight or twelve. The panel splits that leverage into four tracks and grades each one, because a manager who is brilliant at roadmaps but cannot have a hard performance conversation will quietly lose their best people, and a manager who is a beloved coach but cannot say no to a stakeholder will burn the team out on a roadmap of everyone-else's-priorities.

The four tracks every DE manager loop probes.

People. Hiring, onboarding, growth, performance management, conflict, retention. This is the track most IC candidates under-prepare and it is usually the highest-weighted. The question behind the question is: "will the team be stronger or weaker a year after you take it over?"
Project / roadmap. Planning a quarter or a year, prioritizing under finite capacity, sequencing dependencies, negotiating scope. The question behind the question is: "when reality changes — headcount, deadlines, an outage — do you re-plan with a clear head or thrash?"
Technical / architecture judgment. Not "write this query" but "review this design," "make this build-vs-buy call," "decide when the platform investment is worth it." The question behind the question is: "can you still tell a good decision from a bad one now that you are not the one typing?"
Cross-functional. Managing up to your director and VP, sideways to product and analytics, and down to your team; running incidents; representing the data org to the rest of the company. The question behind the question is: "can you get things done through influence, not authority?"

What actually changes in the IC to EM transition.

Your output is the team's output. On your best week as a manager you may write zero production code and still have been enormously effective — because you unblocked three people, killed a doomed project, and closed a great hire. Candidates who describe their impact in terms of their own deliverables have not made the shift.
Your feedback loop gets slower and noisier. A failing test tells you in seconds. A bad hire or a demotivated senior engineer tells you in months. Managers who need fast, clean feedback to feel effective struggle; the job rewards patience and leading indicators.
You trade depth for breadth. You will know less about every system than the ICs who own them — and your job is to make peace with that, ask sharp questions, and trust the team, while keeping enough technical depth to smell a bad plan.
Your influence is mostly indirect. You rarely get to command. You set context, remove obstacles, coach, and decide the few things only you can decide. "I told them to" is a weak answer; "I gave them the context and the constraints and they made the call" is a strong one.

What interviewers listen for.

Do you describe impact as team outcomes, not personal heroics? — senior signal.
Do you say "it depends, here are the two or three things I'd need to know" before committing to an answer, rather than jumping to a solution? — required behaviour.
Do you name the trade-off in every decision — what you gave up to get what you got? — senior signal.
Do you own failures plainly and describe the system change you made after, not just the recovery? — required behaviour.
Do you distinguish decisions you'd make yourself from decisions you'd delegate and say why? — senior signal.

Worked example — the four-track EM interview rubric

Detailed explanation. The single most useful artifact for an EM loop is a memorised map of what each interviewer on the panel is grading, because the loop is deliberately split so that no single conversation covers everything. If you know which track you are in, you can steer your answer to the signal that interviewer needs. Walk through building the rubric for a typical five-round data-engineering-manager loop.

Round shape. Recruiter screen, hiring-manager (people + delivery), a peer manager (cross-functional), a technical/architecture panel, and a skip-level or director (strategy + values).
What varies. Each round weights one or two tracks heavily and touches the others lightly.
The trap. Giving the same "here's how I'd architect it" answer in every round — which lands well in the technical round and flat everywhere else.

Question. Map the five rounds of a DE-manager loop to the four tracks and the dominant signal each interviewer is trying to extract.

Input.

Round	Interviewer	Dominant track	Secondary track
Screen	Recruiter	People (motivation)	Cross-functional
Hiring manager	Your future boss	People + Project/roadmap	Judgment
Peer manager	Another EM	Cross-functional	Project/roadmap
Technical panel	Senior/Staff ICs	Judgment (architecture)	People (mentoring)
Skip-level	Director / VP	Strategy + values	People

Code.

EM interview-track rubric — what to lead with per round
=======================================================

Recruiter screen
  Lead with: why management, why this team, one team-outcome story.
  Avoid: deep architecture; save it.

Hiring manager (your future boss)
  Lead with: how you run a team (1:1s, planning), a hard people call,
             how you'd approach the first 90 days here.
  They are imagining working with you daily.

Peer manager (another EM)
  Lead with: a cross-team conflict you resolved, how you negotiate
             scope and dependencies, how you handle a dropped handoff.
  They are checking: are you a good neighbour or a territory-grabber?

Technical / architecture panel
  Lead with: judgment, not recall. Ask clarifying questions, name
             trade-offs, say what you'd delegate vs decide yourself.
  They fear: a manager who has gone stale and rubber-stamps bad designs.

Skip-level (director / VP)
  Lead with: how you connect the data roadmap to business outcomes,
             platform-vs-product thinking, values under pressure.
  They are checking: can you own a mission, not just a backlog?

Step-by-step explanation.

The recruiter screen is a motivation and communication filter — they are checking that you genuinely want to manage (not that you see it as a promotion you're owed) and that you can tell a crisp story. Leading with architecture here wastes the round.
The hiring-manager round is the highest-stakes one because that person will live with the decision daily. They weight people and delivery: how you run 1:1s, how you plan, how you handle a struggling engineer. Bring concrete mechanisms, not platitudes.
The peer-manager round is a "good neighbour" test. Other EMs want to know whether you will fight fair over shared roadmap, own your handoffs, and escalate cleanly. A story about resolving a cross-team dependency conflict is gold here.
The technical panel is judgment, not a coding gauntlet. Senior ICs are terrified of a manager who has gone stale and approves bad architecture. Demonstrate you can still reason — ask the clarifying questions, name the trade-offs, and be explicit about what you'd trust the team to decide.
The skip-level round is strategy and values. Directors want a manager who owns a mission and connects the data roadmap to business outcomes. This is where platform-vs-product thinking and "what would you do if we cut your budget 20%" live.

Output.

Track	Rounds that weight it	If you skip it
People	Screen, hiring manager, skip-level	Read as "still an IC at heart"
Project / roadmap	Hiring manager, peer manager	Read as "can't run a quarter"
Judgment	Technical panel, hiring manager	Read as "gone stale technically"
Cross-functional	Peer manager, screen	Read as "will create silos"

Rule of thumb. Before every round, ask yourself "which track is this person grading?" and lead with a story tuned to that signal. The same generic answer in five rounds reads as one-dimensional; five tuned answers read as a rounded manager.

Worked example — the IC-vs-EM time-allocation shift

Detailed explanation. Interviewers frequently probe the transition by asking "how do you spend your week?" or "what did you stop doing when you became a manager?" The strong answer is quantitative and honest: a real week reallocates most hours away from personal delivery and toward people and planning. Walk through the before-and-after allocation and what it reveals.

The IC week. Dominated by focused build time — designing, coding, reviewing, debugging.
The EM week. Dominated by 1:1s, planning, unblocking, hiring, and cross-functional syncs, with a small protected slice for hands-on work to stay credible.
The failure mode. A new manager who keeps 60% build time — they are doing two jobs badly and starving the team of attention.

Question. Contrast a healthy IC week with a healthy first-line-EM week and identify the biggest reallocation.

Input.

Activity	IC week (hrs)	EM week (hrs)
Focused build (code/design/debug)	28	6
Code review + design review	6	5
1:1s and coaching	1	8
Planning / roadmap / prioritization	2	8
Cross-functional + managing up	1	7
Hiring (screens, loops, sourcing)	0	4
Incident / on-call leadership	2	2

Code.

Time-allocation answer template
===============================

"As an IC my week was ~70% build time. As a first-line manager,
 build time drops to well under 20% and mostly moves to design
 reviews and small unblocking tasks — I keep just enough hands-on
 work to review architecture credibly and to backfill in an
 emergency, never on the critical path.

 The hours flow into three buckets:
   1. People — 1:1s, coaching, growth, feedback (~8h).
   2. Planning — roadmap, prioritization, capacity (~8h).
   3. Cross-functional + hiring — managing up, partner syncs,
      interview loops (~11h).

 The mistake I avoided was staying on the critical path for
 delivery. The first time a sprint slipped because *I* was the
 bottleneck, I learned to treat my own coding capacity as zero
 for planning purposes."

Step-by-step explanation.

Naming the roughly-70%-to-under-20% build-time drop signals you understand the job is fundamentally different, not "IC plus some meetings." Interviewers hear this shift as evidence you've actually made the transition rather than aspiring to it.
Keeping a small, deliberate slice of hands-on work — design reviews, prototypes, glue code off the critical path — is the credible middle position. Claiming zero technical work reads as detached; claiming heavy coding reads as unable to let go.
The three destination buckets (people, planning, cross-functional) map directly onto the four interview tracks, so this answer doubles as a preview that you know where a manager's leverage comes from.
Explicitly disowning the critical path — "I treat my coding capacity as zero when planning" — is the senior move. New managers who keep themselves in the delivery plan create a single point of failure and can't do the actual job when a crisis hits.
Attaching the lesson to a concrete failure ("the first time a sprint slipped because I was the bottleneck") turns an abstract principle into evidence, which is what behavioral interviewers reward.

Output.

Signal	Weak version	Senior version
Build time	"I still code a lot"	"under 20%, off the critical path"
Where hours go	vague "more meetings"	people / planning / cross-functional
Self on the plan	counts own capacity	treats own capacity as zero
Evidence	assertion	a specific slip they learned from

Rule of thumb. Answer "how do you spend your week?" with numbers and a named failure. The reallocation away from personal build time is the whole point — if your week still looks like an IC's, you haven't made the shift the panel is checking for.

Worked example — the "why management" answer template

Detailed explanation. Almost every EM loop opens with some form of "why do you want to manage?" It is a trap for two common bad answers: the status answer ("it's the next step / more money / a bigger title") and the control answer ("I want to decide how things get built"). The strong answer is about deriving energy from other people's growth and from multiplying impact. Walk through constructing an honest, non-clichéd version.

Avoid the status framing. Management is a change of profession, not a promotion above ICs; senior ICs can out-earn and out-rank managers.
Avoid the control framing. Wanting to manage so you can dictate design is a red flag for micromanagement.
Ground it in evidence. The best "why management" answers point at things you already did — mentoring, leading without the title, unblocking the team — that you found energizing.

Question. Draft a "why do you want to be a manager?" answer that is honest, evidence-backed, and free of the two clichéd traps.

Input.

Ingredient	Weak answer	Strong answer
Motivation	"next step in my career"	"I get more energy from the team winning than from my own PR merging"
Evidence	none	"I already mentor two juniors and led the migration without the title"
Trade-off awareness	ignores the downside	"I know I'll code less and my feedback loop gets slower — I've made peace with that"
Failure honesty	"I'll be great at it"	"the part I'll have to work at is patience with slow signals"

Code.

"Why management" answer template
================================

Hook (motivation, honest):
  "Over the last two years the work I found most energizing wasn't
   my own delivery — it was unblocking the team, mentoring the two
   juniors on my squad, and leading the warehouse migration even
   though I didn't have the title. When my PR merges I feel fine;
   when someone I coached ships something hard and grows from it,
   that's the part I want more of."

Evidence:
  "I already do a lot of the job informally — running planning,
   representing the team in cross-functional syncs, giving feedback."

Trade-off awareness:
  "I've thought about the costs. I'll write far less code, my
   feedback loop gets slower and noisier, and I'll know less about
   each system than the ICs. I've made peace with all three."

The honest gap:
  "The muscle I'll have to build is patience — a bad hire or a
   demotivated engineer takes months to show up, unlike a failing
   test. I'm working on trusting leading indicators."

Step-by-step explanation.

The hook grounds motivation in a genuine energy source — other people's growth and the team's wins — which is the trait that predicts a happy, durable manager. Interviewers have heard the status answer a thousand times; the growth answer stands out.
The evidence section proves you're not romanticizing the role: you already do the informal version (mentoring, planning, representing the team) and still want more of it. Wanting to manage after tasting it is far more credible than wanting it in the abstract.
Naming the trade-offs — less code, slower feedback, less depth — pre-empts the interviewer's biggest fear that you don't understand what you're signing up for. Volunteering the costs is a senior move.
Admitting the honest gap (patience with slow signals) shows self-awareness without torpedoing yourself. Choose a real, non-disqualifying growth area — "patience" and "delegation" are safe; "I struggle with conflict" in a people-heavy role is not.
The whole answer avoids both traps: no "next step / more money" and no "I want to control the design." It reads as someone changing profession with eyes open.

Output.

Component	Purpose	Interviewer reads it as
Growth-energy hook	show the right motivation	"will enjoy the actual job"
Informal evidence	prove it's tested, not romantic	"already doing the role"
Trade-off list	show eyes-open realism	"understands the costs"
Honest gap	show self-awareness	"coachable, self-aware"

Rule of thumb. Answer "why management?" with an energy source (other people's growth), evidence you already do the job informally, and an explicit list of the trade-offs you've accepted. Never lead with title, money, or control — those three are the fastest disqualifiers in the loop.

Senior interview question on the IC-to-EM transition

A senior interviewer often opens with: "You are a Staff data engineer who has never formally managed. Convince me you are ready to run a team of six, including two engineers more tenured than you. Walk me through how you'd think about the first 90 days, what you'd stop doing, and how you'd earn the trust of people who were your peers last week."

Solution Using a first-90-days plan built on listening, one team win, and explicit role redefinition

First-90-days plan for a new DE manager (peer-to-boss transition)
=================================================================

Days 0–30 — LISTEN and stabilize
  - 1:1 with every team member: "what's working, what's broken,
    what do you want from a manager, what should I not break?"
  - Map the systems, the on-call reality, the in-flight roadmap.
  - Change almost nothing. Fix only obvious, low-risk pain
    (a broken alert, a missing runbook) to build credibility.
  - Meet every key stakeholder (product, analytics, infra, my boss).

Days 30–60 — DIAGNOSE and align
  - Synthesize the listening tour into 3 themes (e.g. "on-call is
    burning people out", "roadmap has no clear priority", "two
    seniors feel stalled").
  - Co-write a lightweight team charter / operating model with the
    team, not for them: how we plan, how we do 1:1s, on-call rota.
  - Pick ONE visible team win to land by day 90.

Days 60–90 — DELIVER one win and set the operating rhythm
  - Ship the one win (e.g. cut on-call pages 40% via alert cleanup).
  - Stand up the durable rhythm: weekly 1:1s, biweekly planning,
    quarterly growth conversations.
  - Give the two senior engineers explicit scope/ownership so they
    grow through me, not around me.

Explicit role redefinition (the peer-to-boss part)
  - Name it out loud in the first 1:1s: "our working relationship
    is changing; here's how I'll try to be useful, and I need your
    help and candour."
  - Stop competing on output. My job is now their success.
  - Give the more-tenured engineers MORE autonomy, not less —
    trust is the currency that converts former peers into allies.

Step-by-step trace.

Phase	Primary activity	Signal to the team
Days 0–30	Listening tour + stabilize	"listens before acting"
Days 30–60	Diagnose themes + co-write operating model	"involves us, has a plan"
Days 60–90	Land one win + set rhythm	"delivers, not just talks"
Throughout	Redefine the peer relationship explicitly	"handled the awkward part head-on"
Seniors	Grant scope and autonomy	"grows us, doesn't threaten us"

The plan resists the new manager's strongest urge — to prove value by immediately reorganizing everything. It front-loads listening, earns credibility with one concrete win, and treats the peer-to-boss awkwardness as something to name directly rather than pretend away. The two tenured engineers are handled by giving them more ownership, converting a potential rivalry into a partnership.

Output:

Metric of a good transition	90-day target
1:1s established with all reports	100% by day 14
Key stakeholders met	100% by day 30
Visible team win landed	1 by day 90
Team operating model documented	co-written by day 60
Attrition during transition	0 regretted departures

Why this works — concept by concept:

Listen before you act — a new manager has the least context they will ever have; changing things in week one on thin information destroys trust and often breaks something that worked. The listening tour buys context and signals respect.
One visible win — credibility is earned with a concrete outcome (fewer pages, a killed zombie project), not with a reorg. It proves you can deliver through the team early.
Explicit role redefinition — the peer-to-boss shift is awkward; naming it out loud ("our relationship is changing, here's how I'll be useful") disarms the awkwardness that otherwise festers.
More autonomy for senior peers — the instinct to assert authority over former peers backfires. Granting ownership converts tenured engineers into allies who grow through you rather than route around you.
Cost — the plan is deliberately slow on visible change for 30 days, which can feel unproductive and requires discipline to hold. The payoff is durable trust; the alternative — a fast reorg on no context — is O(team) in regretted attrition. Slow-then-steady beats fast-then-firefight.

Design
Topic — design
Design problems that build architecture-review judgment

Practice →

SQL Topic — sql SQL problems to keep your technical credibility sharp

Practice →

2. People management: hiring, growth, performance, and conflict

The manager's core loop is hire → onboard → grow → evaluate → retain — and the interview probes every stage

The mental model in one line: people management is a loop — you hire the right engineers, onboard them to productivity, grow them toward the next level, evaluate them honestly, and retain the ones you want to keep — and the data engineering manager interview probes every stage because the through-line, "will the team be stronger a year from now?", is the single strongest predictor of a good manager. Every stage has a mechanism a strong manager can describe concretely, and vague answers ("I have an open-door policy," "I give feedback regularly") are the fastest way to sound like someone who has read about management but not done it.

Hiring — the highest-leverage decision a manager makes.

Define the bar before you see resumes. Write the role's must-haves and nice-to-haves and the signals you'll test for; otherwise the loop drifts toward "did I like them?"
Structured over vibes. Consistent questions, a rubric, and a debrief where each interviewer commits to a rating before hearing others — this counters groupthink and bias.
Hire for the trajectory, not just today. A data team needs range: SQL and modeling depth, pipeline/infra skill, and increasingly data-quality and stakeholder instincts. Balance the team, don't clone yourself.
A bad hire is more expensive than a slow hire. The cost of a mis-hire — ramp time, the mess they leave, the exit process, team morale — dwarfs the cost of a longer search.

Onboarding — the first 90 days set the ceiling.

A ramp plan, not a laptop and good luck. A written 30/60/90 with a first small shippable task in week one, a buddy, and clear "what good looks like."
Early wins build confidence. Sequence the first tasks from small-and-safe to meaningful, so momentum compounds.

Growth — the retention engine.

The skill/will matrix. Match your style to the person: high-skill/high-will → delegate; high-skill/low-will → re-engage and excite; low-skill/high-will → coach and teach; low-skill/low-will → direct closely (and consider fit).
Career ladders and growth plans. Every engineer should be able to name the two or three things standing between them and the next level, and see you actively creating opportunities to close them.
Sponsorship, not just mentorship. Mentoring is advice; sponsorship is spending your capital to put them on the visible, career-making project.

Performance & conflict — the part IC candidates fear.

Feedback early and specifically. Situation-behaviour-impact, close to the event, no surprises at review time.
The underperformer path. Diagnose (skill? will? fit? context?) → set clear, written, time-boxed expectations → support hard → and if it doesn't turn, part ways with dignity. A formal plan is a tool to help someone succeed, not a paperwork prelude to firing.
Conflict. Get to the interests behind the positions, mediate directly, and don't let two strong engineers' feud quietly tax the whole team.

Common interview probes on people management.

"Tell me about a time you managed an underperformer." — the single most common EM behavioral question.
"How do you grow a senior engineer who's plateaued?" — sponsorship, stretch scope, new domain.
"How do you handle two engineers in constant conflict?" — interests, mediation, boundaries.
"How do you know your 1:1s are working?" — leading indicators: candour, they bring problems early, growth is visible.

Worked example — a 1:1 and growth-plan doc template

Detailed explanation. The 1:1 is the manager's primary instrument, and interviewers probe whether yours are structured (theirs, forward-looking, and captured) or a status meeting in disguise. Pair it with a lightweight living growth plan, and you have the retention engine most teams lack. Walk through a template you can describe verbatim in the loop.

The 1:1 is the report's meeting. Their agenda first; status belongs in tickets, not here.
Capture and follow through. A shared running doc so commitments don't evaporate and growth is visible over time.
The growth plan is living. Current level, target level, the two or three gaps, and the concrete opportunities you're creating to close them.

Question. Design a running 1:1 doc and a growth-plan block that a manager and a mid-level data engineer keep together.

Input.

Section	Owner	Cadence
Their topics	Report	Every 1:1
My topics + feedback	Manager	Every 1:1
Action items	Both	Every 1:1
Growth plan	Both	Reviewed monthly
Career target + gaps	Both	Reviewed quarterly

Code.

# 1:1 — <Report name> / <Manager name>
_Standing doc. Newest notes on top. Their agenda first._

## 2026-08-14
### Their topics (they drive)
- Blocked on Airflow upgrade — needs infra to approve the maintenance window
- Wants to own the data-quality framework next quarter

### My topics / feedback
- Praise: the incident writeup last week was excellent — clear timeline, real root cause
- Growth nudge: in design review, state your recommendation first, then the options

### Action items
- [ ] (me) escalate the Airflow maintenance window to infra by Fri
- [ ] (them) draft a 1-pager on the data-quality framework
- [x] (me) confirmed conference budget approved

---

## Growth plan (reviewed monthly)
- **Current level:** DE II (mid)
- **Target level:** Senior DE
- **Gaps to close:**
  1. Lead a project end-to-end across ≥2 teams (scope + influence)
  2. Raise design-doc quality — drive the decision, not just options
  3. Mentor one junior through a full feature
- **Opportunities I'm creating:**
  - Owns the data-quality framework project (cross-team) starting Q4
  - Pairs with junior on the CDC pipeline
  - Presents the framework design at the DE guild (visibility)

Step-by-step explanation.

Putting their topics first, in a doc they can edit, structurally enforces that the 1:1 is the report's meeting. Managers whose 1:1s are status updates are the ones whose reports feel unheard and eventually leave.
The feedback block pairs a specific piece of praise with one forward-looking growth nudge every session, so feedback is continuous and low-stakes — there are no surprises at review time because the review just summarizes the doc.
Action items with owners and checkboxes make the manager accountable too; a report seeing their manager reliably close "escalate the maintenance window" learns the 1:1 is where problems actually get solved.
The growth plan names the current and target level and — critically — the two or three gaps, so growth is concrete rather than "keep doing great." An engineer who can name what's between them and the next level is a retained engineer.
The "opportunities I'm creating" section is the sponsorship half: the manager is putting the person on cross-team, visible work that closes the gaps. This is the difference between telling someone to grow and building the runway for it.

Output.

Artifact	What it prevents
Their-topics-first agenda	1:1 decaying into status
Continuous SBI feedback	review-time surprises
Owned action items	commitments evaporating
Named growth gaps	"how do I get promoted?" ambiguity
Created opportunities	plateau and attrition

Rule of thumb. Run 1:1s from a shared running doc, their agenda first, with one specific piece of feedback every time and a living growth plan that names the gaps and the opportunities you're creating to close them. If you can describe this doc in an interview, you sound like a manager; if you say "open-door policy," you sound like an IC.

Worked example — a performance-conversation script for an underperformer

Detailed explanation. "Tell me about managing an underperformer" is the most common EM behavioral question, and the strong answer shows a clear, kind, documented process — not avoidance and not a surprise firing. The performance conversation itself has a structure you can rehearse. Walk through a script for a mid-level engineer whose delivery has slipped.

Diagnose first. Is it skill, will, fit, or context (a life event, a bad project match)? The fix differs completely.
Be direct and specific. Name the gap with evidence; don't soften it into unrecognizability.
Own your part. Ask what support has been missing — sometimes the manager is a contributing cause.
Agree on clear, written, time-boxed expectations. And on the support you'll provide.

Question. Script the first performance conversation with an engineer whose last two projects slipped and whose design reviews are shallow, structured so it helps them turn it around.

Input.

Element	Weak approach	Strong approach
Timing	wait for review cycle	now, close to the evidence
Framing	vague ("be better")	specific behaviours + impact
Ownership	all on them	ask what support is missing
Outcome	unspoken threat	clear written expectations + a check-in date
Tone	either soft or harsh	clear AND kind

Code.

Performance conversation script (first, supportive)
===================================================

Open (direct, no ambush):
  "I want to talk about how the last two projects have gone, because
   I think there's a gap and I want to help you close it. This is a
   supportive conversation, not a formal process."

Name it with evidence (SBI):
  "On the ingestion project and the SLA dashboard, both slipped
   more than a sprint past the estimate, and in the last two design
   reviews the docs listed options but didn't land a recommendation.
   The impact is that partners started routing around the team, and
   I had to step in on the review."

Ask (diagnose + own my part):
  "Help me understand what's going on. Is the scope unclear? Are you
   stretched across too much? Is something outside work weighing on
   you? And — is there support I should be giving that I'm not?"

Agree on expectations (clear, written, time-boxed):
  "Here's what 'back on track' looks like over the next 6 weeks:
   design docs that state a recommendation, and the next project
   delivered within its estimate or re-scoped early with a heads-up.
   I'll pair with you on the first design doc and clear two of your
   side tasks so you can focus. Let's check in weekly."

Close (kind, honest):
  "I'm telling you this because I think you can do this level of
   work — that's why the gap is worth closing. I'm on your side."

Step-by-step explanation.

The open removes the ambush: naming that a gap exists and that this is supportive (not yet a formal process) lets the person hear the content instead of panicking about their job. Surprise is the enemy of a productive performance conversation.
Naming the gap with specific situation-behaviour-impact evidence (which projects, which reviews, what the downstream effect was) makes it undeniable and coachable. Vague feedback ("be more proactive") gives them nothing to act on.
Asking what's going on and owning your part does two things: it diagnoses the real cause — skill and will and context need different responses — and it models that this is a shared problem, which keeps the person engaged rather than defensive.
The written, time-boxed expectations convert a fuzzy "do better" into observable targets (docs that state a recommendation; next project on estimate or re-scoped early) with a support commitment and a cadence. Clarity is kindness here.
The close reaffirms belief in the person, which is what makes the whole thing land as help rather than a threat. If a manager doesn't believe the person can succeed, that's a different (exit) conversation — but the first performance conversation should always be a genuine attempt to turn it around.

Output.

Stage	Purpose	If skipped
No-ambush open	let them hear it	defensiveness, panic
SBI evidence	make it coachable	"vague and unfair"
Diagnose + own	find the real cause	wrong fix applied
Written expectations	observable targets	ambiguity, no accountability
Belief close	frame as help	reads as a threat

Rule of thumb. A performance conversation is clear and kind: specific evidence, an honest diagnosis of skill-vs-will-vs-context, written time-boxed expectations, real support, and a genuine statement of belief. If the answer to "manage an underperformer" is a story about avoidance or a surprise firing, it fails; if it's this structure with a real outcome (turned around or parted ways with dignity), it lands.

Worked example — the skill/will matrix for tailoring your management style

Detailed explanation. A frequent probe is "do you manage everyone the same way?" — and the strong answer is no, you diagnose each person on skill and will and adapt. The skill/will matrix is the crisp mental model. Walk through placing four real reports and choosing a style for each.

Skill. Do they have the competence for the task at hand?
Will. Do they have the motivation and confidence for it?
The four quadrants. Each needs a different default: delegate, coach, excite/re-engage, or direct.

Question. Place four data engineers on the skill/will matrix and prescribe the management style and the risk for each.

Input.

Engineer	Skill	Will	Situation
A — senior, thriving	high	high	owns the platform, wants more
B — senior, disengaged	high	low	great but bored, eyeing the door
C — junior, eager	low	high	new grad, hungry, learning fast
D — mid, struggling	low	low	wrong project fit, morale sinking

Code.

Skill/Will matrix — style per quadrant
======================================

              HIGH WILL                     LOW WILL
          +--------------------------+--------------------------+
HIGH      | A: DELEGATE              | B: EXCITE / RE-ENGAGE    |
SKILL     | - autonomy + stretch     | - find the motivation gap|
          | - sponsor for promo      | - new domain / bigger    |
          | - risk: under-challenge  |   scope / more autonomy  |
          |                          | - risk: they leave first |
          +--------------------------+--------------------------+
LOW       | C: COACH                 | D: DIRECT (then decide)  |
SKILL     | - teach, pair, clear     | - close direction + fast |
          |   feedback, safe wins    |   feedback loop          |
          | - risk: overwhelm too    | - diagnose: fit? context?|
          |   fast                    | - risk: is it the right  |
          |                          |   seat, or the right bus?|
          +--------------------------+--------------------------+

Step-by-step explanation.

Engineer A (high/high) gets delegation, stretch scope, and active sponsorship for promotion. The trap is neglect — high performers are easy to ignore because they need the least day-to-day attention, and that's exactly how you lose them; they need growth, not supervision.
Engineer B (high/low) is the most urgent case and the one new managers miss. A bored senior is a flight risk; the job is to find the motivation gap and close it with a new domain, bigger scope, or more autonomy — not more oversight, which accelerates the exit.
Engineer C (low/high) gets coaching: teaching, pairing, clear feedback, and a sequence of safe wins. The risk is overwhelming their eagerness with too much too fast; protect the early confidence-building wins.
Engineer D (low/low) needs close direction and a tight feedback loop and an honest diagnosis: is this a skill gap, a bad project fit, or a context problem? Sometimes it's the wrong seat (fixable by reassignment) and sometimes it's the wrong bus (a fit conversation). Don't confuse the two.
The meta-point for the interview is that great managers apply different styles to different people deliberately, and can explain why. "I treat everyone the same / fairly" sounds nice but is actually a red flag — fair means giving each person what they need, not identical treatment.

Output.

Engineer	Style	Biggest risk if mismanaged
A high/high	delegate + sponsor	neglect → attrition
B high/low	excite / re-engage	flight risk realized
C low/high	coach	overwhelmed, confidence lost
D low/low	direct + diagnose fit	wrong-seat vs wrong-bus confusion

Rule of thumb. Diagnose every report on skill and will and adapt: delegate to high/high, re-engage high/low before they leave, coach low/high with safe wins, and direct-then-diagnose low/low. "I manage everyone the same" is the wrong answer; "I give each person what they specifically need" is the right one.

Senior interview question on people management

A senior interviewer might ask: "You inherit a team with a senior data engineer who is technically the strongest person on the team but is toxic in reviews — dismissive, makes juniors afraid to ask questions, and two people have privately said they'll leave if it continues. Walk me through exactly how you'd handle it, including what you'd do if the behaviour doesn't change."

Solution Using direct feedback, a clear behaviour bar, and a willingness to lose a top performer

Handling a brilliant-but-toxic senior engineer
==============================================

Step 1 — Gather specifics, not vibes
  - Note concrete incidents: dates, what was said, the effect
    ("in Tue review, said 'this is obvious' to the junior who then
     stopped asking questions"). Feedback needs evidence.

Step 2 — Direct, private, specific feedback (assume good intent first)
  "Your technical judgment is the best on the team and I rely on it.
   But in reviews the delivery is landing as dismissive — here are
   two specific moments — and the effect is that juniors have stopped
   asking questions in front of you. That's costing us more than your
   technical output is adding. I need that to change."

Step 3 — Make the standard explicit and non-negotiable
  - Name the behaviour bar: reviews are for making the work and the
    person better; questions are always welcome; disagreement is
    fine, contempt is not. This applies to everyone, seniors included.

Step 4 — Support the change
  - Offer concrete tools: ask questions instead of declaring, praise
    in public / correct in private, review the code not the coder.
  - Check in; acknowledge improvement genuinely when it comes.

Step 5 — If it doesn't change, act — even at a cost
  - Escalate the consequence honestly: performance rating impact,
    then a formal plan, then exit. Document throughout.
  - Be willing to lose the strongest IC. Culture is set by the worst
    behaviour you tolerate from your best people. Two good engineers
    leaving over one toxic one is a terrible trade.

Step-by-step trace.

Step	Action	Underlying principle
1	Collect specific incidents	feedback needs evidence
2	Direct private feedback, good intent	respect + clarity
3	State the non-negotiable bar	standards apply to everyone
4	Support the change	it's coaching, not punishment
5	Act if unchanged, accept the cost	culture > any single IC

The answer refuses the two failure modes: tolerating toxicity because the person is talented (which teaches the team that skill buys a pass on behaviour) and ambushing them with a punishment (which is unfair and usually backfires). It starts with respect and clarity, supports the change, and — crucially — commits to acting even if it means losing the best engineer, because the two departures the toxic behaviour is about to cause are the real business risk.

Output:

Outcome	Result
Behaviour changes	keep a strong engineer, team stabilizes
Behaviour partially changes	keep coaching, monitor, protect juniors
Behaviour doesn't change	manage out; protect the two at-risk engineers
Manager tolerates it	lose 2 good engineers, culture rots

Why this works — concept by concept:

Specific evidence over vibes — "you're toxic" is unactionable and easy to deny; two dated incidents with their effect on named behaviours are undeniable and coachable. Feedback is only useful when it's specific.
Direct feedback with good intent — assuming the person doesn't realize the impact, and saying so plainly and privately, gives the best chance of a change while preserving their dignity.
The non-negotiable behaviour bar — stating that standards apply to everyone, seniors included, prevents the corrosive lesson that talent buys a pass. This is the culture-setting move.
Willingness to lose the top performer — the senior signal. A manager who won't part with a toxic star to save two good engineers is optimizing for short-term output over the team, which is exactly the judgment failure the interview is testing for.
Cost — acting may cost your single strongest IC and some short-term delivery; tolerating it costs two good engineers, the trust of the whole team, and your credibility as a manager. The trade favours acting decisively — the cost of inaction compounds across the whole team, not one person.

SQL
Topic — sql
SQL problems to calibrate your team's technical hiring bar

Practice →

Design Topic — design Design problems for coaching engineers through architecture

Practice →

3. Roadmaps, planning, and prioritization under constraints

A roadmap is a sequence of bets sized to real capacity — not a wishlist, and the interview tests what you cut

The mental model in one line: roadmap planning is the discipline of converting infinite requests into a ranked, capacity-sized, dependency-aware sequence of bets — and because capacity is always finite and reality always changes, the real test of a data engineering manager is not the plan they draw when everything is calm but what they choose to cut when they lose two engineers mid-quarter or the deadline moves. Interviewers probe roadmaps because a manager who can't prioritize will let the loudest stakeholder set the agenda, over-commit the team, and then miss everything.

Prioritization — turning requests into a ranking.

A framework beats a gut feel. RICE (Reach × Impact × Confidence ÷ Effort) or a simpler impact-vs-effort 2×2 forces every request onto the same scale so you're comparing apples to apples.
Score, then sanity-check. The number is an input to judgment, not a replacement for it — a strategic bet with a low RICE can still be right, but you should be able to say why you're overriding the score.
Say no explicitly. Every yes is a no to something else. A roadmap without a visible "not now / not doing" list is a roadmap that's lying about capacity.

Capacity — the math most managers skip.

Headcount is not capacity. Six engineers is not six engineers of output. Subtract on-call, meetings, interviews, holidays, leave, ramp time for new hires, and a keep-the-lights-on tax (incidents, maintenance, support).
Reserve for the unplanned. Leave 15–25% unallocated; a plan booked to 100% has no slack for the incident that will happen.
Sequence around dependencies. If project B needs the platform work from project A, A must land first — a dependency-blind roadmap looks great and delivers nothing on time.

The data-team keep-the-lights-on tax.

Data teams carry unusual toil. Broken upstream schemas, late data, backfills, ad-hoc analyst requests, and on-call for pipeline freshness eat real capacity every week.
Budget it explicitly. If KTLO is 30% of the team's time, a plan that assumes 100% availability for new work is fiction.

Stakeholder trade-offs and communication.

Make the trade-off visible. "If we take on your request, here's what slips" — decisions made in the open build trust; decisions made silently breed resentment.
Re-plan out loud when reality changes. A lost headcount or a moved deadline should trigger a transparent re-rank, not quiet heroics that end in a missed quarter.

Common interview probes on roadmaps.

"How do you prioritize when everything is urgent?" — a framework (RICE), capacity math, and an explicit no-list.
"You lose two engineers mid-quarter — what do you do?" — re-rank, protect the SLA, cut the tail, communicate early.
"How do you handle a VP who drops a top-priority request mid-quarter?" — surface the trade-off, let them choose what slips.
"How do you estimate a data project with unknown data quality?" — spike first, estimate ranges, re-plan after discovery.

Worked example — a RICE prioritization table with a scoring calc

Detailed explanation. RICE forces competing initiatives onto one comparable scale so prioritization is a defensible ranking rather than a shouting match. It's a favourite interview prop because you can walk it end-to-end. Build the table and the scoring calc for four data-team initiatives.

Reach. How many people/systems this affects in a period (users, teams, pipelines).
Impact. How much it moves the needle per unit reached (a 0.25–3 scale).
Confidence. How sure you are of Reach and Impact (a percentage).
Effort. Person-weeks. RICE = (Reach × Impact × Confidence) ÷ Effort.

Question. Score four initiatives with RICE and produce the ranked roadmap order.

Input.

Initiative	Reach	Impact	Confidence	Effort (pw)
Self-serve metrics layer	200	2.0	0.8	16
Migrate legacy Airflow → managed	50	1.0	0.9	8
New revenue dashboard (VP ask)	30	3.0	0.7	4
Data-quality alerting framework	120	1.5	0.8	10

Code.

# RICE scoring — (Reach * Impact * Confidence) / Effort
initiatives = [
    # name,                         reach, impact, conf, effort_pw
    ("Self-serve metrics layer",     200,   2.0,   0.80, 16),
    ("Airflow -> managed migration",  50,   1.0,   0.90,  8),
    ("Revenue dashboard (VP ask)",    30,   3.0,   0.70,  4),
    ("Data-quality alerting",        120,   1.5,   0.80, 10),
]

def rice(reach, impact, conf, effort):
    return round((reach * impact * conf) / effort, 1)

ranked = sorted(
    ((name, rice(r, i, c, e)) for name, r, i, c, e in initiatives),
    key=lambda x: x[1],
    reverse=True,
)

for rank, (name, score) in enumerate(ranked, 1):
    print(f"{rank}. {name:32s} RICE={score}")

# 1. Data-quality alerting              RICE=14.4
# 2. Self-serve metrics layer           RICE=20.0
# 3. Revenue dashboard (VP ask)         RICE=15.8
# 4. Airflow -> managed migration       RICE=5.6

Step-by-step explanation.

Each initiative is scored on the same four factors, which is the whole point: the self-serve metrics layer (RICE 20.0) beats the VP's revenue dashboard (15.8) despite the dashboard's higher raw impact, because the metrics layer reaches far more people. RICE surfaces that reach-times-breadth comparison that gut feel misses.
Effort in the denominator rewards cheap high-value work: the revenue dashboard scores respectably (15.8) mostly because it's only 4 person-weeks. Small-and-valuable rises; big-and-speculative sinks.
The Airflow migration scores lowest (5.6) — it's real work but narrow reach and modest per-unit impact. RICE correctly flags it as "important but not now," which is exactly the kind of item that eats a roadmap if you let infrastructure enthusiasm override the numbers.
Confidence is the honesty knob: the VP dashboard's 0.7 confidence (are the impact assumptions real?) pulls its score down appropriately. Padding confidence to make a pet project win is the most common way people game RICE — and interviewers will ask how you keep confidence honest.
The output is a defensible ranking you can show a stakeholder. When the VP asks why their dashboard is second, you can point at the metrics layer's reach — the conversation is about the inputs, not your taste.

Output.

Rank	Initiative	RICE
1	Self-serve metrics layer	20.0
2	Revenue dashboard (VP ask)	15.8
3	Data-quality alerting	14.4
4	Airflow → managed migration	5.6

Rule of thumb. Use RICE (or a simpler impact/effort 2×2) to turn prioritization into a defensible ranking, keep the confidence factor honest, and treat the score as an input to judgment — you can override it for a strategic bet, but you must be able to say why. A ranking you can defend with inputs beats an opinion you defend with seniority.

Worked example — capacity math and the quarter-plan doc

Detailed explanation. The most common planning error is treating headcount as capacity. Real capacity is headcount minus everything that isn't new-feature work, and interviewers love watching you do the subtraction. Walk through the math for a six-person team and turn it into a quarter plan.

Start with raw weeks. 6 engineers × 12 weeks = 72 person-weeks in the quarter.
Subtract the taxes. On-call, meetings/planning, holidays/leave, ramp for a new hire, and KTLO/incidents.
Reserve slack. Keep ~20% unallocated for the unplanned.

Question. Compute the six-person team's real quarterly capacity and lay out a quarter plan that fits it.

Input.

Deduction	Person-weeks
Raw (6 × 12)	72
On-call rotation	−6
Meetings / planning / 1:1s (~15%)	−10
Holidays + planned leave	−6
New-hire ramp (1 hire, half-productive)	−6
KTLO / incidents / ad-hoc (~20%)	−14
Slack reserve (~15% of remainder)	−5

Code.

# Q4 2026 Data Platform — Quarter Plan
_Team: 6 engineers. Real capacity after taxes: ~25 person-weeks._

## Capacity
- Raw: 72 pw  |  After on-call/meetings/leave/ramp/KTLO/slack: ~25 pw
- Rule: we plan to ~25 pw of NEW work, not 72.

## Committed (fits in ~25 pw)
1. Self-serve metrics layer — phase 1        (16 pw)  [RICE #1]
2. Data-quality alerting — MVP                (8 pw)  [RICE #3]
   → total committed: 24 pw

## Stretch (only if slack materializes)
- Revenue dashboard (VP ask)                  (4 pw)  [RICE #2]

## Not now (explicit no-list)
- Airflow -> managed migration               (8 pw)  [RICE #4] — Q1
- Real-time CDC pilot                          — needs discovery spike first

## Dependencies
- Data-quality alerting depends on the metrics layer's event schema
  → metrics layer phase 1 must land by week 6.

## Risks
- If the new hire ramps slower, drop data-quality alerting to Q1.
- KTLO spike (schema breakages) is the top threat to the plan.

Step-by-step explanation.

The headline number does the teaching: 72 raw person-weeks collapse to about 25 of real new-feature capacity once you subtract on-call, meetings, leave, ramp, KTLO, and slack. A manager who plans to 72 will over-commit by nearly 3× and miss everything.
The new-hire ramp deduction (−6) captures a subtlety interviewers probe: a new hire is a negative to short-term capacity (they consume mentoring time) before they're a positive. Adding a head mid-quarter doesn't add a head of output that quarter.
Committing to 24 of the ~25 available person-weeks leaves the plan realistic; the revenue dashboard is explicitly "stretch," which tells the VP the truth — it happens only if slack materializes — rather than a false promise.
The explicit no-list ("Airflow migration → Q1," "CDC pilot needs a spike") is the most important section. It makes the trade-offs visible and gives you something concrete to point at when a new request lands: "yes, and here's what moves to make room."
The dependencies and risks sections turn the plan from a wishlist into a bet with a sequence and a contingency: the metrics layer must land by week 6 because data-quality alerting depends on its schema, and the named fallback (drop alerting to Q1 if ramp is slow) is the pre-agreed cut.

Output.

Bucket	Person-weeks	Status
Committed	24	fits real capacity
Stretch	4	only if slack appears
Not now	8+	explicit, dated
Slack reserve	5	protected for the unplanned

Rule of thumb. Compute real capacity by subtracting on-call, meetings, leave, new-hire ramp, KTLO, and a ~15–20% slack reserve from raw person-weeks — then plan to that number, not to headcount. A quarter plan with a committed list, a stretch list, and an explicit dated no-list is a plan; a list of everything everyone wants is a wish.

Worked example — sequencing a dependency chain

Detailed explanation. A roadmap that ignores dependencies looks fully loaded and delivers nothing on time, because half the work is blocked on the other half. Interviewers probe this with "walk me through how you'd sequence these." Walk through ordering a four-project chain where later work depends on earlier platform work.

Identify the edges. Which project needs an output of which other project?
Topologically order. Blockers first; parallelize what's independent.
Watch the critical path. The longest dependency chain sets the floor on the timeline, regardless of how many people you add.

Question. Given four projects with dependencies, produce a delivery sequence and identify the critical path.

Input.

Project	Depends on	Effort (pw)
A — event schema / contract	—	3
B — self-serve metrics layer	A	16
C — data-quality alerting	A	10
D — revenue dashboard	B	4

Code.

Dependency graph and sequence
=============================

   A (3) ──► B (16) ──► D (4)
      └────► C (10)

Topological order:  A first, then B and C in parallel, then D.

Critical path (longest chain):  A → B → D = 3 + 16 + 4 = 23 pw
  (C at 10pw runs alongside B; it is NOT on the critical path.)

Sequencing decisions:
  - A is the bottleneck-unlock: nothing starts until the schema
    contract lands. Staff it first, even though it's small.
  - B and C are independent after A — run them in parallel if
    capacity allows; if not, B first (D is blocked on it).
  - Adding people to C does NOT speed up delivery of D, because
    D waits on B. Throwing bodies at the wrong project is the
    classic sequencing mistake.

Step-by-step explanation.

Project A (the event schema/contract) is small at 3 person-weeks but blocks everything, so it must be staffed first despite its size. New managers often deprioritize small foundational work because it looks low-impact in isolation — sequencing shows why that's wrong.
Once A lands, B and C are independent and can run in parallel if capacity allows. Recognizing the parallelizable branches is how you compress the timeline without adding scope.
The critical path is A → B → D = 23 person-weeks. This is the floor on the timeline: even with infinite engineers you can't finish D before 23 weeks of sequential work complete, because each link waits on the previous.
The key insight interviewers want: adding people to C does nothing for D's delivery date, because D is blocked on B, not C. Understanding that capacity only helps on the critical path is the difference between effective and wasted staffing.
In practice this changes staffing: pour effort into A then B (the critical path), and treat C as fill-in work for whoever isn't needed on the path. The sequence, not the raw effort sum, drives the plan.

Output.

Step	Projects active	Cumulative weeks (critical path)
1	A	3
2	B (and C in parallel)	19
3	D	23

Rule of thumb. Sequence by dependency: staff the small blockers first, parallelize independent branches, and find the critical path — the longest chain sets the timeline floor. Adding people off the critical path speeds up nothing; that misallocation is the most common planning error interviewers test for.

Senior interview question on planning under constraints

A senior interviewer might ask: "You're three weeks into a quarter with a committed roadmap when two of your six engineers resign, and your VP still expects the revenue dashboard on the original date. Walk me through exactly what you do in the next 48 hours and the next two weeks — the re-plan, what you cut, what you protect, and how you communicate it."

Solution Using a transparent re-rank that protects the SLA, cuts the tail, and surfaces the trade-off

# Re-plan after losing 2 of 6 engineers mid-quarter

## Next 48 hours — stop the bleeding, get the facts
1. Recompute capacity: 6 → 4 engineers, minus the same taxes,
   minus knowledge-transfer time from the two leaving.
   Real new-work capacity roughly HALVES (~25pw → ~12pw).
2. Protect keep-the-lights-on FIRST: on-call and pipeline SLAs
   are non-negotiable; a data team that stops delivering fresh
   data loses trust faster than one that ships a feature late.
3. Capture the leavers' knowledge: runbooks, ownership handoff,
   pair sessions before they go. Bus-factor is the acute risk.

## Next 2 weeks — re-rank and communicate
4. Re-run RICE against the new ~12pw capacity. The committed list
   no longer fits. Cut from the BOTTOM of the ranking, not the top.
5. Present the VP a CHOICE, not a no:
   "With 4 engineers I have ~12pw of new work this quarter.
    I can deliver the revenue dashboard on time OR the metrics
    layer, not both. The dashboard is 4pw and high-visibility;
    the metrics layer is our biggest long-term lever. Here's my
    recommendation and the trade-off — which do you want?"
6. Cut the tail explicitly and publish the revised plan:
   - KEEP: SLA/on-call, revenue dashboard (VP's call), metrics
     layer phase 1 only if it fits.
   - CUT to Q1: data-quality alerting, Airflow migration.
7. Do NOT solve it with heroics/overtime — that hides the problem,
   burns out the remaining 4, and risks a third resignation.

## Ongoing
8. Start backfill hiring immediately; set expectations that new
   hires are net-negative capacity this quarter.

Step-by-step trace.

Move	Rationale
Recompute capacity honestly	4 engineers ≠ 4/6 of output; KT time hurts more
Protect SLA/on-call first	trust dies faster on stale data than a late feature
Capture leaver knowledge	bus-factor is the acute risk
Re-rank, cut from the bottom	preserve the highest-value work
Give the VP a choice, not a no	surface the trade-off; let them own the priority
Refuse heroics	overtime hides the problem and risks a 3rd exit

The answer's spine is transparency over heroics: it recomputes capacity honestly, protects the non-negotiable SLA before feature work, and converts the VP's impossible expectation into an explicit either/or the VP gets to decide. It cuts from the bottom of a re-ranked list so the highest-value work survives, and it explicitly rejects the tempting-but-fatal move of covering the gap with overtime.

Output:

Item	Decision
On-call / pipeline SLA	protected (non-negotiable)
Revenue dashboard (VP ask)	VP chooses; trade-off surfaced
Metrics layer phase 1	only if it fits ~12pw
Data-quality alerting	cut to Q1
Airflow migration	cut to Q1
Team overtime	explicitly avoided

Why this works — concept by concept:

Honest capacity recomputation — losing 2 of 6 doesn't cut output by a third; knowledge-transfer time and lost redundancy make the real hit closer to half. Managers who do this math avoid re-committing to an impossible plan.
Protect the SLA first — a data team's trust is built on reliable, fresh data; letting the pipelines rot to chase a feature is the wrong trade. Keep-the-lights-on comes before new work under constraint.
Re-rank and cut from the bottom — reprioritizing against the new capacity and cutting the lowest-value items preserves the most valuable work, rather than slicing a little off everything (which delivers nothing well).
Give the VP a choice, not a no — surfacing the explicit trade-off ("dashboard OR metrics layer, your call") respects their authority over priority while being honest about capacity. It moves the decision to the right owner and builds trust.
Cost — the honest path costs you a hard conversation with a VP and a publicly reduced roadmap. The heroics path "costs" nothing visible now but risks burnout, a third resignation, and a bigger miss later. Transparency is cheaper than heroics once you price in the compounding risk.

Optimization
Topic — optimization
Optimization problems that sharpen prioritization instinct

Practice →

Design Topic — design Design problems for sequencing dependent systems

Practice →

4. Platform vs product: the central data-team trade-off

Platform buys leverage, product buys impact — the interview tests whether you can sequence them, not pick one forever

The mental model in one line: platform vs product is the defining tension of data-team leadership — platform work (shared pipelines, reusable frameworks, self-serve tooling) buys leverage that compounds across many teams over time, while product work (customer-facing data features, dashboards, the thing the business asked for) buys impact that shows up this quarter — and the senior answer is never "always platform" or "always product" but a defensible judgment about when each pays, backed by a build-vs-buy discipline and a tech-debt budget. Interviewers probe this because it's where data managers most often go wrong: over-investing in a gold-plated platform nobody adopts, or drowning in one-off product asks until the team can't move.

When platform investment pays.

The rule of three. When the same pain shows up across three or more teams or projects, a shared solution starts to earn its keep. Solving it once for one team is product; solving it once for everyone is platform.
Leverage compounds. A self-serve pipeline framework that saves each of ten teams two weeks a quarter returns far more than the feature you'd have shipped instead — but only if it's actually adopted.
The adoption trap. Platform value is zero until people use it. A beautiful internal tool with no users is worse than no tool — it cost the quarter and delivered nothing. Treat internal platforms like products: find users, solve their real pain, measure adoption.

When product wins.

Impact is now, and now sometimes matters most. A revenue-driving dashboard or a launch-blocking data feature can be the right call even when platform work is "more strategic," because credibility and business trust are earned on delivery.
Premature platforming is a classic failure. Building the general framework before you understand the specific problem produces the wrong abstraction. Ship the specific thing two or three times, then extract the platform once the pattern is clear.

Build vs buy — the discipline underneath.

Buy the undifferentiated; build the differentiating. If a vendor solves it well and it isn't your competitive edge (orchestration, CDC, BI), buy or adopt open-source. Build only what's genuinely specific to you.
Total cost of ownership, not sticker price. Build has a large hidden maintenance tail; buy has recurring cost and lock-in. Score both on cost, time-to-value, fit, and long-run maintenance.

Tech-debt budgeting and platform ROI.

Budget debt paydown explicitly. Reserve ~15–20% of capacity for reliability, refactoring, and debt, continuously — not "we'll fix it later," which means never.
Measure platform ROI. Adoption (teams/pipelines using it), time saved per team, and reliability delta. If you can't measure it, you can't defend it — and a VP will ask.

Common interview probes on platform vs product.

"How do you decide between building a platform and shipping the feature?" — rule of three, adoption, sequence.
"Justify a platform investment to a product-focused VP." — frame in their currency: throughput, time saved, risk reduced.
"When have you built the wrong abstraction?" — premature platforming story with the lesson.
"How much of your team's time goes to tech debt?" — a deliberate budget, not zero and not ad hoc.

Worked example — the platform-vs-product decision matrix

Detailed explanation. The strong answer to "how do you decide?" is a repeatable rubric, not a preference. A simple decision matrix scores a candidate initiative on the factors that actually predict whether platform investment pays. Walk through scoring three candidate initiatives.

Repetition. How many teams/projects hit this same pain?
Compounding. Does solving it once keep paying, or is it one-and-done?
Adoption certainty. Will people actually use it, or is it "build it and hope"?
Opportunity cost. What product impact are we giving up to build it now?

Question. Score three initiatives on the platform-vs-product matrix and decide platform-now, product-now, or wait.

Input.

Initiative	Teams hit	Compounding?	Adoption certainty	Opportunity cost
Shared ingestion framework	6	high	high (teams asking)	medium
One team's custom export	1	none	n/a	low
Generic ML feature store	2	medium	low (speculative)	high

Code.

Platform-vs-product decision matrix
===================================

Score each factor 1-3; platform-now needs repetition AND
compounding AND adoption certainty to clear the bar.

                       Repeat  Compound  Adoption  OppCost   Verdict
Shared ingestion fwk     3        3         3        2      PLATFORM NOW
  → 6 teams, compounds, teams are literally asking. Build it.

One team's custom export  1        1         -        3      PRODUCT NOW
  → single team, no reuse. Just ship it as a feature; don't
    "platformize" a one-off.

Generic ML feature store  2        2         1        1      WAIT
  → only 2 teams, adoption speculative, high opp cost. Ship
    the specific feature 2-3x first, THEN extract the platform
    if the pattern holds. Premature = wrong abstraction.

Step-by-step explanation.

The shared ingestion framework clears every gate: six teams hit the same pain, the solution compounds, and teams are already asking (adoption is near-certain). This is the textbook platform-now case — you're extracting a pattern that's already proven across the org.
The one-team custom export scores the opposite: single team, no reuse, one-and-done. The right call is to ship it as a plain feature and resist the urge to "platformize" it — generalizing a one-off produces cost with no leverage.
The ML feature store is the dangerous middle: it sounds strategic, but only two teams need it and adoption is speculative. The matrix says wait — ship the specific feature two or three times first, then extract the platform once the real pattern is clear. This is the guard against premature abstraction.
Adoption certainty is the factor that most often gets ignored and most often kills platforms. A high-compounding idea with low adoption certainty is a bet, not a sure thing; treat it as one and de-risk it before committing a quarter.
The matrix turns "platform vs product" from a philosophy debate into a per-initiative decision you can defend. In an interview, walking a specific initiative through these gates is far stronger than asserting "I balance both."

Output.

Initiative	Verdict	Why
Shared ingestion framework	Platform now	repeats, compounds, adoption certain
One team's custom export	Product now	one-off, no reuse
Generic ML feature store	Wait	speculative adoption, high opp cost

Rule of thumb. Decide platform-vs-product per initiative on repetition, compounding, and adoption certainty — platform-now requires all three, a genuine one-off ships as product, and a speculative "strategic platform" waits until you've shipped the specific version enough times to know the right abstraction. Premature platforming is the most expensive mistake a data manager can make.

Worked example — a build-vs-buy scorecard

Detailed explanation. "Build or buy?" is a near-guaranteed probe, and the strong answer scores both on total cost of ownership, not the sticker price. Walk through a scorecard for a CDC/ingestion capability the team needs.

Score dimensions. Upfront cost, time-to-value, fit to your needs, long-run maintenance, and strategic control.
The hidden tail. Build's real cost is the years of maintenance, on-call, and feature-chasing after v1 ships.
The heuristic. Buy/adopt the undifferentiated heavy-lifting; build only the genuinely differentiating.

Question. Score build vs buy vs adopt-open-source for a CDC ingestion capability and make the call.

Input.

Dimension	Build in-house	Buy managed SaaS	Adopt open-source
Time to value	slow (months)	fast (weeks)	medium
Upfront cost	high (eng time)	low	medium
Ongoing cost	high (maintenance)	recurring $$	medium (self-host)
Fit to needs	perfect	good	good
Maintenance burden	all on us	vendor	shared/community
Strategic control	full	low (lock-in)	high

Code.

Build-vs-buy scorecard (CDC ingestion) — score 1-5, higher=better
=================================================================

Dimension          Build   Buy(SaaS)  Adopt(OSS)
------------------------------------------------
Time to value        2         5          4
Upfront cost         2         5          3
Ongoing cost (TCO)   2         3          4
Fit to needs         5         4          4
Maintenance burden   1         5          3
Strategic control    5         2          5
------------------------------------------------
TOTAL               17        24         23

Decision: BUY (managed) or ADOPT (OSS) — NOT build.
  - CDC ingestion is undifferentiated heavy-lifting; it is not our
    competitive edge. Building it means owning a maintenance tail
    for years to reinvent a solved problem.
  - Choose managed SaaS if speed matters most and budget allows.
  - Choose OSS (e.g. a Debezium-based stack) if strategic control
    and TCO matter more than time-to-value.
  - Build ONLY the thin layer that IS differentiating (our
    domain-specific transforms), on top of the bought/adopted core.

Step-by-step explanation.

Scoring both options on total cost of ownership — not the license price — flips the naive intuition that "building is free because we have engineers." Build scores lowest (17) precisely because of the maintenance and time-to-value penalties that don't show up on a spreadsheet but dominate the real cost.
Buy (SaaS) wins on speed and maintenance but is docked on strategic control and lock-in; OSS nearly ties it by trading a bit of time-to-value for control and better long-run TCO. The scorecard makes the trade-off explicit rather than a matter of taste.
The core heuristic is the tiebreaker: CDC ingestion is undifferentiated heavy-lifting — a solved problem — so building it reinvents a wheel and saddles the team with a multi-year maintenance tail for no competitive gain.
The nuanced senior move is the last line: build only the thin differentiating layer (your domain-specific transforms) on top of a bought or adopted core. This captures the "build the edge, buy the commodity" principle rather than treating it as all-or-nothing.
The choice between SaaS and OSS then comes down to what the org values more — speed and low maintenance (SaaS) or control and TCO (OSS) — which is a conversation you can have concretely because the scorecard laid out the axes.

Output.

Option	Total	When to choose
Buy (managed SaaS)	24	speed + low maintenance priority
Adopt (OSS)	23	control + TCO priority
Build in-house	17	only the differentiating thin layer

Rule of thumb. Score build vs buy on total cost of ownership — including build's multi-year maintenance tail — and default to buying or adopting the undifferentiated heavy-lifting while building only the thin layer that's genuinely your competitive edge. "We'll build it, we have engineers" ignores the maintenance tail that sinks the real cost.

Worked example — tech-debt budgeting and measuring platform ROI

Detailed explanation. Two probes cluster here: "how much time goes to tech debt?" and "how do you prove the platform was worth it?" The strong answers are a deliberate, standing debt budget and a small set of ROI metrics. Walk through both.

The debt budget. A fixed, continuous slice of capacity (~15–20%) for reliability, refactoring, and paydown — protected, not raided when deadlines loom.
Platform ROI metrics. Adoption (teams/pipelines using it), time saved per team, and reliability delta (incidents, freshness). Measure them or you can't defend the investment.

Question. Define a tech-debt budget policy and the ROI metric set for the shared ingestion framework.

Input.

Lever	Policy / metric
Debt budget	20% of capacity, standing, protected
Adoption metric	# teams / pipelines migrated onto the framework
Efficiency metric	eng-days saved per new pipeline vs before
Reliability metric	pipeline incidents + freshness-SLA breaches
Review cadence	quarterly ROI review

Code.

Tech-debt budget + platform ROI dashboard
=========================================

Tech-debt policy:
  - Reserve 20% of every sprint for reliability + debt paydown.
  - It is PROTECTED: not the first thing cut when a deadline slips.
    (Raiding the debt budget is borrowing at high interest.)
  - Track debt as a visible backlog, ranked, with a paydown rate.

Platform ROI (shared ingestion framework) — quarterly:
  Adoption:     8 / 10 teams migrated   (target 10)   ↑ from 3
  Efficiency:   new pipeline now 2 eng-days vs 9 before  (−78%)
  Reliability:  pipeline incidents 12/qtr → 4/qtr        (−67%)
  Freshness:    SLA breaches 15/mo → 3/mo                (−80%)

  ROI framing for a VP (their currency, not ours):
    "The framework let us ship 40 new pipelines this quarter with
     the same headcount that shipped 9 last year, and cut data
     incidents two-thirds. That's throughput and reliability, not
     an internal science project."

Step-by-step explanation.

A standing 20% debt budget beats "we'll fix it after the deadline," which is how debt compounds until the team grinds to a halt. Making it a fixed, recurring reserve normalizes paydown as part of the work rather than an apology.
The protection clause is the whole point: the debt budget must not be the first casualty when a deadline slips. Raiding it is borrowing at high interest — you get a little speed now and pay it back with reliability incidents later.
Tracking debt as a visible, ranked backlog with a paydown rate makes it manageable and defensible; "we have a lot of tech debt" is a vibe, "we have 40 ranked items and pay down 6 a quarter" is a plan.
The ROI metrics answer the VP's real question — was the platform worth it? Adoption (8/10 teams), efficiency (2 vs 9 eng-days per pipeline), and reliability (incidents down 67%) are concrete and hard to argue with. Without them, the platform is faith-based.
The framing line translates engineering wins into the VP's currency — throughput and reliability, not "internal tooling." This is the skill that gets platform investment funded: speaking impact, not architecture.

Output.

Metric	Before	After	Delta
Teams on framework	3	8	+5
Eng-days per new pipeline	9	2	−78%
Pipeline incidents / qtr	12	4	−67%
Freshness SLA breaches / mo	15	3	−80%

Rule of thumb. Run a standing, protected ~20% tech-debt budget and measure platform ROI in adoption, time saved, and reliability delta — then translate those into the VP's currency (throughput, reliability, risk). A platform you can't measure is a platform you can't defend, and a debt budget you raid under pressure is one you don't really have.

Senior interview question on platform vs product

A senior interviewer might ask: "Your product-focused VP wants the whole team on customer-facing data features next quarter. You believe a shared ingestion platform is the higher-leverage investment because six teams keep rebuilding the same brittle pipelines. Convince me — as that VP — to give you a quarter for platform work instead of features."

Solution Using a business-currency case, a de-risked scope, and a measurable ROI commitment

Pitching a platform investment to a product-focused VP
======================================================

Frame in THEIR currency (throughput, revenue, risk — not architecture):
  "Right now six teams each rebuild the same ingestion pipeline.
   We spend ~9 engineer-days per new pipeline and we had 12 data
   incidents last quarter — several delayed the very product
   launches you care about. This isn't an internal science
   project; it's the tax slowing down every feature we ship."

Quantify the return:
  "A shared ingestion framework cuts a new pipeline from ~9 days
   to ~2, and should cut data incidents by more than half. Across
   six teams that's roughly [X] engineer-weeks a quarter back into
   product work — permanently, compounding every quarter after."

De-risk the ask (don't ask for a blank quarter):
  "I'm not asking to disappear for a quarter. I'll take 2 engineers
   for 6 weeks on a phase-1 framework that migrates the 2 teams in
   the most pain, while the rest of the team keeps shipping your
   features. We prove adoption and the time-saved number before we
   scale it."

Commit to a measurable checkpoint:
  "At 6 weeks I'll show you: teams migrated, eng-days saved per
   pipeline, and incident delta. If the numbers aren't there, we
   stop and put everyone back on features. You hold me to that."

Acknowledge the trade-off honestly:
  "The cost is 2 engineers off features for 6 weeks — one feature
   slips a sprint. The return is faster feature delivery for every
   quarter after. I think that trade is worth it; here's the data."

Step-by-step trace.

Move	Why it lands with a product VP
Frame in throughput/risk, not architecture	speaks their language, not yours
Quantify time saved + incidents avoided	makes leverage concrete and credible
De-risk: phase-1, 2 engineers, 6 weeks	not a blank-cheque quarter
Measurable checkpoint with a kill switch	shows accountability, lowers their risk
Name the trade-off honestly	builds trust, not a hard sell

The pitch never argues architecture with a VP who doesn't care about it. It reframes the platform as a tax on the product velocity the VP wants, quantifies the return in their currency, de-risks the ask to a small phased bet with a kill switch, and owns the trade-off. It gives the VP an easy, low-risk yes instead of an all-or-nothing philosophical fight.

Output:

Element	The VP hears
Currency framing	"this helps my features ship faster"
Quantified ROI	"the leverage is real and measured"
Phased, de-risked scope	"low downside, I can say yes"
Checkpoint + kill switch	"I stay in control"
Honest trade-off	"she's leveling with me"

Why this works — concept by concept:

Business-currency framing — a product VP funds throughput, revenue, and risk reduction, not "good architecture." Translating the platform into faster feature delivery makes it their win, not just yours.
Quantified, compounding ROI — concrete numbers (9→2 eng-days, incidents halved, across six teams, every quarter) turn a leverage argument from hand-waving into a defensible business case.
De-risked phased ask — requesting 2 engineers for 6 weeks on a phase-1 that targets the worst pain is a small bet, not a blank quarter. Small asks with proof-of-value get approved; grand ones get deferred.
Measurable checkpoint with a kill switch — committing to show adoption and time-saved at 6 weeks, and to stop if the numbers aren't there, transfers risk off the VP and demonstrates the accountability a manager is supposed to have.
Cost — the honest price is one feature slipping a sprint and 2 engineers off product for 6 weeks; the return is compounding velocity for every quarter after. Framed as a phased, measured bet, the expected value is strongly positive and the downside is capped — which is exactly how you win a resource argument.

Design
Topic — design
Design problems for platform-vs-product architecture calls

Practice →

ETL Topic — etl ETL problems on the reusable pipeline patterns platforms extract

Practice →

5. Cross-functional influence, metrics, and the behavioral loop

You get things done through influence and evidence — the behavioral loop tests both, and the failure story is the pivot

The mental model in one line: the last track is cross-functional leadership — managing up to your leadership, sideways to product and analytics, and outward to the rest of the company — measured through the behavioral loop, where every answer is really a request for evidence that you can influence without authority, run a data team by the right metrics, lead an incident calmly, and learn from failure. Interviewers weight this heavily for team leadership because a data manager who can't influence stakeholders or represent the org will have a technically excellent team whose work nobody trusts or uses.

Managing up and across.

Managing up is a skill, not sycophancy. Give your leadership no-surprises visibility, bring problems with proposed options (not just problems), and align the team's work to what they're accountable for.
Influence without authority. You rarely command peers in product or analytics; you persuade with shared goals, data, and reliability. Being the team that ships and keeps its promises is the deepest source of influence.
Represent the team outward. Shield the team from chaos, translate business needs into technical work and technical constraints into business language, and take the blame publicly while giving credit publicly.

Metrics for a data team.

Reliability and freshness first. Data teams live or die on trust: pipeline uptime, data-freshness SLOs/SLAs, and data-quality (incidents, failed checks) are the core.
DORA-style delivery metrics, adapted. Deployment frequency, lead time for change, change-failure rate, and mean-time-to-restore translate well to data engineering and signal delivery health without becoming a stack-ranking weapon.
Metrics inform, they don't rank. Use them to find systemic problems, not to grade individuals — the moment a metric becomes a target for individual performance, it gets gamed.

Incident leadership.

Calm, roles, comms. As incident commander you assign roles, keep a running timeline, communicate status to stakeholders on a cadence, and protect the responders from the peanut gallery.
Blameless postmortems. The goal is the systemic fix, not a culprit. "What in the system let this happen, and what did we change?" is the only useful question.

The behavioral loop and STAR.

STAR keeps you concrete. Situation, Task, Action, Result — with the Result quantified. Rambling context with no outcome is the most common failure.
The failure question is the pivot. "Tell me about a project that failed" tests ownership and learning. The strong answer owns it plainly, shows what you personally did, and — the key part — names the system change you made so it can't recur.

Common interview probes on cross-functional leadership.

"Tell me about a time you influenced a decision without authority." — shared goals + data.
"How do you measure a data team's health?" — reliability/freshness SLOs + adapted DORA, not lines of code.
"Walk me through an incident you led." — commander role, comms cadence, blameless postmortem.
"Tell me about a project that failed and what you changed." — ownership + system change.

Worked example — a data-team metrics and SLO table with a freshness query

Detailed explanation. "How do you measure your team?" is a standard probe, and the strong answer is a small, balanced scorecard that leads with reliability and adapts DORA — plus the willingness to actually instrument it. Walk through the scorecard and a real freshness-SLO query.

Reliability tier. Pipeline uptime, data-freshness SLO, data-quality incidents.
Delivery tier (adapted DORA). Deploy frequency, lead time, change-failure rate, MTTR.
Guardrail. These are team/system health signals, never individual stack-ranking.

Question. Define the data-team scorecard and write the SQL that measures the freshness SLO for a critical table.

Input.

Metric	Tier	Target
Pipeline uptime	reliability	≥ 99.5%
Freshness SLO (critical tables < 1h old)	reliability	≥ 99%
Data-quality incidents	reliability	≤ 3 / quarter
Change-failure rate	delivery (DORA)	≤ 15%
MTTR (pipeline)	delivery (DORA)	< 1h

Code.

-- Freshness SLO: % of hourly windows where the critical table
-- was updated within its 1-hour freshness target, last 30 days.
WITH loads AS (
    SELECT
        date_trunc('hour', loaded_at) AS load_hour,
        max(loaded_at)                AS last_load
    FROM   pipeline_audit.load_log
    WHERE  table_name = 'analytics.fct_orders'
      AND  loaded_at >= now() - INTERVAL '30 days'
    GROUP  BY 1
),
windows AS (
    SELECT
        load_hour,
        -- "fresh" if the load landed within 60 min of the hour boundary
        (last_load <= load_hour + INTERVAL '60 minutes') AS met_sla
    FROM   loads
)
SELECT
    count(*)                                             AS total_windows,
    count(*) FILTER (WHERE met_sla)                      AS windows_met,
    round(100.0 * count(*) FILTER (WHERE met_sla)
                 / nullif(count(*), 0), 2)               AS freshness_slo_pct
FROM   windows;
-- → freshness_slo_pct = 99.4  (meets the >= 99% target)

Step-by-step explanation.

The scorecard deliberately leads with the reliability tier — uptime, freshness, quality — because a data team's entire value proposition is trustworthy, timely data. A team with dazzling delivery velocity but stale, wrong data has failed at the actual job.
The delivery tier adapts DORA to data engineering: deploy frequency and lead time show how fast the team ships changes, change-failure rate and MTTR show how safely. These translate cleanly from software delivery and signal health without measuring "output" in a gameable way.
The freshness query operationalizes the SLO: it buckets loads by hour, marks each window as meeting or missing the 60-minute target, and computes the percentage. This is the difference between claiming an SLO and measuring one — interviewers notice when you can write it.
The guardrail matters as much as the metrics: stating that these are team/system signals and never individual stack-ranking pre-empts the "metrics-driven micromanager" fear. The moment freshness becomes an individual's performance target, someone games the load_log.
The balanced set resists both extremes — no vanity metrics (lines of code, tickets closed) and no single number that can be gamed. It's a small dashboard that tells you where the system is unhealthy.

Output.

Metric	Reading	Status
Pipeline uptime	99.6%	meets ≥ 99.5%
Freshness SLO	99.4%	meets ≥ 99%
Data-quality incidents	2 this qtr	meets ≤ 3
Change-failure rate	11%	meets ≤ 15%
MTTR	42 min	meets < 1h

Rule of thumb. Measure a data team with a small balanced scorecard — reliability (uptime, freshness SLO, quality) first, adapted DORA (deploy frequency, lead time, change-failure rate, MTTR) second — instrumented for real, and use it to find systemic problems, never to stack-rank individuals. If you can write the freshness query, you sound like a manager who's actually run the dashboard.

Worked example — a STAR story-bank template

Detailed explanation. The behavioral loop is won or lost on preparation: strong candidates keep a bank of 8–12 stories mapped to the themes interviewers probe, each pre-structured as STAR with a quantified result. Walk through building the bank and one fully-worked entry.

Cover the themes. Conflict, failure, influence-without-authority, hard people call, ambiguity, a delivery win, a hard prioritization call.
Reusable stories. A good story often answers several prompts; a bank of 8–12 covers most loops.
Quantify the Result. A number or a concrete outcome, plus what you learned.

Question. Design a STAR story-bank template and fill one entry for a "hard prioritization call" story.

Input.

Theme	Story slot
Conflict	two engineers feuding
Failure	a project that missed
Influence w/o authority	platform pitch to a VP
Hard people call	managed out a toxic senior
Ambiguity	undefined data project
Prioritization	cut a roadmap under headcount loss

Code.

# STAR story bank

## Slot: Hard prioritization call
**Prompts it answers:** "prioritize under pressure", "say no to a
stakeholder", "make a tough trade-off", "lose headcount".

- **Situation:** 3 weeks into Q3, lost 2 of 6 engineers; VP still
  expected the revenue dashboard on the original date.
- **Task:** Re-plan to ~half the new-work capacity without breaking
  the pipeline SLA or burning out the remaining team.
- **Action:** Recomputed real capacity, protected on-call/SLA first,
  re-ran RICE, and gave the VP an explicit either/or (dashboard OR
  metrics layer) instead of quiet heroics. Cut two items to Q4,
  captured the leavers' runbooks, refused overtime.
- **Result:** Shipped the VP's dashboard on time and the SLA held
  at 99.5%; zero further attrition; the VP later cited the
  transparent trade-off as why they trusted the team's estimates.
- **Learned:** Surfacing the trade-off early beats absorbing it
  silently — I now publish a capacity number every planning cycle.

## Slot: Failure  → (see next worked example)
## Slot: Conflict → (toxic-senior story from section 2)
## Slot: Influence → (platform pitch from section 4)

Step-by-step explanation.

Organizing by theme rather than by story means you walk into the loop with a lookup table: any prompt maps to a prepared slot. This beats improvising, which is where candidates ramble or freeze.
Listing the prompts each story answers ("prioritize under pressure", "say no", "lose headcount") makes stories reusable — one strong prioritization story covers four common questions, so a bank of 8–12 genuinely covers most loops.
The STAR structure keeps the answer tight: a one-line Situation and Task, a specific Action (the actual decisions you made), and a Result. The most common behavioral failure is 90 seconds of Situation and no Result.
Quantifying the Result (dashboard on time, SLA held at 99.5%, zero attrition) turns a claim into evidence, and adding what you learned shows growth — interviewers explicitly probe for the learning.
Cross-referencing slots to other real situations (the toxic-senior conflict story, the platform pitch) shows the bank is a system: the same well-run career furnishes evidence across every theme, which is exactly the coherence a panel is looking for.

Output.

STAR element	Weak version	Strong version
Situation/Task	long, meandering	two crisp sentences
Action	"we decided"	"I did X, Y, Z"
Result	"it went well"	quantified + what changed
Learning	omitted	explicit system change

Rule of thumb. Walk into the loop with a bank of 8–12 STAR stories mapped to themes (conflict, failure, influence, hard people call, ambiguity, prioritization), each with a quantified Result and a lesson. Keep Situation short and Action first-person; the number in the Result is what makes it evidence instead of an assertion.

Worked example — leading an incident as commander

Detailed explanation. "Walk me through an incident you led" tests whether you can create calm and structure under pressure. The strong answer shows the incident-commander mechanics and a blameless follow-up. Walk through the structure with a data-freshness outage.

Declare and assign roles. Commander (you, coordinating — not fixing), a lead investigator, a comms owner, a scribe.
Communicate on a cadence. Regular stakeholder updates even when there's nothing new, so the org isn't guessing.
Blameless postmortem. Timeline, contributing factors, and system-level action items with owners.

Question. Structure your handling of a Sev-2 where the revenue pipeline is 6 hours stale during business hours.

Input.

Role	Owner	Job
Incident commander	You	coordinate, decide, shield the team
Investigator	On-call DE	find and fix root cause
Comms	You / delegate	update stakeholders on cadence
Scribe	Anyone free	keep the timeline

Code.

Incident command — revenue pipeline 6h stale (Sev-2)
====================================================

T+0   Declare Sev-2. Open a channel. Assign roles:
        me = commander (coordinate, do NOT dive into the fix),
        on-call DE = investigator, teammate = scribe.
T+5   First stakeholder comms: "Revenue data delayed, investigating,
        next update in 30 min." Set the cadence up front.
T+15  Investigator finds an upstream schema change broke ingestion.
        Commander decision: roll forward a fix vs backfill? → hotfix
        the parser now, backfill after.
T+30  Comms update on schedule (even though "still working").
T+50  Fix deployed; backfill running. Comms: "Pipeline restored,
        backfilling the 6h gap, ETA 40 min."
T+90  Backfill complete, data verified fresh. Declare resolved.
        Thank the team publicly.

Next day — BLAMELESS postmortem:
  - Timeline (from the scribe's notes).
  - Contributing factors: upstream changed schema without notice;
    we had no schema-contract check to catch it.
  - System action items (with owners + dates):
    1. Add a data-contract/schema-validation gate on ingestion.
    2. Establish a change-notification SLA with the upstream team.
    3. Add a freshness alert at 1h, not 6h.
  - No blame on the on-call engineer. The SYSTEM let it happen.

Step-by-step explanation.

The first move is declaring severity and assigning roles — including the counterintuitive one where the commander coordinates and explicitly does not dive into the fix. A manager who grabs the keyboard loses the coordination the incident actually needs.
Setting a comms cadence at T+5 ("next update in 30 min") and holding it even when there's no news is what keeps stakeholders calm and stops the flood of "any update?" pings that distract the responders. Predictable comms is the leadership.
The commander makes the call that requires a decision-maker (hotfix-now vs backfill-first) so the investigator can execute without debating trade-offs mid-fire. Clear decision ownership speeds resolution.
Public thanks at resolution and shielding the on-call engineer set the culture: incidents are team events, not individual failures. This is what makes people willing to be on-call and to surface problems early.
The blameless postmortem is where the real value is: it converts one outage into permanent system improvements (a schema-contract gate, an upstream change SLA, an earlier freshness alert). "What in the system let this happen?" — not "who broke it?" — is the only question that prevents recurrence.

Output.

Phase	Outcome
Detection → declaration	roles assigned, calm established
During	cadence comms, clear decisions
Resolution	data restored + verified, team thanked
Postmortem	3 system fixes, zero blame

Rule of thumb. Lead incidents as a commander who coordinates rather than codes: assign roles, communicate on a fixed cadence, own the decisions only you can make, and run a blameless postmortem that ships system-level fixes. The signal interviewers want is calm-plus-structure and "what did the system need," not heroics and not a culprit.

Senior interview question on the behavioral loop

A senior interviewer might ask: "Tell me about a project that failed — one you owned. Walk me through what happened, your role in the failure, how you handled it in the moment, and specifically what you changed afterward so it couldn't happen again."

Solution Using plain ownership, honest role, and a concrete system change

Failure story — the analytics migration that missed (STAR)
==========================================================

Situation:
  "We committed to migrating the analytics warehouse to a new
   platform in one quarter, with a hard date tied to a finance
   reporting cycle."

Task:
  "I owned the migration end-to-end as the team's manager — scope,
   plan, and delivery."

Action / what went wrong (own MY part plainly):
  "We missed by six weeks. My mistake was concrete: I let the team
   commit to a fixed date before we'd done discovery on data-quality
   in the legacy source. When we started migrating, we found years
   of undocumented edge cases and silent nulls. I'd optimized for a
   confident-sounding plan over an honest one. In the moment I
   escalated early — told finance at week 4, not week 11 — gave them
   a realistic revised date, and stood up a parallel-run so they
   kept their old reports until the new ones were verified."

Result:
  "Finance had their reports (on the old system) with no gap, the
   migration landed 6 weeks late but correct, and no bad numbers
   ever reached a finance report."

What I CHANGED (the pivot — system, not just apology):
  "Three permanent changes: (1) no fixed external date before a
   time-boxed discovery spike — we now estimate ranges and commit
   after discovery; (2) a data-quality assessment is a mandatory
   first phase of any migration; (3) parallel-run is now our default
   migration pattern. The next migration used all three and landed
   on time."

Step-by-step trace.

Element	What it demonstrates
Owns the miss plainly (6 weeks late)	no deflection, real accountability
Names MY specific mistake	self-awareness, not vague "we"
Early escalation (week 4, not 11)	handled it like a manager
Parallel-run protected the stakeholder	judgment under a bad situation
Three concrete system changes	learning that generalizes

The answer resists the two failure modes: the humblebrag fake-failure ("I work too hard") and the blame-shift ("the upstream team let us down"). It owns a real, consequential miss, names the specific decision that caused it, shows manager-grade handling in the moment (early escalation, protecting the stakeholder), and lands on concrete, permanent system changes that a later project actually used — which is the whole point of the question.

Output:

Dimension	This answer
Failure realness	genuine, consequential miss
Ownership	specific personal mistake named
In-the-moment handling	early escalation + parallel-run
System change	3 permanent process changes
Proof it worked	next migration landed on time

Why this works — concept by concept:

Plain ownership — naming a real six-week miss and your specific decision (committing to a date before discovery) signals the accountability the question exists to test. Fake failures and blame-shifting both fail it instantly.
Manager-grade handling — escalating at week 4 instead of week 11 and standing up a parallel-run shows you handled a bad situation with judgment: honest, early, and protective of the stakeholder rather than hopeful and silent.
The system change is the pivot — the answer's weight is on the three permanent changes (discovery spike before dates, mandatory data-quality phase, parallel-run default). Learning that changes the system is what separates a manager from someone who just apologizes.
Proof it generalized — "the next migration used all three and landed on time" closes the loop: the lesson wasn't a nice sentiment, it changed outcomes. That evidence is what makes the story land.
Cost — the honest path costs you the discomfort of describing a real failure to a stranger who's judging you. The payoff is credibility: an interviewer trusts a candidate who owns a genuine miss and shows growth far more than one with a suspiciously flawless record. Vulnerability, scoped well, is the strongest move in the behavioral loop.

SQL
Topic — sql
SQL problems for the freshness, SLO, and metrics queries you'll defend

Practice →

Streaming
Topic — streaming
Streaming problems on the real-time pipelines behind incident leadership

Practice →

Cheat sheet — EM interview recipes

The one-line frame. The data engineering manager interview grades judgment across four tracks — people, project/roadmap, technical/architecture judgment, and cross-functional — and every answer secretly asks "can your judgment scale past your own hands?" Describe impact as team outcomes, name the trade-off in every decision, and say "it depends, here's what I'd need to know" before committing.
IC → EM shift. Your output is the team's output; your feedback loop gets slower and noisier; you trade depth for breadth; your influence is indirect. Answer "how do you spend your week?" with numbers: build time drops from ~70% to under 20%, off the critical path, with hours flowing to people, planning, and cross-functional.
"Why management" answer. Lead with an energy source (other people's growth), evidence you already do the job informally, and an explicit list of trade-offs you've accepted. Never lead with title, money, or control — the three fastest disqualifiers.
First 90 days. Listen (30) → diagnose and co-write the operating model (60) → deliver one visible win and set the rhythm (90). For a peer-to-boss move, name the changed relationship out loud and give tenured engineers more autonomy, not less.
People loop. Hire → onboard → grow → evaluate → retain, with a mechanism for each: structured hiring with pre-committed ratings; a 30/60/90 ramp; a skill/will matrix (delegate / re-engage / coach / direct); continuous SBI feedback; and sponsorship for growth. "Open-door policy" is an IC answer.
1:1 doc. Their agenda first, in a shared running doc, one specific piece of feedback every time, owned action items with checkboxes, and a living growth plan that names the two-to-three gaps and the opportunities you're creating to close them.
Underperformer / performance conversation. Diagnose skill vs will vs fit vs context, then run it clear and kind: no-ambush open, SBI evidence, own your part, written time-boxed expectations with real support, and a genuine statement of belief. Be willing to lose even a top performer if the behaviour is toxic — culture is set by the worst behaviour you tolerate from your best people.
Prioritization. Use RICE (Reach × Impact × Confidence ÷ Effort) or an impact/effort 2×2 to make ranking defensible; keep the confidence factor honest; treat the score as an input to judgment and say why when you override it. Every roadmap needs a visible, dated no-list.
Capacity math. Headcount ≠ capacity. From raw person-weeks subtract on-call, meetings, leave, new-hire ramp (a net negative short-term), and a ~20% KTLO tax, then reserve ~15–20% slack. Plan to that number. Sequence by dependency and staff the critical path — adding people off it speeds up nothing.
Lose-headcount answer. Recompute capacity honestly (2 of 6 ≈ half the output), protect the SLA/on-call first, re-rank and cut from the bottom, give stakeholders an explicit either/or choice rather than a no, capture leavers' runbooks, and refuse heroics/overtime.
Platform vs product. Platform buys leverage, product buys impact; decide per initiative on repetition (rule of three), compounding, and adoption certainty. Ship the specific thing 2–3× before extracting a platform — premature platforming is the most expensive data-manager mistake. Buy the undifferentiated, build only your edge; run a standing, protected ~20% tech-debt budget; and measure platform ROI in adoption, time saved, and reliability delta.
Pitch platform to a product VP. Frame in their currency (throughput, revenue, risk, not architecture), quantify the compounding return, de-risk to a small phased bet with a measurable checkpoint and a kill switch, and name the trade-off honestly.
Cross-functional + behavioral. Manage up with no-surprises visibility and problems-with-options; measure the team with reliability (uptime, freshness SLO, quality) first and adapted DORA second, never to stack-rank; lead incidents as a coordinating commander with cadence comms and a blameless postmortem; and bring a bank of 8–12 STAR stories with quantified Results. For the failure question, own it plainly and land on the concrete system change you made.

Frequently asked questions

What does a data engineering manager actually do?

A data engineering manager is responsible for the output and health of a data team, not for personally writing the most pipelines — the job is leverage, turning a group of engineers into more than the sum of their commits. Concretely the role spans four areas: people (hiring, growing, evaluating, and retaining engineers), planning (building and prioritizing a data engineering roadmap against finite capacity), technical judgment (reviewing designs, making build-vs-buy and platform-vs-product calls without being the one who codes them), and cross-functional leadership (managing up to leadership, aligning with product and analytics, and representing the data org). Most managers keep a small slice of hands-on work to stay technically credible, but if their week still looks like an IC's, they haven't made the transition. The through-line is judgment: converting ambiguous business pressure into a sequence of good decisions other people execute.

How is the engineering manager interview different from the IC interview?

The IC loop tests whether you can do the work — coding, SQL, system design under a whiteboard clock; the engineering manager interview assumes you can and tests whether your judgment scales past your own hands. Instead of "write this query," you get "your team missed a deadline — what did you do," "you lost two engineers mid-quarter — what do you cut," and "justify a platform investment to a skeptical VP." The panel is split into tracks — people, project/roadmap, technical judgment, and cross-functional — and grades behavioral evidence (STAR stories with quantified results) rather than code correctness. The single biggest adjustment for strong ICs is to stop describing impact as personal deliverables and start describing it as team outcomes and decisions. "It depends, here's what I'd need to know" is a strong opening in an EM loop and a weak one in a coding round.

Do engineering managers still code?

It depends on the level and the team, but the honest answer for most first-line data engineering managers is "a little, deliberately, and off the critical path." Build time typically drops from roughly 70% as an IC to under 20% as a manager, and that remaining slice goes to design reviews, prototypes, glue code, and emergency backfill — never to owning a critical deliverable, because a manager on the critical path becomes a single point of failure who can't actually manage during a crisis. Some staff/principal-adjacent or very small-team managers code more; some senior managers and directors code essentially zero. In the interview, claim neither extreme: "I code enough to review architecture credibly and help in a pinch, but I plan as if my coding capacity is zero" is the credible, senior position. The goal of EM interview prep here is to sound like someone who has let go of being the top IC without going stale.

How do you answer people-conflict and underperformer questions?

Lead with a process, not a vibe, and ground it in a real story. For an underperformer: diagnose whether it's skill, will, fit, or context (each needs a different response), then have a conversation that is clear and kind — no-ambush opening, specific situation-behaviour-impact evidence, owning your part, written time-boxed expectations with genuine support, and a real statement of belief. For conflict between engineers: get past positions to the underlying interests, mediate directly, and set boundaries so a feud doesn't tax the whole team. The senior signal across both is a willingness to act decisively when things don't improve — including managing out a technically strong but toxic engineer, because people management means protecting the team's health over any single person's output. Avoidance stories and surprise-firing stories both fail; a story with a clear process and an honest outcome (turned around, or parted ways with dignity) lands.

Platform vs product — how do you decide?

There is no "always platform" or "always product" answer; the senior move is a per-initiative judgment. Invest in platform when the same pain repeats across three or more teams, the solution compounds over time, and adoption is genuinely likely (teams are asking) — that's leverage worth a quarter. Ship product when impact is needed now, when it's a one-off with no real reuse, or when you haven't yet shipped the specific thing enough times to know the right abstraction — building the general framework prematurely produces the wrong one, the single most expensive data-manager mistake. Underneath sits build-vs-buy discipline (buy the undifferentiated heavy-lifting, build only your competitive edge, score on total cost of ownership) and a standing tech-debt budget. And when you pitch platform to a product-focused leader, translate it into their currency — throughput, reliability, risk — and de-risk it to a small phased bet with a measurable checkpoint.

How do you prepare for the EM interview loop?

Treat EM interview prep as building evidence, not memorizing answers. First, build a bank of 8–12 STAR stories mapped to the themes every panel probes — conflict, failure, influence without authority, a hard people call, ambiguity, a delivery win, and a hard prioritization call — each with a quantified Result and a lesson. Second, prepare your frameworks so you can walk them live: the skill/will matrix, RICE plus capacity math, the platform-vs-product decision matrix, and a build-vs-buy scorecard. Third, know which track each interviewer is grading and lead with a story tuned to that signal instead of giving the same generic answer five times. Fourth, keep your technical judgment sharp — the architecture round tests whether you've gone stale, so practice reasoning about team leadership-adjacent design and trade-off problems rather than raw coding speed. Finally, rehearse the failure story until you can own a real miss plainly and land it on a concrete system change — that one question reveals more than almost any other.

Practice on PipeCode

Drill the design practice library → for the architecture-review, build-vs-buy, and platform-vs-product judgment the technical and skip-level rounds probe.
Keep the fundamentals sharp on the SQL practice library → so your technical credibility holds up — the freshness/SLO and data-quality queries a manager should still be able to reason about.
Rehearse the prioritization instinct on the optimization practice library → for the trade-off and constraint problems that mirror capacity-limited roadmap planning.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to keep the whole technical foundation current while you build the people, roadmap, and platform-vs-product muscles the interview grades.

Lead a data team with judgment you can defend

Frameworks explain the theory. PipeCode drills build the judgment — when to invest in platform vs product, how to reason about a design under constraints, and how to keep the technical credibility that a manager who's gone stale quietly loses. Pipecode.ai is Leetcode for Data Engineering — trade-off-first practice tuned for the decisions data engineering managers actually make.

Practice design problems →
Practice SQL problems →

Staff & Principal Data Engineer Interviews: Scope, Impact, Cross-Team Architecture Loops

Gowtham Potureddi — Sat, 01 Aug 2026 13:21:57 +0000

The staff data engineer interview is the first interview in a data engineer's career where writing correct code is table stakes and no longer the thing being graded — the interviewer already assumes you can model a warehouse, tune a query, and ship a pipeline, so the entire loop is engineered to measure something harder to fake: the scope and impact of the problems you choose, and whether you can move a decision across teams you do not manage. Where the senior loop asks "walk me through how you'd build this," the staff loop asks "three teams disagree on the schema and the deadline is in six weeks — what do you do?" The questions look similar on the surface, but they are testing the opposite muscle. The senior answer optimizes a system; the staff answer optimizes an organization's ability to build many systems.

That shift catches strong senior engineers off guard, because the very habits that earned the promotion to senior — diving deep, owning the hardest module, out-coding the room — are the habits that cap you at senior if you carry them into a staff loop unchanged. This guide is the walkthrough for the transition: it decodes the leveling rubric that separates senior from staff from principal data engineer, it shows how interviewers probe for cross-team architecture leverage and technical leadership rather than raw throughput, and it gives you the artifacts — the scope narrative, the RFC/ADR skeleton, the design-review script, the promo packet — that turn vague "impact" into a defensible, quantified story. Each section pairs a teaching block with a worked interview answer: a template, a step-by-step trace, an output table, then a concept-by-concept breakdown of why the answer lands.

When you want hands-on reps on the design and coding rounds that still gate the staff loop, drill the design practice library →, keep the fundamentals sharp on the SQL practice library →, and rehearse the platform muscle on the streaming practice library →.

On this page

Why the staff/principal bar is scope and impact, not more code
Demonstrating scope: from project to platform to org
Cross-team architecture loops and the RFC/design-review muscle
System design at staff/principal altitude: trade-offs and failure narratives
Influence without authority, promo packets, and the staff interview loop
Cheat sheet — staff and principal interview recipes
Frequently asked questions
Practice on PipeCode

1. Why the staff/principal bar is scope and impact, not more code

The leveling line is a change of unit — from "owns a system" to "owns a strategy"

The one-sentence invariant: the promotion from senior to staff is not "more of the same, done better" — it is a change in the unit of work you are measured in, from a system you own end-to-end to a problem space you make the whole org better at, and the staff data engineer interview is built to detect whether you already operate in the larger unit or merely aspire to. A senior engineer is graded on the systems they ship; a staff engineer is graded on the leverage they create so that other people ship better systems; a principal data engineer is graded on the technical strategy that decides which systems the org should build at all. Every question in a staff loop is a probe for which of those three units you naturally reach for when handed an ambiguous prompt.

This is why "I rewrote the ingestion service and cut latency 40%" — a strong senior answer — is a weak staff answer if that is where the story ends. The staff version is "I noticed six teams were each building their own ingestion service, so I drove an RFC for a shared platform, got three staff engineers to disagree-and-commit, and now those six teams ship ingestion in a day instead of a quarter." Same person, same technical depth; the difference is the sphere of influence the story operates in.

The sphere-of-influence ladder — what each level's blast radius looks like.

Senior (self + immediate team). Owns a system or a large component end-to-end. Makes the right call under ambiguity within the team's scope. Mentors juniors. The blast radius of a great senior engineer is their team's roadmap.
Staff (multiple teams / a platform). Owns a problem across team boundaries. Identifies leverage — the shared platform, the standard, the migration — that makes many teams faster. Drives alignment without managing anyone. The blast radius is an org's ability to execute.
Principal (org / company). Owns technical strategy — which bets the org makes, which architectures it standardizes on for the next three years, which it sunsets. Influences the roadmap of roadmaps. The blast radius is the company's technical direction.
The key discontinuity. Between senior and staff, the authority does not grow — a staff engineer still manages no one — but the scope does. That gap is the entire skill: producing org-level outcomes with team-level authority. Interviewers hunt for evidence you have closed it.

What interviewers actually probe in a staff loop.

Scope selection. Given a vague prompt, do you scope up (find the leveraged version of the problem) or scope down (retreat to the comfortable component)? Staff engineers instinctively ask "who else has this problem?"
Ambiguity tolerance. Senior questions have a right answer; staff questions have trade-offs and no clean answer. The signal is whether you can make a defensible decision and name what you're trading away rather than freeze waiting for more requirements.
Cross-team leverage. Do your stories move other teams, or just your own deliverables? The word "I" versus "we drove" versus "the org adopted" is diagnostic.
Judgment and taste. Do you know when not to build, when to buy, when to standardize versus let a thousand flowers bloom? Principal-level judgment is knowing which fights are worth having.

The four-signal grading model interviewers score you against.

Signal 1 — Scope. Was the problem org-sized or task-sized? Did you expand a narrow ask into the real leveraged problem?
Signal 2 — Impact. Was the outcome a multiplier (others got faster) or additive (you personally shipped)? Is it quantified?
Signal 3 — Leadership. Did you drive alignment, mentor, write the doc the org rallied around — or did you just execute a plan someone handed you?
Signal 4 — Judgment. Under trade-offs, did you make a sound call and own its downside — or did you either freeze or hand-wave?

Worked example — the Senior → Staff → Principal leveling rubric

Detailed explanation. The single most useful artifact for a staff interview is a memorized leveling rubric that maps each behavioral dimension across the three levels. Every staff loop implicitly grades you against a rubric like this; having it in your head lets you aim your stories at the level you're interviewing for instead of telling a great senior story in a staff loop. Walk through building the rubric across the dimensions interviewers actually score.

Dimension: scope of ownership. From "a component" (senior) to "a platform / cross-team problem" (staff) to "a domain strategy" (principal).
Dimension: ambiguity. From "solves well-defined problems" to "defines the problem" to "decides which problems the org should solve."
Dimension: influence. From "influences the team" to "influences multiple teams without authority" to "influences the org / other staff+ engineers."
Dimension: what they're a force multiplier for. From "junior engineers" to "a whole team / adjacent teams" to "the engineering org's technical direction."

Question. Build the four-dimension leveling rubric for senior, staff, and principal data engineers, and state the one behavior that most distinguishes each boundary.

Input.

Dimension	Senior	Staff	Principal
Scope	one system / component	a platform / cross-team problem	a domain / technical strategy
Ambiguity	solves defined problems	defines the problem	picks which problems matter
Influence	the team	multiple teams, no authority	the org and other staff+
Force multiplier for	juniors	teams	technical direction

Code.

LEVELING RUBRIC — data engineering IC track
===========================================

SENIOR
  Unit of work ...... a system
  Signature move .... "I owned the hardest part and shipped it right."
  Grows ............. depth
  Boundary up ....... starts asking "who else has this problem?"

STAFF
  Unit of work ...... a problem space across teams
  Signature move .... "I drove an RFC 3 teams aligned on; they all got faster."
  Grows ............. leverage (authority stays flat, scope grows)
  Boundary up ....... starts shaping *which* problems the org invests in

PRINCIPAL
  Unit of work ...... a multi-year technical strategy
  Signature move .... "I set the architecture direction the org standardized on."
  Grows ............. judgment / taste at company scale
  Distinguishing .... says 'no' to good ideas to protect the few great ones

THE ONE-LINE TEST PER BOUNDARY
  Senior -> Staff:     do your outcomes require other teams to change?
  Staff  -> Principal: do you decide the roadmap of roadmaps?

Step-by-step explanation.

Read the rubric down a column first to internalize one coherent level, then read across a row to feel the boundary. The scope row is the spine: everything else (ambiguity, influence, force multiplication) is downstream of how big a unit you operate in.
The senior→staff boundary is the "who else has this problem?" reflex. A senior fixes their pipeline; a staff engineer notices five teams have the same broken pipeline and builds the shared fix. Same trigger, wider aperture.
The staff→principal boundary is problem selection at org scale — a principal decides the org should stop investing in a dying architecture and standardize on another, and says "no" to locally-good ideas that don't fit the strategy. Taste, not throughput.
Crucially, authority does not appear in the rubric as something that grows — a staff engineer still manages no one. The scope grows while the org-chart authority stays flat, which is exactly why influence-without-authority (section 5) is the load-bearing skill.
Aim your interview stories at the level you're targeting. Telling a beautifully-executed senior story (deep, solo, one-team) in a staff loop reads as "not yet ready" — not because the story is bad, but because it's evidence for the wrong level.

Output.

Boundary	The distinguishing behavior	Interview tell
Senior → Staff	outcomes require other teams to change	stories say "we drove alignment," not "I built"
Staff → Principal	you decide which problems the org invests in	stories include a deliberate "we chose not to"
Any level	quantified, owned trade-offs	names the downside of the chosen path

Rule of thumb. Before any staff loop, write your target level's rubric column on one line and test each of your stories against it. If a story doesn't touch a team other than your own, it's a senior story — keep it in reserve for the deep-dive round, and lead the behavioral rounds with the cross-team ones.

Worked example — decoding a job ladder / leveling.fyi-style rung

Detailed explanation. Job ladders and public leveling references (the leveling.fyi-style rungs, internal engineering ladders) are written in deliberately abstract language — "demonstrates broad technical influence," "operates with significant ambiguity" — and candidates who can translate that abstraction into concrete evidence they'll present interview far better than those who read it as boilerplate. Walk through decoding a representative staff rung and mapping each phrase to a story you'll tell.

Ladder phrase: "significant scope / ambiguity." Translate to: I was handed a goal, not a spec, and I defined the problem.
Ladder phrase: "influence beyond your team." Translate to: another team changed what they built because of my work.
Ladder phrase: "technical leadership." Translate to: I wrote the doc / drove the decision the group rallied around.
Ladder phrase: "force multiplier." Translate to: my output made N other engineers measurably faster.

Question. Decode a staff data engineer ladder rung into a prep checklist that maps each abstract requirement to a concrete, quantified story you can defend.

Input.

Ladder phrase (abstract)	What it really asks	Evidence to prepare
operates with significant ambiguity	you define problems, not just solve them	a "goal, no spec" story
influence beyond the team	others changed course because of you	a cross-team adoption story
technical leadership	you drove the decision	an RFC / design you authored
force multiplier	you made others faster	a platform / tooling adoption metric

Code.

LADDER-DECODE CHECKLIST — staff data engineer
=============================================

[ ] AMBIGUITY  ->  Story where the ask was "improve data reliability"
                   (a goal) not "add retries to job X" (a task).
                   Prove: I chose the metric, scoped the work, defended it.

[ ] INFLUENCE  ->  Story where Team B adopted my standard / changed
                   their design. Prove with THEIR outcome, not mine.
                   "After the contract, their on-call pages dropped 60%."

[ ] LEADERSHIP ->  A document I authored (RFC/ADR/strategy) that a
                   group of 3+ engineers aligned on. Bring the doc.

[ ] MULTIPLIER ->  A platform/library/pattern N teams reuse.
                   Metric: "12 pipelines migrated; onboarding 2wk -> 1day."

[ ] JUDGMENT   ->  A time I chose NOT to build / chose to deprecate.
                   Prove taste: what I traded away, and why it was right.

RED FLAG SELF-CHECK
  If every checklist item is filled by the SAME project, your scope is
  narrow. Staff evidence should span 2-3 distinct efforts.

Step-by-step explanation.

The decode step is deliberately literal: for each abstract ladder phrase, write the one-sentence behavior it is really asking for. Interviewers grade behaviors, not adjectives, so you must arrive with behaviors pre-mapped.
The ambiguity item forces a "goal, not spec" story. If your best story starts with a well-defined ticket, it's evidence for senior. Reach for the time a director said "our data is unreliable, fix it" and you turned that into a scoped program.
The influence item must be proven with the other team's outcome, not yours. "I built a data contract framework" is your output; "their on-call pages dropped 60% after adopting it" is your influence. Interviewers discount the former and reward the latter.
The multiplier item needs a number that divides work across people: "onboarding a new pipeline went from two weeks to one day, across twelve teams." That is leverage made legible.
The red-flag self-check catches the most common failure: filling every rung from one heroic project. Real staff scope shows up as a portfolio of cross-team wins, because a person operating at org scope naturally accumulates several.

Output.

Checklist item	Weak evidence (senior)	Strong evidence (staff)
Ambiguity	"I implemented the spec well"	"I defined the metric and the scope"
Influence	"I built a framework"	"Team B's pages fell 60% after adopting it"
Leadership	"I gave feedback in review"	"I authored the RFC the org aligned on"
Multiplier	"my pipeline is fast"	"12 teams onboard in 1 day, was 2 weeks"

Rule of thumb. Never present a ladder requirement back in the ladder's own words. Translate every abstract phrase into a concrete behavior with a number attached, and make sure your evidence spans multiple efforts — a single project that ticks every box signals a project win, not a level.

Worked example — the STAR-for-scope impact story template

Detailed explanation. The standard STAR template (Situation, Task, Action, Result) was designed for senior behavioral answers and quietly under-serves staff candidates, because it puts all the weight on Action (what you personally did) and treats Result as a single line. Staff stories need the weight on Scope (how big and ambiguous the problem was) and Leverage (how the result multiplied across the org). The STAR-for-scope variant re-balances the template for the staff loop. Walk through it.

Scope (replaces a thin Situation). How ambiguous and how large was the problem? Who else was affected? Why did it need you rather than a well-scoped ticket?
Tension. What made it hard — competing teams, no authority, unclear ownership, a real trade-off?
Action, framed as leadership. Emphasize the driving (RFC, alignment, mentoring) over the doing (the code you wrote).
Result, framed as a multiplier. Quantify the outcome as leverage: N teams, X% faster, $Y saved, an on-call burden removed — the other party's metric, not yours.
Reflection. What you'd do differently and what you learned about operating at scope — the signal of a self-aware operator.

Question. Convert a raw senior-flavored accomplishment ("I built a data quality framework") into a STAR-for-scope answer aimed at a staff loop.

Input.

STAR-for-scope slot	Prompt to yourself
Scope	how ambiguous / how many teams?
Tension	why was it hard without authority?
Action	what did I drive, not just do?
Result	what multiplier, in the other team's numbers?
Reflection	what did I learn about scope?

Code.

STAR-FOR-SCOPE — worked answer
==============================

SCOPE
  "Data quality was everyone's complaint and no one's job. Seven teams
   shipped dashboards off pipelines with no shared reliability bar.
   Leadership's ask was a goal, not a spec: 'stakeholders don't trust
   the numbers.' I scoped it into an org-wide data-contract program."

TENSION
  "I owned none of those seven pipelines and managed none of the teams.
   Two teams thought it was overhead. I had to win it on the merits."

ACTION (as leadership)
  "I wrote an RFC proposing schema + freshness contracts enforced in CI.
   I ran three design reviews, took a dissent from the streaming team
   that improved the freshness spec, and got disagree-and-commit from
   the last holdout. I built the reference implementation, then mentored
   one engineer per team to own their team's contracts."

RESULT (as multiplier)
  "12 pipelines under contract in a quarter. Data incidents down 55%.
   Stakeholder-reported 'bad number' tickets down 70%. New pipeline
   onboarding to the contract: 2 weeks -> 1 day. I wrote none of the
   12 teams' pipelines; they own them now."

REFLECTION
  "I over-invested in the framework before socializing it; next time I'd
   ship the RFC and one design review BEFORE writing a line of code."

Step-by-step explanation.

The Scope slot does the work that STAR's "Situation" usually wastes: it establishes that the problem was org-sized and ambiguous ("everyone's complaint and no one's job," a goal not a spec). This one paragraph is what makes the whole story a staff story.
The Tension slot names the absence of authority explicitly — "owned none, managed none." Staff interviewers are listening for whether you can produce outcomes without positional power; stating the constraint up front frames everything that follows as influence.
The Action slot deliberately foregrounds driving verbs (wrote the RFC, ran reviews, took a dissent, got disagree-and-commit, mentored) and backgrounds the coding ("I built the reference implementation" is one clause). This ratio is the tell of a staff operator.
The Result slot is quantified in other teams' metrics — incidents down 55%, their onboarding from two weeks to one day — and explicitly disclaims personal execution ("I wrote none of the 12 teams' pipelines"). That is leverage stated as a multiplier, not additive work.
The Reflection slot signals judgment and growth: admitting you over-built before socializing shows you understand that at staff scope, alignment precedes code. Interviewers read this as a self-correcting operator who will keep leveling.

Output.

Slot	Senior instinct	Staff-calibrated version
Situation/Scope	"my team's dashboards were flaky"	"7 teams, no shared bar, a goal not a spec"
Action	"I coded a QA framework"	"I drove the RFC + reviews + mentoring"
Result	"our data got more reliable"	"12 teams, incidents -55%, onboarding -90%"
Reflection	(often omitted)	"alignment should precede code"

Rule of thumb. Weight a staff behavioral answer 40% Scope+Tension, 30% Action-as-leadership, 30% Result-as-multiplier — the inverse of the senior weighting, which is mostly Action. If you find yourself spending the answer describing code you wrote, you are telling a senior story in a staff room.

Senior interview question on scope and leveling

A staff interviewer often opens the behavioral round with: "Tell me about the most impactful thing you've done. Then tell me why it was staff-level and not senior-level work." The trap is that most candidates answer the first half brilliantly — a deep, well-executed technical project — and then cannot articulate the second half, because the project was senior-level: solo, single-team, additive. The question is really asking you to demonstrate that you understand the leveling line well enough to locate your own work on it.

Solution Using the scope-first reframing of an impact story

ANSWER SCAFFOLD — "most impactful, and why it's staff"
======================================================

1. LEAD WITH SCOPE, NOT THE ARTIFACT
   Don't open with "I built X." Open with the org-sized problem:
   "Six teams were each rebuilding CDC ingestion; collectively ~4
    engineer-years/yr of duplicated effort and 6 inconsistent
    delete-handling stories feeding the same warehouse."

2. NAME WHY IT NEEDED A STAFF, NOT A TICKET
   "No single team owned 'ingestion' as a platform. It required
    someone to work ACROSS the teams with no authority over any."

3. DRIVE, THEN BUILD
   "I authored the platform RFC, aligned the 6 teams in 3 reviews,
    absorbed the security team's dissent on secrets handling, and
    got disagree-and-commit from the team that preferred its own
    tool. THEN I built the shared connector framework."

4. MULTIPLIER RESULT (other teams' numbers)
   "6 bespoke pipelines -> 1 platform, 18 connectors. New source
    onboarding: 3 weeks -> 2 days. ~3.5 engineer-years/yr reclaimed.
    Delete-handling now consistent across all sources."

5. EXPLICIT LEVELING CLAIM
   "It's staff-level because the unit of impact was the org's
    ingestion capability, not a system I own. The authority was
    influence; the outcome was other teams shipping faster."

Step-by-step trace.

Step	Senior framing	Staff framing (this answer)
Opening	"I built a connector framework"	"6 teams duplicated ~4 eng-years/yr"
Problem ownership	"my team needed better ingestion"	"no one owned ingestion as a platform"
Action emphasis	the code / architecture	the RFC, alignment, dissent, mentoring
Result unit	"our pipeline got faster"	"onboarding 3wk→2day across 6 teams"
Leveling claim	(absent)	"the unit of impact was the org's capability"

Walking the scaffold, the candidate first resizes the problem from a system to an org capability, then makes the absence of authority the central difficulty, spends the bulk of the answer on driving rather than doing, quantifies in the reclaimed engineer-years and the other teams' onboarding time, and finally closes by explicitly locating the work above the leveling line. The interviewer gets both halves of the question answered, and — critically — sees that the candidate can reason about levels, which is itself staff-level metacognition.

Output:

Grading signal	Weak answer	This answer
Scope	one system	the org's ingestion capability
Impact	additive (I shipped)	multiplier (6 teams faster)
Leadership	executed a plan	drove RFC + alignment + dissent
Judgment / metacognition	can't say why it's staff	names the leveling line explicitly
Quantification	"faster"	3wk→2day, ~3.5 eng-years/yr

Why this works — concept by concept:

Scope-first framing — leading with the org-sized problem instead of the artifact immediately signals the correct unit of work. The interviewer's first-impression grade is set by whether sentence one is about a system or about an org capability.
Authority-gap naming — explicitly stating "no authority over any of the six teams" reframes the entire story as influence-without-authority, which is the defining staff skill. It preempts the interviewer's follow-up ("how did you get them to agree?").
Drive-then-build ratio — spending four-fifths of the Action on driving (RFC, alignment, dissent, mentoring) and one-fifth on building matches how staff work actually decomposes, and inoculates against the "great senior, not yet staff" verdict.
Multiplier metrics — quantifying in reclaimed engineer-years and other teams' onboarding time expresses impact as leverage. Additive metrics ("my code is 40% faster") measure a system; multiplier metrics measure an org.
Cost — the preparation cost is real: you must have actually done cross-team work to tell this story, and you must have captured the numbers at the time. The recurring habit that makes this cheap is keeping a running "brag doc" of quantified, cross-team outcomes — O(minutes/week) that compounds into a promo packet and an interview arsenal.

Design
Topic — design
System design problems for the staff loop

Practice →

SQL Topic — sql SQL depth problems that still gate staff candidates

Practice →

2. Demonstrating scope: from project to platform to org

Scope is measured in blast radius, not lines of code — quantify who else got faster

The mental model in one line: scope and impact at staff level is measured in blast radius — the number of teams, systems, and future decisions your work touches — and the single most important reframe is from additive impact (work you personally completed) to multiplier impact (leverage that makes other people's work faster), because interviewers grade the multiplier and discount the additive. Two engineers can spend the same quarter and produce wildly different scope: one ships a beautiful pipeline used by one team; the other ships a slightly-less-beautiful platform twelve teams build on. The second is the staff story, and the difference is entirely blast radius.

The project → platform → org ladder — what each rung looks like in a data org.

Project (x1 leverage). You build a system your team uses. A revenue-attribution pipeline, a dbt model layer, a specific dashboard's backend. Real value, single-team blast radius. This is excellent senior work.
Platform (x10 leverage). You build something many teams reuse — a shared ingestion framework, a data-contract library, a self-serve backfill tool, a metrics layer. Your one quarter of work saves each of ten teams a quarter. The multiplier is the point.
Org (x100 leverage). You change how the whole org works — a standard everyone adopts, a migration off a dying architecture, a strategy that redirects the data roadmap for years. The blast radius is the org's technical direction.
The ladder is about aperture, not effort. Moving up a rung is rarely "work harder." It's "notice the version of this problem that N teams share, and solve that one." Staff engineers habitually scope up.

Quantifying scope — turning "impact" into defensible numbers.

Count the teams. "Adopted by 12 teams" is scope made legible. Always know the N.
Count the reclaimed time. "Onboarding a new source: 3 weeks → 2 days" across N teams is engineer-years reclaimed. Time is the universal currency.
Count the incidents removed. "On-call pages down 60%," "data incidents down 55%" — reliability leverage that other teams feel.
Count the dollars where honest. Compute cost cut, revenue enabled, headcount avoided. Use real numbers; a fabricated dollar figure is a fast way to fail a staff loop when the interviewer probes the derivation.

Multiplier vs additive — the distinction interviewers grade.

Additive impact. "I wrote 40 dbt models." "I cut our query time 40%." Value, but it's your output; it stops when you stop.
Multiplier impact. "I built the model-testing template 12 teams now use." "I made backfills self-serve so no team files a ticket to me anymore." Value that compounds through other people and persists without you.
The tell in a story. Additive stories have "I" as the subject of every verb. Multiplier stories have other teams as the subject of the result verb: "they onboard in a day," "their pages dropped."
The trap. Heroic additive work can even be negative multiplier — if you're the only one who understands the critical pipeline, you've created a bus-factor-one liability, which is an anti-signal at staff level.

Writing the scope narrative — the doc that makes your impact reviewable.

One page per major effort. Problem (org-sized), your role (drove vs did), the multiplier outcome (quantified), the artifacts (RFC, adoption metrics).
Lead with the number. "Reclaimed ~3.5 engineer-years/yr" before the story of how. Reviewers and interviewers skim; the number is the hook.
Name the counterfactual. "Without the platform, each of 6 teams would have kept rebuilding CDC." Counterfactuals make leverage concrete.
Keep it running. The scope narrative is the raw material for both the promo packet (section 5) and the interview answer; maintain it continuously, not the night before.

Worked example — the scope-sizing rubric

Detailed explanation. When an interviewer asks about impact, you need to size a piece of work quickly and honestly on the spot. The scope-sizing rubric is a scoring instrument that converts a vague accomplishment into a defensible level by scoring it on blast radius, leverage type, durability, and difficulty. Walk through the rubric and score two efforts with it.

Axis: blast radius. 1 = self, 2 = team, 3 = several teams, 4 = org, 5 = company/industry.
Axis: leverage type. additive (you did it) vs multiplier (others got faster) — a hard gate, not a slider.
Axis: durability. does the value persist without you (owned by others) or evaporate when you leave?
Axis: difficulty / ambiguity. was it a defined task or an ambiguous, contested problem?

Question. Score two efforts — "rewrote our ingestion service, cut latency 40%" and "drove a shared ingestion platform adopted by 6 teams" — and state which one is staff-level evidence and why.

Input.

Effort	Blast radius	Leverage	Durability	Ambiguity
Rewrote our ingestion service	2 (team)	additive	medium (I own it)	low (defined)
Drove shared ingestion platform	3–4 (6 teams / org)	multiplier	high (teams own theirs)	high (contested)

Code.

SCOPE-SIZING RUBRIC
===================
Score each effort; a STAFF effort needs multiplier leverage AND
blast radius >= 3 AND durability = high.

              blast   leverage     durability   ambiguity   -> level
              radius
------------------------------------------------------------------------
Ingestion       2      additive      medium        low       SENIOR
rewrite                (I did it)    (bus factor 1)

Ingestion       3-4    MULTIPLIER    high          high      STAFF
platform               (6 teams      (teams own    (contested
                        faster)       their part)   design)

GATES (all must pass for "staff evidence")
  [x] multiplier, not additive
  [x] blast radius >= 3 teams
  [x] value persists without me
  [x] I drove alignment under real ambiguity

VERDICT: lead the loop with the platform story; hold the rewrite for
         the deep-dive round as proof of depth.

Step-by-step explanation.

Score blast radius first — it's the fastest disqualifier. The ingestion rewrite tops out at "team" (2); the platform reaches "several teams / org" (3–4). Anything scoring 1–2 is senior evidence no matter how technically impressive.
Apply the leverage gate. This is binary, not a slider: the rewrite is additive (your output), the platform is a multiplier (others' output accelerates). An additive effort cannot be staff evidence regardless of its other scores.
Check durability. The rewrite has a hidden negative: if you're the sole owner, it's bus-factor-one — a liability. The platform pushes ownership out to each team, so its value persists without you. Durability distinguishes leverage from heroics.
Weigh ambiguity. The rewrite was a defined task; the platform was a contested, cross-team design with no clear owner. Ambiguity is where staff judgment shows, so it lifts the platform further.
Convert scores to a strategy, not just a label: lead the behavioral rounds with the platform story (staff evidence) and keep the rewrite for the deep-dive round, where depth is exactly what's being graded. Both stories are useful; the rubric tells you where to deploy each.

Output.

Effort	Rubric verdict	Where to use it in the loop
Ingestion rewrite	senior evidence (depth)	deep-dive / coding round
Ingestion platform	staff evidence (leverage)	behavioral / scope round

Rule of thumb. Score every candidate story on blast radius and the additive-vs-multiplier gate before an interview. Lead with multiplier stories that score ≥3 on blast radius and high on durability; hold additive-but-deep stories for the technical rounds where depth is the thing being tested.

Worked example — the one-page scope narrative doc

Detailed explanation. The scope narrative is a one-page document per major effort that you maintain continuously and mine for both promo packets and interviews. It forces you to state the org-sized problem, your driving role, and the quantified multiplier before you need it under pressure. Walk through the template and fill it for a real effort.

Header. The effort name and the one-line multiplier ("Reclaimed ~3.5 eng-years/yr across 6 teams").
Problem. The org-sized, ambiguous framing — why it needed a staff, not a ticket.
Role. What you drove (RFC, alignment, mentoring) vs did (built the reference impl).
Outcome. Quantified in other teams' metrics, with the counterfactual.
Artifacts. Links to the RFC, the adoption dashboard, the testimonials.

Question. Write the one-page scope narrative for the shared ingestion platform effort so it can seed both a promo packet and an interview answer.

Input.

Section	Content to capture
Header + multiplier	the headline number
Problem	org-sized, ambiguous framing
Role	drove vs did
Outcome	other teams' numbers + counterfactual
Artifacts	RFC, adoption metrics, testimonials

Code.

# Scope narrative — Shared Ingestion Platform (2026 H1)

**Multiplier:** ~3.5 engineer-years/yr reclaimed across 6 teams;
new-source onboarding 3 weeks -> 2 days.

## Problem (org-sized, ambiguous)
Six teams each maintained a bespoke CDC ingestion service. Duplicated
~4 eng-years/yr, 6 inconsistent delete-handling stories feeding one
warehouse, and no single owner of "ingestion" as a capability.
Leadership ask was a goal ("ingestion is a mess"), not a spec.

## My role (drove > did)
- Authored the platform RFC (linked).
- Ran 3 cross-team design reviews; 6 teams + security.
- Absorbed security's dissent on secrets handling into the design.
- Got disagree-and-commit from the team that preferred its own tool.
- Built the reference connector framework.
- Mentored 1 engineer/team to own their connectors (durability).

## Outcome (their numbers + counterfactual)
- 6 bespoke pipelines -> 1 platform, 18 connectors.
- Onboarding a new source: 3 weeks -> 2 days.
- Delete-handling now consistent across all sources.
- Counterfactual: without it, 6 teams keep rebuilding CDC yearly.

## Artifacts
- RFC: link  |  Adoption dashboard: link  |  Testimonials: 3 (linked)

Step-by-step explanation.

The header leads with the multiplier number because every reader — promo committee, interviewer, skip-level — skims. "~3.5 engineer-years/yr across 6 teams" front-loads the leverage so the rest of the page is justification, not suspense.
The Problem section is written to be unambiguously org-sized: six teams, one warehouse, no owner, and a goal-not-spec mandate. This framing is what makes the effort promo-worthy and staff-level rather than a nice project.
The Role section maintains the drive-then-build ratio in bullet form — five driving bullets, one building bullet — and includes the mentoring bullet explicitly, because durability (teams owning their own connectors) is what separates leverage from a bus-factor-one hero project.
The Outcome section quantifies in the other teams' currency (onboarding time, consistency) and states the counterfactual, which is what converts "we built a platform" into "we prevented six teams from wasting four engineer-years a year."
The Artifacts section makes the whole thing auditable: the RFC proves leadership, the adoption dashboard proves the multiplier, the testimonials prove influence. A scope claim without artifacts is discounted; with artifacts it's evidence.

Output.

Consumer of the doc	What they extract
Promo committee	multiplier + artifacts = a leveling case
Staff interviewer	scope-first behavioral story, pre-quantified
Your manager	ammunition for the calibration room
Future you	the numbers you'll otherwise forget

Rule of thumb. Keep one scope-narrative page per major effort and update it the week the outcome lands, while the numbers are fresh. The single most expensive interview mistake is having done staff work but being unable to quantify it six months later.

Worked example — reframing an additive answer into a multiplier answer live

Detailed explanation. Interviewers will sometimes catch you giving an additive answer and probe: "That's impressive, but what was the broader impact?" You need to reframe on the fly from the system you built to the leverage it created. Walk through the live reframing move on a common additive answer.

The additive answer. "I optimized our nightly batch from 6 hours to 90 minutes."
The reframe axes. Who else was blocked by the 6-hour window? What did the freed time enable? Did the technique spread?
The multiplier version. The optimization unblocked downstream teams and the technique became a standard.

Question. Reframe "I cut our nightly batch from 6h to 90m" from an additive answer into a multiplier answer without inventing facts.

Input.

Reframe axis	Additive answer	Multiplier reframe
Who was blocked	"our job was slow"	"3 downstream teams waited on it"
What it enabled	"job finished sooner"	"morning SLA met; 3 teams' dashboards fresh by 7am"
Did it spread	"I tuned our job"	"I documented the partition-pruning pattern; 5 teams adopted it"

Code.

LIVE REFRAME — additive -> multiplier
=====================================

ADDITIVE (what most candidates say)
  "I cut our nightly batch from 6 hours to 90 minutes by fixing
   partition pruning and a skewed join."

  -> grades as: strong senior. Depth, single system.

MULTIPLIER (same facts, reframed for scope)
  "Our 6-hour batch was the critical-path dependency for 3 downstream
   teams whose dashboards missed the 7am SLA. I cut it to 90 minutes
   (partition pruning + skew fix), which unblocked all 3 teams' morning
   SLA. I then wrote up the partition-pruning pattern as an internal
   playbook; 5 other teams applied it to their own jobs. The fix became
   a standard, not a one-off."

  -> grades as: staff-adjacent. Blast radius (3 blocked + 5 adopters),
     multiplier (a spreading technique), durability (a playbook).

RULE: never state a technical win without stating who ELSE it unblocked
      and whether the TECHNIQUE spread.

Step-by-step explanation.

Start from the same true facts — the 6h→90m optimization is real. Reframing is not fabrication; it's surfacing the leverage that was always there but that the additive framing hid.
Add the who was blocked axis: the slow batch was a critical-path dependency for three downstream teams missing a 7am SLA. This converts "my job was slow" into "three teams were blocked," expanding the blast radius honestly.
Add the what it enabled axis: the fix unblocked all three teams' morning SLA. The outcome is now stated in the other teams' currency (their dashboards, their SLA), not yours (your runtime).
Add the did it spread axis: documenting the partition-pruning pattern as a playbook that five teams adopted turns a one-off tuning into a spreading technique — the durability + multiplier signal.
The result grades a full level higher with zero invented facts. The lesson generalizes into a hard rule: never present a technical win without immediately attaching who else it unblocked and whether the technique propagated.

Output.

Framing	Blast radius	Leverage	Interviewer grade
Additive ("my job is faster")	1 team	additive	strong senior
Multiplier (blocked + spread)	3 + 5 teams	multiplier + durable	staff-adjacent

Rule of thumb. For every technical accomplishment, prepare the two extra sentences — "who else it unblocked" and "whether the technique spread" — before the interview. These two sentences are the difference between a senior grade and a staff grade on the identical underlying work.

Senior interview question on demonstrating scope

A staff interviewer might ask: "Tell me about your biggest-impact project. Walk me through how you'd measure that impact if I were skeptical of the number." The second sentence is the real test: it probes whether your impact claim survives adversarial scrutiny, which separates candidates who genuinely operated at scope from those who inflated a team project into an org story. You must be able to derive the number, name the counterfactual, and concede its limits.

Solution Using a defensible, counterfactual-anchored impact derivation

DEFENSIBLE IMPACT DERIVATION — biggest-impact project
=====================================================

THE CLAIM
  "The shared ingestion platform reclaimed ~3.5 engineer-years/yr
   across 6 teams."

THE DERIVATION (show your work)
  - Before: each of 6 teams spent ~0.7 eng-yr/yr building + maintaining
    a bespoke CDC connector set  ->  6 x 0.7 = 4.2 eng-yr/yr baseline.
  - After: the platform team spends ~0.7 eng-yr/yr; the 6 teams spend
    ~0 on ingestion internals (they configure, they don't build).
  - Net reclaimed: 4.2 - 0.7 = ~3.5 eng-yr/yr.

THE COUNTERFACTUAL
  "Without the platform, headcount growth would have forced a 7th and
   8th bespoke connector set this year. We avoided that."

THE HONEST LIMITS
  - "The 0.7/team figure is from 3 teams' time logs, extrapolated to 6.
     It's an estimate, not instrumented; I'd call it +/- 25%."
  - "Some of the reclaimed time went to other work, not pure savings;
     the durable win is CONSISTENCY (one delete-handling story) more
     than the headline hours."

THE MULTIPLIER RESTATEMENT
  "Even at the pessimistic end (~2.6 eng-yr/yr), the value is a
   multiplier: 6 teams ship ingestion in days, not weeks, forever."

Step-by-step trace.

Step	What the answer does	Why it survives scrutiny
State the claim	one crisp number	gives the interviewer a target
Show the derivation	6 × 0.7 − 0.7 = 3.5	the number is reconstructable, not asserted
Name the counterfactual	avoided 7th/8th bespoke set	leverage is concrete, not hypothetical
Concede the limits	±25%, partly extrapolated	honesty signals senior+ integrity
Restate the multiplier	even pessimistic case is leverage	the conclusion is robust to the error bars

The answer opens with a single crisp claim, then shows its work — a simple 6 × 0.7 − 0.7 arithmetic any interviewer can follow and challenge. It anchors the value in a counterfactual (the bespoke connectors that were never built), then volunteers the estimate's limits before the interviewer has to extract them, which converts a potential credibility risk into an integrity signal. Finally it restates the conclusion at the pessimistic end of the error bars to show the multiplier survives even if the headline number is soft.

Output:

Grading signal	Weak answer	This answer
Number quality	asserted, round, unfalsifiable	derived, bounded, falsifiable
Counterfactual	absent	explicit (avoided 7th/8th build)
Intellectual honesty	defends the number to the death	concedes ±25% unprompted
Robustness	collapses under one probe	holds even at pessimistic end
Real durable win	conflated with the hours	named separately (consistency)

Why this works — concept by concept:

Show-your-work derivation — reconstructing the number from first principles (per-team cost × teams, minus platform cost) makes it falsifiable, and a falsifiable number reads as true. Round, un-derived numbers ("saved millions") read as inflated and invite a takedown.
Counterfactual anchoring — impact is only meaningful against what would have happened otherwise. Naming the seventh and eighth bespoke connectors that were never built turns an abstract "savings" into a concrete avoided cost.
Volunteered limits — conceding the ±25% and the extrapolation before being asked is the single strongest credibility move. It signals you reason about your own claims adversarially, which is exactly the judgment staff level requires.
Multiplier-at-pessimistic-end — restating the conclusion at the low end of the error bars proves the argument doesn't depend on the optimistic number. A conclusion robust to its own uncertainty is far more persuasive than a precise-but-fragile one.
Cost — the enabling investment is instrumentation you set up before the win: time logs, adoption dashboards, before/after baselines. Capturing baselines is O(hours) up front but makes every future impact claim cheap and defensible; skipping it makes every claim a guess.

Design
Topic — design
Platform and scope design problems

Practice →

Optimization Topic — optimization Optimization problems behind multiplier impact stories

Practice →

3. Cross-team architecture loops and the RFC/design-review muscle

Drive the decision in writing, then let the room converge — consensus, not command

The mental model in one line: the cross-team architecture loop is the repeatable cycle by which a staff engineer drives a contested decision to closure across teams they don't manage — write the proposal down (RFC), run it through structured design review, absorb and integrate dissent, record the decision (ADR), and let the room converge on a choice it now co-owns — and mastering this loop in writing is the core technical leadership skill the staff interview probes, because it is how you produce org-level alignment with team-level authority. The junior instinct is to win the argument in the meeting; the staff instinct is to make the argument reviewable so the decision outlives the meeting and the room feels ownership of it.

The architecture-decision RACI — who does what in a cross-team decision.

Responsible. The staff engineer driving the RFC — writes it, runs the reviews, integrates feedback. This is you.
Accountable. Usually a single decision-maker (a principal, a director, or an explicitly-named DRI) who owns the final call if consensus stalls. Name this person before the review, not after it deadlocks.
Consulted. The teams affected — they provide input, dissent, constraints. Their sign-off is what makes the decision stick.
Informed. Downstream teams and leadership who need to know but don't shape the call. Over-consulting the "informed" group is a classic way to stall a decision.

The RFC / design-doc loop — the artifact that carries the decision.

Why writing wins. A meeting persuades the people in the room once; a document persuades everyone, asynchronously, forever, and forces you to find the holes before the room does. Amazon's "narrative over slides" culture exists for exactly this reason.
The RFC shape. Context and problem → goals and non-goals → the options considered → the recommendation → the trade-offs → the rollout and rollback. Non-goals are as important as goals; they prevent scope creep during review.
The review cadence. Circulate async for comments, then hold a focused review meeting for the unresolved disagreements only — never read the doc aloud. The meeting is for dissent, not narration.
The ADR. After the decision, record it as an Architecture Decision Record: the decision, the context, the alternatives rejected, and the consequences. The ADR is what stops the org re-litigating the same choice in six months.

Handling dissent — the disagree-and-commit machinery.

Steel-man the dissent. Restate the objecting team's position better than they did. This earns the right to be disagreed with, and often improves your design.
Integrate what's right. Real dissent usually contains a real constraint you missed. Fold it into the RFC and credit the objector — this converts an adversary into a co-owner.
Name one-way vs two-way doors. Reversible decisions (two-way doors) can be made fast and revisited; irreversible ones (one-way doors) deserve more consensus. Say which kind you're facing.
Escalate cleanly when stuck. If consensus genuinely fails, take the narrowed disagreement to the Accountable DRI with a recommendation, not a shrug. Disagree-and-commit means the room commits to the DRI's call even if some still disagree.

The tech radar and decision records — making decisions durable.

Tech radar. An adopt / trial / assess / hold quadrant for technologies. It turns "should we use X?" from a per-team debate into an org-level standard, which is leverage.
Decision records over tribal knowledge. Every consequential choice gets an ADR. New engineers read the ADRs instead of re-discovering the reasoning; the org stops paying the "why did we do this?" tax.
One-way vs two-way in the radar. "Hold" is a one-way-door signal ("don't adopt this without a strong case"); "trial" is a two-way door ("try it, we can back out").

Worked example — an RFC / ADR skeleton

Detailed explanation. The RFC is the load-bearing artifact of cross-team leadership, and having a skeleton you can instantiate fast is what lets you lead in writing under time pressure. Walk through a skeleton that covers context, goals/non-goals, options, recommendation, trade-offs, and rollout, plus the companion ADR that records the outcome.

RFC front matter. Title, author, reviewers (by team), status, one-way-or-two-way-door.
Body. Problem → goals/non-goals → options (with trade-offs) → recommendation → rollout/rollback → open questions.
ADR companion. The decision, the alternatives rejected, the consequences — written after the review.
The discipline. Non-goals and rejected-alternatives are mandatory; they're what stop re-litigation.

Question. Provide an RFC skeleton for a contested cross-team schema decision and the ADR that records the outcome.

Input.

Artifact	Purpose
RFC	drive the decision, async + review
Options section	make trade-offs explicit and comparable
Non-goals	prevent scope creep in review
ADR	record the decision so it isn't re-litigated

Code.

# RFC-014: Canonical event schema for the orders domain

- Author: <you>            Status: In review (target: 2026-08-15)
- Reviewers: Orders (DRI: Priya), Payments, Search, Data Platform, Security
- Door: TWO-WAY for field names (renamable via alias); ONE-WAY for the
        partition key (rewrites history) -> partition key needs consensus.

## 1. Problem
Three teams emit an "order" event with incompatible shapes. The warehouse
runs 3 reconciliation jobs to unify them; Search re-derives fields the
others already have. No canonical schema owner exists.

## 2. Goals / Non-goals
Goals:     one canonical orders event; backward-compatible evolution;
           a single delete-handling story.
Non-goals: unifying the CUSTOMER domain (separate RFC); changing the
           event bus; real-time (this is about schema, not transport).

## 3. Options considered
| Option | Pros | Cons |
|--------|------|------|
| A. Superset schema (all fields) | no team loses data | wide, sparse, unclear ownership |
| B. Core + extensions per team | clean core; teams extend | extension governance needed |
| C. Status quo + reconciliation | no migration | 3 recon jobs forever; drift |

## 4. Recommendation
Option B. Core owned by Orders; teams add namespaced extensions;
freshness + schema enforced in CI via data contracts.

## 5. Trade-offs
We trade some upfront governance (an extension review) for eliminating
3 reconciliation jobs and permanent schema drift.

## 6. Rollout / rollback
Dual-write core+legacy for 1 sprint; migrate consumers; deprecate legacy.
Rollback = keep reading legacy (dual-write makes this reversible).

## 7. Open questions
- Who approves new extensions? (proposed: Orders DRI + Data Platform)

# ADR-014: Adopt core+extensions schema for the orders domain

Status: Accepted (2026-08-15). Supersedes ad-hoc per-team schemas.

Decision: Option B (core + namespaced extensions), per RFC-014.

Context: 3 incompatible order events; 3 reconciliation jobs; no owner.

Alternatives rejected:
- Superset (A): unbounded width, no clear ownership.
- Status quo (C): permanent reconciliation cost and drift.

Consequences:
- (+) 3 reconciliation jobs removed; one delete-handling story.
- (+) Orders owns the core; extension review governs additions.
- (-) New process: extension approvals (Orders DRI + Data Platform).
- Dissent integrated: Search's latency concern -> extensions are
  read-time optional, not required joins. (credit: Marco, Search)

Step-by-step explanation.

The front matter names the reviewers by team and the DRI up front, and classifies the decision's reversibility per-field. Declaring "partition key is a one-way door, needs consensus" tells the room where to spend its scrutiny and where to move fast.
Goals/non-goals do the scope-control work. By explicitly ruling the customer domain and the event bus out of scope, the RFC prevents the review from ballooning into an everything-schema debate — the most common way cross-team RFCs die.
The options table makes trade-offs comparable rather than advocating a single answer. Presenting A/B/C with honest cons (including for the recommended option) signals you've genuinely considered alternatives, which earns the room's trust and preempts "did you think about X?"
The recommendation is decisive but grounded in the trade-off it accepts (upfront governance in exchange for killing three reconciliation jobs). Naming what you trade away is the staff move; a recommendation with no stated cost reads as naive.
The ADR, written after the review, records the decision and the integrated dissent — Search's latency concern became "extensions are read-time optional," credited to the objector. This is the disgree-and-commit machinery made durable: the objection is honored in writing, and the org never re-litigates it.

Output.

RFC element	What it prevents
Non-goals	scope creep during review
Options table with honest cons	"did you consider X?" ambush
Door classification	over-consensus on reversible calls
ADR + integrated dissent	re-litigating the decision later

Rule of thumb. Never drive a cross-team decision in a meeting alone — write the RFC first, make non-goals and rejected alternatives mandatory sections, classify each sub-decision as a one-way or two-way door, and close with an ADR that credits the dissent you integrated. The document is the leadership; the meeting is just where the last disagreements get resolved.

Worked example — a design-review facilitation script

Detailed explanation. Running the review meeting well is a distinct skill from writing the RFC. A staff engineer facilitates so that the decision gets made, dissent is heard, and the room converges — without the meeting devolving into narration or a rehash of resolved comments. Walk through a facilitation script for a cross-team design review.

Pre-read enforcement. The doc was circulated async; the meeting assumes it's read. Reserve the first few minutes of silent reading only if you must.
Agenda by disagreement. Spend time only on the unresolved comments and the open questions, not on the parts everyone agreed with.
Timeboxing and parking. Each contested point gets a timebox; unresolvable-in-the-room items get parked with an owner and a date.
Explicit decision + disagree-and-commit. End by stating the decision, confirming commit even from dissenters, and naming the ADR owner.

Question. Write the facilitation script a staff engineer uses to run the RFC-014 schema review to a decision in 45 minutes.

Input.

Phase	Time	Goal
Frame	5 min	restate decision + door type + DRI
Dissent	25 min	work only the unresolved comments
Converge	10 min	decide or escalate; disagree-and-commit
Close	5 min	ADR owner, follow-ups, dates

Code.

DESIGN-REVIEW FACILITATION SCRIPT — RFC-014 (45 min)
====================================================

[0:00] FRAME (5 min)
  "This review decides the orders event schema. Door: partition key is
   one-way (needs consensus today); field names are two-way. DRI if we
   don't converge: Priya. The doc was pre-read; we won't narrate it.
   Agenda is the 4 unresolved comments + 1 open question."

[0:05] WORK THE DISSENT (25 min) — one contested item at a time
  For each item:
   1. "Marco (Search), restate your concern in one sentence."
   2. Steel-man it back: "So extensions as required joins add read
      latency for Search. Is that right?"
   3. Options on the table, timebox 6 min.
   4. Decide or park: "Resolved: extensions are read-time optional.
      Marco, does that address it?" -> integrate into doc live.

[0:30] CONVERGE (10 min)
  "Recommendation stands: Option B, core owned by Orders. Any blocking
   objection to the ONE-WAY item (partition key = order_id)?"
   - If none: decided.
   - If blocked and unresolved: "We disagree-and-commit to Priya's call:
     partition key = order_id. Noted dissent: Payments prefers composite."

[0:40] CLOSE (5 min)
  "Decision: Option B, partition key order_id. ADR owner: me, by Fri.
   Follow-ups: extension-approval process (owner: Priya + Data Platform,
   next week). Thank you — dissent from Payments is recorded, not erased."

Step-by-step explanation.

The Frame phase sets the decision to be made, the door types, and the escalation DRI in the first five minutes. Rooms drift when they don't know whether they're deciding or discussing; naming the DRI up front means a stall has a pre-agreed resolution rather than an awkward silence.
The "work the dissent" phase is explicitly not a walkthrough — it iterates the unresolved comments only. For each, the facilitator makes the objector restate the concern, then steel-mans it back, which both confirms understanding and signals respect, defusing defensiveness.
Integrating resolutions live into the doc ("extensions are read-time optional") means the artifact leaves the meeting already updated, and the objector sees their concern honored in real time — the fastest path to genuine commit rather than grudging silence.
The Converge phase isolates the one-way-door decision (the partition key) for explicit consensus, and if consensus fails, invokes disagree-and-commit to the named DRI with the dissent recorded. This is how a decision gets made without steamrolling the minority.
The Close phase assigns the ADR owner and the follow-up owners with dates, and explicitly states that dissent is "recorded, not erased." Leaving the room with owners and honored dissent is what makes the decision stick and keeps the dissenters willing to engage next time.

Output.

Facilitation move	Failure it prevents
Frame the decision + DRI	endless discussion with no decision
Agenda = unresolved only	re-reading the doc aloud
Steel-man each dissent	objectors feeling unheard → later sabotage
Disagree-and-commit + recorded dissent	steamrolling or false consensus
ADR owner + dated follow-ups	the decision quietly not happening

Rule of thumb. Facilitate design reviews by decision, not by document: frame the call and the DRI in the first five minutes, spend the middle only on unresolved dissent (steel-manned and integrated live), and close with an explicit decision, disagree-and-commit, a recorded dissent, and dated owners. A review that ends without a decision and an owner was a status meeting, not a design review.

Senior interview question on cross-team alignment

A staff interviewer might ask: "Three teams need to agree on a shared schema. Two of them have already shipped incompatible versions and each thinks theirs should win. You have no authority over any of them and a six-week deadline. Walk me through exactly how you'd get to a decision everyone actually commits to." This is the signature staff scenario: contested, cross-team, deadline-bound, zero authority. The interviewer is grading your alignment process, not your schema opinion.

Solution Using the write-review-integrate-record alignment loop

ALIGNMENT LOOP — 3 teams, incompatible schemas, no authority, 6 weeks
=====================================================================

WEEK 1 — UNDERSTAND BEFORE PROPOSING
  - 1:1 with each team lead. Goal: understand WHY each shipped what they
    did. Capture the real constraint behind each version (Search needs
    field X denormalized; Payments needs a composite key for idempotency).
  - Name the Accountable DRI now (align with the 3 managers on one).

WEEK 2 — WRITE THE RFC
  - Options table: Team A's shape, Team B's shape, a synthesized Option C.
  - Steel-man A and B in the doc so both feel represented.
  - Classify doors: field names two-way; partition key one-way.
  - Circulate async; collect comments in the doc.

WEEK 3 — DESIGN REVIEW #1
  - Work only the unresolved comments. Integrate Payments' idempotency
    constraint into Option C live. Park the one truly-contested item.

WEEK 4 — RESOLVE THE PARKED ITEM
  - The one-way-door partition key. If A and B still disagree, take the
    NARROWED question + a recommendation to the DRI. Disagree-and-commit.

WEEK 5 — ADR + MIGRATION PLAN
  - Record the decision + rejected alternatives + integrated dissent.
  - Dual-write plan so BOTH existing versions have a reversible path in.

WEEK 6 — LAND + FOLLOW-UP OWNERS
  - Reference implementation + one owner per team mentored.
  - Extension-governance follow-up assigned with a date.

WHY THEY COMMIT
  Each team's real constraint is visibly in the final design; the DRI
  broke the one genuine tie; dissent is recorded, not erased.

Step-by-step trace.

Week	Move	Alignment mechanism
1	1:1s to find each team's real constraint	influence starts with understanding, not proposing
1	name the DRI	a pre-agreed tiebreaker prevents deadlock
2	RFC with steel-manned options	both teams feel represented in writing
3	review works unresolved items; integrate live	dissent becomes co-ownership
4	escalate the narrowed one-way-door item	disagree-and-commit on the one real tie
5–6	ADR + dual-write + mentored owners	reversible migration + durability

The loop front-loads understanding — a full week of 1:1s to surface why each team built what it did — because you cannot align teams whose real constraints you don't know. It names the DRI early so the one genuinely-contested decision has a pre-agreed resolution. The RFC steel-mans both existing versions so neither team feels erased, the review integrates the discovered constraints live, and only the single one-way-door decision (the partition key) is escalated — narrowed to one question with a recommendation. Dual-write gives both incumbent versions a reversible on-ramp, and mentoring one owner per team makes the outcome durable. Teams commit because each sees its real constraint in the final design and because the one true tie was broken by a legitimate DRI, not by whoever argued loudest.

Output:

Grading signal	Weak answer	This answer
Process	"I'd get everyone in a room"	a 6-week write-review-integrate-record loop
Understanding	jumps to proposing a schema	week of 1:1s to find real constraints
Authority handling	"I'd convince them"	named DRI + disagree-and-commit
Dissent	"I'd get buy-in"	steel-manned, integrated, recorded
Durability	"then we ship"	dual-write + mentored per-team owners

Why this works — concept by concept:

Understand-before-propose — starting with 1:1s to surface each team's real constraint is what makes alignment possible; a proposal written before you understand the constraints will be rejected on contact. Influence without authority is built on being the person who understood everyone's problem.
Named DRI + disagree-and-commit — pre-agreeing on who breaks a tie converts a potential deadlock into a bounded decision. Disagree-and-commit lets the minority stay engaged: they commit to execution without pretending to agree, and their dissent is recorded.
Steel-manned options in writing — representing both incumbent schemas fairly in the RFC means neither team enters the review feeling it has already lost, which is the precondition for genuine (not grudging) commitment.
Doors classification — spending consensus only on the one irreversible decision (the partition key) and moving fast on the reversible ones (field names, aliasable) is how you hit a six-week deadline without either steamrolling or stalling.
Cost — the loop costs ~6 weeks of calendar and a week of pure listening before any code — expensive in the short run, but it's O(1) in re-litigation: the ADR and integrated dissent mean the decision is made once. The alternative (winning the meeting, skipping the doc) is cheap up front and O(N) forever as the choice gets re-argued every quarter.

Design
Topic — design
Cross-team architecture and schema design problems

Practice →

Streaming Topic — streaming Streaming and event-schema alignment problems

Practice →

4. System design at staff/principal altitude: trade-offs and failure narratives

Requirements first, trade-offs second, tech names last — the altitude of a staff answer

The mental model in one line: a staff/principal system-design answer operates at a higher altitude than a senior one — it starts from requirements and non-functional requirements, reasons explicitly about the architecture review trade-offs between cost, latency, and reliability, walks a concrete design, and then narrates the failure modes and their blast radius — whereas a weaker answer jumps straight to naming technologies ("I'd use Kafka and Snowflake") before establishing what problem the design must actually solve. The senior answer builds a system that works; the staff answer defends a system against the ways it will break and the budget it will consume, and names what it deliberately does not optimize.

The altitude ladder — where a staff answer spends its time.

Requirements first. Before any box on the whiteboard: who are the consumers, what's the data volume and velocity, what's the freshness SLA, what's the consistency requirement, what's the budget? A staff candidate spends the first several minutes here and drives the requirement-gathering rather than waiting to be told.
Non-functional requirements (NFRs) second. Latency, throughput, availability, cost, security, multi-tenancy, evolvability. NFRs are where the design's hard trade-offs live; naming them up front is the altitude signal.
Design third. Now draw the boxes — ingestion, storage, transformation, serving — chosen to satisfy the NFRs, not because they're trendy.
Failure narratives last (and most important). What happens when a source is down, a partition is hot, the warehouse is unreachable, a schema changes? Name the blast radius of each failure and the mitigation. This is what separates a staff answer from a diagram.

The cost / latency / reliability trilemma — you cannot max all three.

The dial. Every data-platform decision trades among cost, latency, and reliability. Real-time exactly-once is expensive; cheap batch is high-latency; ultra-reliable multi-region doubles cost. Naming the dial and where you set it is the trade-off muscle.
Latency vs cost. Streaming (seconds) costs more in always-on compute than batch (hours). Pick latency to the requirement, not the maximum — "the BI consumers need hourly, the ML feature store needs sub-minute" justifies a hybrid.
Reliability vs cost. Multi-region replication, exactly-once, and long retention all cost money. Match the reliability tier to the data's criticality; not every dataset deserves five-nines.
The staff move. State the trilemma explicitly, set the dial per-consumer, and justify the setting from the requirements. Never silently max one axis.

Non-functional requirements for a data platform — the checklist.

Freshness / latency SLA. Per consumer. BI hourly, ML feature store sub-minute, audit daily. Different SLAs justify different architectures for different lanes.
Throughput and scale. Events/sec, GB/day, peak vs average. The peak drives the design; the average drives the cost.
Multi-tenancy and isolation. Can one tenant's load or bad data affect another? Isolation is a first-class NFR for platforms.
Evolvability. Schema change, new consumers, new sources — the platform must accommodate change without a rewrite. Staff designs optimize for the changes that will come.

Failure narratives — the part juniors skip and staff lead with.

Source failure. A source DB is down or lagging. Blast radius: which downstream consumers miss SLA? Mitigation: backfill, watermark, graceful degradation.
Hot partition / skew. One key dominates. Blast radius: a single lane backs up. Mitigation: salting, repartitioning, isolation.
Downstream unavailability. The warehouse or bus is unreachable. Blast radius: buffering limits, data loss risk. Mitigation: durable buffering, backpressure, retention.
Bad data / poison record. A malformed record breaks the pipeline. Blast radius: one lane vs the whole platform. Mitigation: dead-letter queue, schema/contract enforcement, quarantine.

Worked example — the NFR trade-off matrix

Detailed explanation. The NFR trade-off matrix is the artifact that turns "I'd use X" into "here's what each option costs on each axis that matters." Building it live in a design interview is the clearest possible altitude signal. Walk through constructing the matrix for the batch-vs-streaming-vs-hybrid decision on a data platform.

The axes. Freshness, cost, operational complexity, reliability/exactly-once, evolvability.
The options. Batch (hourly), streaming (seconds), hybrid (streaming hot path + batch reconciliation).
The scoring. Honest per-axis; the "best" option depends on the requirement, not on a universal winner.
The conclusion. Pick per-consumer, justify from the SLA.

Question. Build the NFR trade-off matrix for batch vs streaming vs hybrid ingestion, and recommend a setting for a platform serving both hourly BI and sub-minute ML features.

Input.

Consumer	Freshness need	Criticality
BI dashboards	hourly	medium
ML feature store	sub-minute	high
Audit / compliance	daily	high (durability)

Code.

NFR TRADE-OFF MATRIX — ingestion architecture
=============================================
                     BATCH        STREAMING       HYBRID
                     (hourly)     (seconds)       (stream hot + batch recon)
------------------------------------------------------------------------
Freshness            hourly       sub-second      sub-min hot / hourly cold
Cost                 $            $$$             $$  (pay stream only for
                                                      the hot lane)
Ops complexity       low          high            medium-high
Reliability /        easy         hard (dedup,    hot=at-least-once +
exactly-once         (idempotent  watermarks,     batch recon fixes dupes
                     reload)      state)          -> effectively exactly-once
Evolvability         high         medium          high (batch is the
                                                      source of truth)
------------------------------------------------------------------------
RECOMMENDATION
  Hybrid. Streaming hot path feeds the ML feature store (sub-minute,
  high criticality). Batch reconciliation feeds BI (hourly) AND repairs
  the hot path's duplicates -> exactly-once semantics without paying
  streaming's exactly-once complexity everywhere.
  Audit reads the batch (durable, daily) tier.

WHY NOT max freshness everywhere: streaming exactly-once for BI would
  triple cost for a consumer that only needs hourly. Set the dial per
  consumer.

Step-by-step explanation.

The matrix scores each option on the axes that the requirements made relevant — freshness, cost, ops, reliability, evolvability — rather than on generic virtues. The axes come from the NFRs, which is why requirement-gathering has to happen first.
The cost row is where the trilemma bites: streaming is $$$ because always-on compute and exactly-once state are expensive. Making cost an explicit row prevents the common failure of maxing freshness while ignoring the bill.
The reliability row exposes the subtle staff insight: pure streaming exactly-once is hard, but a hybrid gets effectively-exactly-once cheaply by letting periodic batch reconciliation repair the hot path's duplicates. This is a trade-off design, not a technology name.
The recommendation sets the dial per consumer: streaming only for the sub-minute high-criticality ML lane, batch for hourly BI and daily audit. This directly answers the requirement instead of applying one architecture everywhere.
The explicit "why not max freshness everywhere" is the altitude clincher — it names the rejected alternative (streaming for BI) and its cost (3× for no benefit), demonstrating that the design is a deliberate trade-off, not a default.

Output.

Consumer	Chosen lane	Justification from requirement
ML feature store	streaming hot path	sub-minute + high criticality
BI dashboards	batch tier	hourly is enough; save the cost
Audit	batch (durable)	daily + durability, not freshness
Dedup / correctness	batch reconciliation	exactly-once without streaming's cost

Rule of thumb. In any staff design round, draw the NFR trade-off matrix before drawing the architecture. Set the cost/latency/reliability dial per consumer from the SLA, and explicitly name the alternative you rejected and why. A design that maxes one axis silently is a senior answer; a design that sets the dial per requirement is a staff answer.

Worked example — a lakehouse/streaming design walkthrough with an SLA freshness check

Detailed explanation. With requirements and NFRs established, the staff candidate walks a concrete design and — critically — shows how the platform measures its own SLA. Walk through a medallion lakehouse with a streaming hot path, and include a real freshness-monitoring query, because "how do you know the SLA is met?" is a guaranteed follow-up.

Ingestion. Streaming hot path (CDC → bus) for the ML lane; batch loads for cold sources.
Storage. Medallion lakehouse: bronze (raw) → silver (cleaned/conformed) → gold (serving marts).
Serving. ML feature store off silver/gold streaming; BI off gold batch; audit off bronze/silver durable.
Observability. A freshness SLA query per gold table, alerted when freshness exceeds the per-consumer budget.

Question. Walk the lakehouse design and provide the freshness-SLA monitoring query that proves the platform meets its per-table freshness budget.

Input.

Layer	Content	Consumer
bronze	raw CDC + batch	audit, reprocessing
silver	cleaned, conformed	ML features, quality
gold	serving marts	BI, ML feature store

Code.

-- Freshness SLA check: is each gold table within its per-consumer budget?
-- gold_sla_config(table_name, freshness_budget_minutes) is the contract.
WITH freshness AS (
    SELECT
        t.table_name,
        t.freshness_budget_minutes,
        EXTRACT(EPOCH FROM (now() - MAX(g.loaded_at))) / 60.0
            AS minutes_since_load
    FROM        gold_sla_config       t
    JOIN        gold_load_audit       g  ON g.table_name = t.table_name
    GROUP BY    t.table_name, t.freshness_budget_minutes
)
SELECT
    table_name,
    ROUND(minutes_since_load, 1)                    AS age_min,
    freshness_budget_minutes                        AS budget_min,
    CASE
        WHEN minutes_since_load <= freshness_budget_minutes THEN 'OK'
        WHEN minutes_since_load <= freshness_budget_minutes * 1.5 THEN 'WARN'
        ELSE 'BREACH'
    END                                             AS sla_status
FROM   freshness
ORDER  BY minutes_since_load / freshness_budget_minutes DESC;  -- worst first

DESIGN WALKTHROUGH — narration to pair with the diagram
=======================================================
1. Ingestion: CDC hot path -> bus -> streaming into bronze (sub-min) for
   the ML lane. Batch loads for cold/low-priority sources into bronze.
2. bronze -> silver: streaming + micro-batch cleaning/conforming. Data
   contracts enforced here (schema + freshness) -> bad data quarantined.
3. silver -> gold: serving marts. ML feature store reads streaming gold
   (sub-min budget); BI reads batch gold (60-min budget).
4. Observability: the freshness query above runs every 5 min; BREACH
   pages the owning team. Each gold table has a budget in gold_sla_config.
5. Failure mode: if the hot path stalls, silver freshness breaches its
   budget, the query flags BREACH, and BI silently falls back to the last
   good batch gold (degraded, not broken). Blast radius = ML lane only.

Step-by-step explanation.

The design assigns each consumer to a lane whose freshness budget matches its SLA — sub-minute streaming gold for ML, 60-minute batch gold for BI — implementing the per-consumer dial from the NFR matrix as concrete boxes.
Data contracts are enforced at the bronze→silver boundary, so a bad record is quarantined before it reaches serving. This bounds the blast radius of a poison record to one lane rather than the whole platform — a failure-narrative decision baked into the architecture.
The freshness query is the observability backbone: it joins each gold table's actual age against its contracted budget in gold_sla_config and classifies OK/WARN/BREACH. Ordering by the ratio of age to budget surfaces the most-at-risk table first, which is what an on-call engineer needs.
The query answers the guaranteed interview follow-up — "how do you know the SLA is met?" — with a running, per-table, contract-driven check rather than a hand-wave. Showing the measurement is a stronger altitude signal than showing the pipeline.
The failure narration closes the loop: a stalled hot path breaches silver freshness, the query flags it, BI degrades gracefully to the last good batch, and the blast radius is confined to the ML lane. Naming the degraded-but-not-broken behavior and the blast radius is exactly the staff move juniors omit.

Output.

table_name	age_min	budget_min	sla_status
ml_features_gold	4.2	5	OK
orders_gold_bi	44.0	60	OK
sessions_gold	78.0	60	WARN
payments_gold	133.0	60	BREACH

Rule of thumb. Always pair a data-platform design with the query or metric that proves its SLA, and always narrate at least one failure mode with its blast radius and graceful-degradation path. A design you can't measure and can't defend against failure is a diagram; a design with an SLA check and a failure narrative is an architecture.

Worked example — reasoning about a one-way vs two-way door platform decision

Detailed explanation. Staff/principal design is as much about which decisions to sweat as about the decisions themselves. The one-way vs two-way door framing (from the disagree-and-commit toolkit) applies directly: spend design rigor on the irreversible choices and move fast on the reversible ones. Walk through classifying the major decisions in the platform design.

One-way doors (sweat these). Storage format/partitioning (rewriting history is expensive), the canonical schema/keys, the tenancy isolation model.
Two-way doors (move fast). Which BI tool, the exact streaming framework, retention windows (tunable), the number of medallion layers.
The interview signal. Correctly classifying decisions by reversibility shows judgment about where to invest scarce consensus and design time.
The trap. Over-designing a two-way door (endless debate on the BI tool) while under-designing a one-way door (picking a partition key casually) is the classic reversed-priority mistake.

Question. Classify the platform's major decisions as one-way or two-way doors and state where design rigor should concentrate.

Input.

Decision	Reversible?
Partition key / storage layout	hard to reverse (rewrites history)
Canonical schema + keys	hard to reverse (all consumers depend)
Tenancy isolation model	hard to reverse (security boundary)
BI tool choice	easy (swap the reader)
Streaming framework	medium (rewire the hot path)
Retention windows	easy (config change)

Code.

DOOR CLASSIFICATION — where to spend design rigor
=================================================
ONE-WAY DOORS  (invest consensus + design time; hard/expensive to undo)
  * Partition key + storage layout   -> rewrites history to change
  * Canonical schema + primary keys   -> every consumer couples to it
  * Multi-tenancy isolation model     -> security + blast-radius boundary
  RULE: RFC + broad review + ADR. Get these right the first time.

TWO-WAY DOORS (decide fast; revisit cheaply)
  * BI tool                           -> swap the reader, data unchanged
  * Streaming framework               -> rewire hot path, schema unchanged
  * Retention windows                 -> config change
  RULE: pick a reasonable default, timebox the debate, move on.

REVERSED-PRIORITY ANTI-PATTERN (the tell of weak judgment)
  Spending 3 meetings on the BI tool (two-way) while choosing the
  partition key (one-way) in a hallway conversation. Invert this.

Step-by-step explanation.

Classify each decision by the cost to reverse it, not by how much people want to argue about it. Storage layout and partition keys are one-way doors because changing them means rewriting history — expensive and risky at scale.
The canonical schema and keys are one-way because every consumer couples to them; a breaking change ripples across the org. These deserve the full RFC + review + ADR treatment from section 3.
The tenancy isolation model is one-way because it's a security and blast-radius boundary — getting it wrong later means a painful, risky migration under compliance pressure. Sweat it up front.
The two-way doors (BI tool, streaming framework, retention) get a reasonable default and a timeboxed debate, because the cost of being wrong is a bounded, reversible swap. Spending scarce consensus here is waste.
The reversed-priority anti-pattern — three meetings on the BI tool, a hallway decision on the partition key — is a direct tell of weak staff judgment. Naming and inverting it in a design round signals you know where rigor pays off.

Output.

Decision	Door	Design investment
Partition key / layout	one-way	RFC + broad review + ADR
Canonical schema + keys	one-way	RFC + broad review + ADR
Tenancy isolation	one-way	RFC + security review + ADR
BI tool	two-way	default + timeboxed debate
Streaming framework	two-way	default + timeboxed debate
Retention windows	two-way	config, revisit as needed

Rule of thumb. Classify every major platform decision as a one-way or two-way door and concentrate your RFCs, reviews, and consensus-building on the one-way doors. Deciding reversible choices quickly and irreversible choices carefully is the essence of staff-level design judgment.

Senior interview question on staff-altitude system design

A staff interviewer might ask: "Design a multi-tenant data ingestion platform that hundreds of internal teams onboard sources to themselves. Cover the requirements you'd gather, the NFRs, the tenancy isolation model, the cost/latency/reliability trade-offs, at least two failure modes with their blast radius, and how you'd measure whether the platform meets its SLA." This is a full staff design prompt — deliberately broad, self-serve, multi-tenant — and it rewards altitude (requirements and trade-offs) over technology name-dropping.

Solution Using a requirements-first, isolation-aware multi-tenant platform design

MULTI-TENANT INGESTION PLATFORM — staff design answer
=====================================================

1. REQUIREMENTS (drive these; don't wait to be told)
   - Consumers: 100s of internal teams, self-serve onboarding.
   - Volume: mixed; some tenants 10 GB/day, some 10 TB/day (skew!).
   - Freshness: per-tenant SLA (sub-min to daily).
   - Non-goal: not a public/external product; not a query engine.

2. NFRs
   - Multi-tenancy ISOLATION is the headline NFR: one noisy/bad tenant
     must not degrade others (blast-radius containment).
   - Self-serve evolvability: onboard a source without platform-team work.
   - Cost attribution: per-tenant so heavy tenants pay their share.

3. TENANCY ISOLATION MODEL (a one-way door -> sweat it)
   - Logical isolation: per-tenant namespaces + resource quotas on the
     shared bus and compute (rate limits, partition quotas).
   - Bad-data isolation: per-tenant dead-letter queues; a poison record
     quarantines to the tenant's DLQ, never blocks the shared pipeline.
   - Noisy-neighbor: per-tenant throughput quotas + priority classes.

4. TRADE-OFFS (set the dial)
   - Shared infra (cost-efficient) vs per-tenant infra (isolated but $$$).
     Choose SHARED with hard quotas -> cost efficiency + bounded blast
     radius. Offer dedicated tier for the few 10-TB/day tenants.

5. SELF-SERVE ONBOARDING
   - Declarative source config (a contract file); CI validates schema +
     freshness budget; platform provisions the lane. No tickets to us.

6. FAILURE MODES + BLAST RADIUS
   - Poison record: quarantined to tenant DLQ. Blast radius = 1 tenant.
   - Hot partition (10-TB tenant skew): per-tenant quota throttles it;
     others unaffected. Blast radius = the offending tenant's lane.
   - Bus outage: durable buffering + backpressure; freshness degrades
     platform-wide but no data loss. Blast radius = freshness SLA, not data.

7. SLA MEASUREMENT
   - Per-tenant freshness query (see gold_sla_config pattern); BREACH
     pages the TENANT'S owner, not the platform team -> ownership at scale.

Step-by-step trace.

Design decision	Rationale	Blast-radius / trade-off
Requirements + non-goals first	scope the prompt before drawing	avoids designing the wrong platform
Isolation = headline NFR	multi-tenant means noisy neighbors	contain failures to one tenant
Shared infra + hard quotas	cost efficiency without shared fate	dedicated tier for the few giants
Per-tenant DLQ	poison record containment	1 tenant, not the platform
Per-tenant quotas	skew/noisy-neighbor containment	offending lane only
Per-tenant SLA + owner paging	ownership scales, platform team doesn't	tenant owns its freshness

The answer opens by driving the requirements and stating non-goals, so the design targets the real problem (self-serve, multi-tenant, skewed volume) rather than a generic pipeline. It elevates isolation to the headline NFR because in a multi-tenant platform the dominant risk is shared fate, then chooses shared infrastructure with hard per-tenant quotas — the cost-efficient option that still contains blast radius — while offering a dedicated tier for the handful of giant tenants. Every failure mode is answered with its containment boundary (one tenant's DLQ, one tenant's lane, freshness-not-data), and the SLA is measured per-tenant with paging routed to the tenant's owner, which is how platform ownership scales to hundreds of teams without the platform team becoming the bottleneck.

Output:

Grading signal	Weak answer	This answer
Altitude	"Kafka + Spark + Snowflake"	requirements → NFRs → isolation → trade-offs
Multi-tenancy	ignored or hand-waved	isolation as the headline NFR
Trade-offs	maxes throughput	shared + quotas, dedicated tier for giants
Failure modes	absent	3 modes, each with a named blast radius
SLA / ownership	"we'd monitor it"	per-tenant SLA, tenant-owner paging

Why this works — concept by concept:

Requirements-first altitude — driving the requirements and non-goals before drawing boxes ensures the design solves the actual prompt (self-serve, multi-tenant, skewed) instead of a generic pipeline. The first five minutes on requirements is the clearest staff-vs-senior tell in a design round.
Isolation as the headline NFR — recognizing that multi-tenancy's dominant risk is shared fate, and elevating isolation above throughput, is the judgment that defines the design. Everything downstream (DLQs, quotas) flows from this one correctly-prioritized NFR.
Shared-plus-quotas trade-off — choosing shared infrastructure with hard per-tenant quotas captures cost efficiency while bounding blast radius, and offering a dedicated tier for the few giant tenants sets the dial per-tenant instead of one-size-fits-all. It's an explicit, defended trade-off, not a default.
Blast-radius-bounded failure modes — answering each failure with its containment boundary (one tenant, one lane, freshness-not-data) is the failure-narrative muscle that separates a staff design from a diagram. Naming the blast radius is more important than naming the tool.
Cost — the design's ongoing cost is the quota/isolation machinery and per-tenant observability, plus a dedicated tier for giants; the payoff is that the platform team's operational load stays O(1) as tenants grow to O(hundreds), because ownership (onboarding, SLA, paging) is pushed to each tenant. A platform without pushed-down ownership scales its own team's toil linearly with adoption — the anti-pattern this design avoids.

Design
Topic — design
Multi-tenant platform and data-system design problems

Practice →

ETL Topic — etl ETL and ingestion-pipeline design problems

Practice →

5. Influence without authority, promo packets, and the staff interview loop

Move the org without a manager title — glue work, force multiplication, and the promo case

The mental model in one line: influence without authority is the defining staff/principal skill — producing org-level outcomes with zero positional power over the people whose behavior must change — and it is expressed through force multiplication (mentoring, tooling, standards that make others faster), glue work (the unglamorous coordination that holds cross-team efforts together), and the ability to drive an initiative you don't own; the promo packet is simply the written, quantified evidence of this influence, and the staff interview loop is a structured attempt to verify it. A manager influences through reporting lines; a staff engineer influences through being the person others choose to align with because their judgment, writing, and reliability have earned it.

The sources of influence when you have no authority.

Earned technical credibility. Being right, repeatedly and visibly, in high-stakes decisions. Credibility is the currency you spend on influence; you earn it by shipping and by good calls under ambiguity.
Writing that others rally around. The RFC, the strategy doc, the ADR. A document that clarifies a messy problem becomes the org's plan, and its author becomes the de-facto leader of the effort.
Relationships and trust. The 1:1s, the reliability, the reputation for making others successful rather than hoarding credit. Influence flows through trust, and trust compounds.
Making others successful. The fastest way to gain influence is to make other people and teams win — mentoring, unblocking, amplifying their work. Influence is a byproduct of generosity, not extraction.

Force multiplication — the mechanism of staff impact.

Mentoring at scale. Not just pairing with one junior — creating the guild, the onboarding path, the design-review culture that levels up many engineers. A force multiplier raises the whole team's ceiling.
Tooling and platforms. The self-serve tool that removes you as a bottleneck (section 4) is force multiplication — every team it unblocks is leverage that persists without you.
Standards and patterns. The playbook, the reference implementation, the "how we do X here." A pattern adopted by ten teams multiplies one person's judgment across the org.
The anti-pattern. Being the indispensable hero who personally fixes everything is negative force multiplication — it creates a bottleneck and a bus-factor-one risk. Staff engineers work themselves out of being the bottleneck.

Glue work — the coordination that holds cross-team efforts together.

What it is. The unowned, often-invisible work of coordination: writing the doc, running the sync, chasing the dependency, noticing the gap between teams. It's what makes multi-team efforts actually ship.
The recognition trap. Glue work is essential but under-credited, and disproportionately falls on some engineers. The staff move is to do the high-leverage glue (driving the RFC, owning the cross-team plan) and to make it visible and quantified, not to drown in low-leverage glue (endless status-chasing).
Making it count. Frame glue work as leadership in your scope narrative: "I owned the cross-team migration plan" is leadership; "I sent a lot of reminder messages" is toil. Same activity, different framing and different leverage.
When to decline. Part of staff judgment is declining low-leverage glue so you can do high-leverage glue. Saying no to organizing the offsite to say yes to driving the architecture is a leveling-appropriate choice.

The promo packet and the staff interview loop.

The promo packet. A written case that you already operate at the target level: scope (the org-sized problems), impact (quantified multipliers), leadership (the RFCs, the mentoring), with peer and cross-team testimonials as evidence. It's the scope narratives (section 2) assembled into a leveling argument.
The staff interview loop. Typically: a system-design round (altitude, section 4), a technical deep-dive (proving depth is still there), one or more cross-team/behavioral rounds (scope, influence, dissent), a leadership round, and a values round. The behavioral rounds carry more weight than in a senior loop.
Staff-project selection. The single most-cited promo blocker is the absence of a demonstrably staff-scoped project. Choosing (and creating) a project with org-level blast radius is a deliberate career move, not luck.
The through-line. Promo packet and interview answers draw from the same well: quantified, cross-team, multiplier stories with artifacts. Build the well once (the running brag doc), draw from it for both.

Worked example — the promo-packet outline

Detailed explanation. The promo packet is the written argument that you already operate at staff level; a well-structured one makes the committee's decision easy by mapping your evidence directly onto the leveling rubric. Walk through an outline that a committee (or a hiring loop) can grade against the rubric in minutes.

Summary. The one-paragraph leveling claim with the headline multipliers.
Scope + impact. 2–3 org-sized efforts, each quantified in other teams' metrics, mapped to the rubric.
Leadership + influence. The RFCs authored, the alignment driven, the engineers mentored — with artifacts.
Evidence. Peer and cross-team testimonials, adoption dashboards, the ADRs.

Question. Write the promo-packet outline for a senior→staff data engineer case, structured so a committee can map it onto the leveling rubric.

Input.

Section	Maps to rubric dimension
Summary	overall level claim
Scope + impact	scope, impact (multiplier)
Leadership	leadership, influence
Judgment	judgment / taste
Evidence	verification (testimonials, artifacts)

Code.

# Promo case: Senior -> Staff Data Engineer — <name>

## Summary (the claim)
Operating at staff scope for ~3 quarters: drove 2 org-level platform
efforts reclaiming ~5 engineer-years/yr across 9 teams, authored the
data-contract standard now org policy, and mentored 4 engineers to
own cross-team surfaces. Requesting Staff.

## 1. Scope & Impact (rubric: scope, multiplier impact)
- Shared ingestion platform: 6 teams, onboarding 3wk->2day,
  ~3.5 eng-yr/yr reclaimed. [scope narrative + adoption dashboard]
- Data-contract standard: 9 teams, data incidents -55%,
  "bad number" tickets -70%. [RFC-014, ADR-014]
- (Each maps to blast radius >= 3 teams, multiplier, durable.)

## 2. Leadership & Influence (rubric: leadership, no-authority influence)
- Authored RFC-014 + 3 others; ran 11 cross-team design reviews.
- Integrated dissent from Security + Search into shipped designs.
- Mentored 4 engineers; 2 now drive their teams' contracts.
- Established the design-review guild (force multiplier).

## 3. Judgment & Taste (rubric: judgment)
- Chose NOT to build a bespoke streaming engine; standardized on the
  managed option -> saved ~1 eng-yr, avoided a maintenance trap.
- Deprecated 2 legacy reconciliation jobs after the contract landed.

## 4. Evidence
- Testimonials: 5 (3 cross-team leads, 1 partner PM, 1 mentee).
- Artifacts: RFC-014, ADR-014, adoption dashboards, guild charter.
- Manager + skip-level endorsement.

Step-by-step explanation.

The Summary states the leveling claim in one paragraph with the headline multipliers up front, so a committee member forms the "this is a staff case" impression in the first ten seconds — the same lead-with-the-number discipline as the scope narrative.
The Scope & Impact section presents 2–3 distinct org-sized efforts (not one project stretched), each quantified in other teams' metrics and each explicitly clearing the rubric gates (blast radius ≥3, multiplier, durable). The portfolio breadth is what proves sustained staff operation.
The Leadership & Influence section maps directly onto the no-authority-influence rubric dimension: RFCs authored, reviews run, dissent integrated, engineers mentored, a guild established. Each is a leadership behavior with an artifact, not an adjective.
The Judgment section supplies the hardest-to-fake signal: deliberate non-building and deprecation decisions. "Chose not to build a bespoke engine" and "deprecated two legacy jobs" demonstrate taste, which is the dimension that most distinguishes staff+ from a productive senior.
The Evidence section makes the whole case verifiable — cross-team testimonials (not just the manager), artifacts, and skip-level endorsement. A promo case without external verification is self-assessment; with it, it's evidence the committee can trust.

Output.

Packet section	Rubric dimension	Strongest evidence type
Summary	overall claim	headline multipliers
Scope & impact	scope + multiplier	other-team metrics + artifacts
Leadership	influence, no authority	RFCs + integrated dissent
Judgment	taste	deliberate non-building / deprecation
Evidence	verification	cross-team testimonials

Rule of thumb. Structure a promo packet to map one-to-one onto the leveling rubric, lead every section with a quantified multiplier, include at least one deliberate non-building decision as judgment evidence, and back every claim with a cross-team testimonial or artifact. A committee grades against the rubric; make their job trivial by pre-mapping your evidence to it.

Worked example — the influence / glue-work leverage rubric

Detailed explanation. Not all influence work is equal — some glue work is high-leverage leadership and some is low-leverage toil, and staff engineers deliberately choose the former and decline the latter. A leverage rubric for glue work helps you (and an interviewer) distinguish "drove the cross-team migration" from "sent status reminders." Walk through the rubric.

Axis: leverage. Does this glue work unblock many people (high) or is it one-off coordination (low)?
Axis: visibility / creditability. Is it framed as leadership with artifacts, or invisible toil?
Axis: durability. Does it create a lasting mechanism (a process, a doc, an owner) or evaporate?
The decision. Do the high-leverage, durable, visible glue; decline or delegate the low-leverage toil.

Question. Score three glue-work activities on the leverage rubric and decide which to own, which to delegate, and which to decline.

Input.

Glue activity	Leverage	Durability	Visibility
Drive the cross-team migration plan	high	high (a plan + owners)	high (you author it)
Mentor 1 engineer per team to own contracts	high	high (durable owners)	medium
Send weekly status-chase reminders	low	none	low

Code.

GLUE-WORK LEVERAGE RUBRIC
=========================
                          leverage  durability  visibility  -> action
--------------------------------------------------------------------------
Drive migration plan        HIGH      HIGH        HIGH        OWN IT
  (the doc becomes the org's plan; you become the de-facto lead)

Mentor 1 owner/team         HIGH      HIGH        MED         OWN IT
  (creates durable cross-team owners; force multiplication)
  -> raise visibility: name the mentees' wins in your scope narrative.

Weekly status reminders     LOW       NONE        LOW         DELEGATE
  (necessary but not leadership; rotate it or automate a dashboard)

STAFF JUDGMENT
  Say YES to high-leverage glue (it IS the leadership).
  Say NO / delegate low-leverage glue so it doesn't crowd out the
  high-leverage work. Doing all glue indiscriminately is a career trap:
  it's essential, under-credited, and infinite.

Step-by-step explanation.

Score each activity on leverage first — how many people it unblocks. Driving the migration plan and mentoring per-team owners are high-leverage; status-chasing is low-leverage even though it feels urgent.
Driving the migration plan scores high on all three axes: it unblocks everyone, it's durable (a plan with owners), and it's visible (you author it and become the de-facto lead). This is glue work that is leadership — own it unconditionally.
Mentoring one owner per team is high-leverage and durable (it creates lasting cross-team owners — force multiplication) but only medium-visibility, so the action includes raising its visibility by naming the mentees' wins in your scope narrative. Otherwise this leadership work goes uncredited.
Weekly status reminders are necessary but low on every axis — they don't scale your judgment, create nothing durable, and read as toil. The action is delegate or automate (a dashboard), freeing your time for the high-leverage glue.
The staff judgment is the meta-point: glue work is essential, under-credited, and infinite, so doing all of it indiscriminately is a career trap. Choosing the high-leverage glue and declining the rest is itself a leveling-appropriate act of judgment.

Output.

Glue activity	Rubric verdict	Decision
Drive migration plan	high/high/high	own it (it's leadership)
Mentor per-team owners	high/high/med	own it + raise visibility
Status reminders	low/none/low	delegate or automate

Rule of thumb. Triage glue work on leverage, durability, and visibility before agreeing to it. Own the high-leverage, durable glue and frame it as the leadership it is; delegate, automate, or decline the low-leverage toil. Indiscriminately absorbing all glue work is the most common way strong engineers stall below staff.

Worked example — a "drive an initiative you don't own" script

Detailed explanation. The purest test of influence without authority is driving an initiative that isn't yours — one that crosses teams, has no clear owner, and requires people who don't report to you to change what they do. Walk through the script a staff engineer uses to take an unowned org problem and move it to done.

Claim the problem, not the authority. Write the problem down clearly; becoming the person who articulated it makes you its de-facto owner.
Build the coalition. Recruit the affected teams by connecting the initiative to their goals, not yours.
Create the plan and the artifacts. The RFC, the roadmap, the owners — make the effort legible and trackable.
Sustain momentum. Run the cadence, unblock, and keep it visible to leadership so it doesn't quietly die.

Question. Write the script for driving an unowned, cross-team initiative (e.g., "our data lineage is a black box") to completion without authority.

Input.

Phase	Move
Frame	write the problem; become its articulator
Coalition	recruit teams via their goals
Plan	RFC + roadmap + named owners
Momentum	cadence + leadership visibility

Code.

DRIVE-WHAT-YOU-DON'T-OWN — script
=================================

1. FRAME THE PROBLEM (claim it by articulating it)
   Write a 1-page problem statement: "No one can answer 'where did this
   number come from?' Lineage gaps cause X incidents/quarter and Y hours
   of debugging across 4 teams." Circulate it. You are now the owner in
   everyone's mind, without a title.

2. BUILD THE COALITION (their goals, not yours)
   - Data Platform: "lineage reduces your on-call load."
   - Analytics: "lineage lets you trust the numbers you present."
   - Each team gets a reason rooted in THEIR pain, not your initiative.
   - Secure an executive sponsor for air cover (informed, not deciding).

3. PLAN + ARTIFACTS (make it legible)
   - RFC for the lineage approach; ADR for the decision.
   - Roadmap with a named owner per milestone (NOT all you).
   - A visible dashboard: % of pipelines with lineage coverage.

4. SUSTAIN MOMENTUM (the glue that ships it)
   - Biweekly 30-min cross-team sync; you facilitate, others report.
   - Unblock actively; escalate blockers to the sponsor with a rec.
   - Monthly update to leadership -> keeps it funded and alive.

5. HAND OFF (durability = you're not the bottleneck)
   - Each team owns its lineage; you own the standard, not the work.
   - Success = the initiative runs without you in the room.

Step-by-step explanation.

The initiative is claimed by articulating it, not by being assigned it — writing the crisp, quantified problem statement makes you the person everyone associates with the problem, which is de-facto ownership without any title change.
The coalition is built by connecting the initiative to each team's own goals — lineage reduces Data Platform's on-call load, lets Analytics trust its numbers. Influence without authority works by making your problem the solution to their problem.
An executive sponsor is recruited for air cover (informed, not deciding) so the initiative has legitimacy and funding, but the day-to-day driving stays with you — this is the difference between borrowing authority and being handed it.
The plan distributes ownership (a named owner per milestone, not all you) and makes progress legible via a coverage dashboard, so the effort is trackable and the multiplier is visible. Distributed ownership is what makes it durable and prevents bus-factor-one.
The final phase is the hand-off: success is explicitly defined as the initiative running without you in the room. Working yourself out of the bottleneck is the force-multiplication signal and the durability signal simultaneously — the opposite of the indispensable-hero anti-pattern.

Output.

Phase	Influence mechanism	Durability outcome
Frame	articulate → de-facto owner	the problem has a face
Coalition	their goals + sponsor air cover	teams opt in willingly
Plan	RFC + per-milestone owners	distributed ownership
Momentum	cadence + leadership visibility	stays funded and alive
Hand-off	teams own it, you own the standard	runs without you

Rule of thumb. To drive an initiative you don't own: claim it by writing the problem down, recruit each team through its own goals, secure a sponsor for air cover, distribute milestone ownership, and define success as the effort running without you. Influence without authority is engineered, not innate — it's the reproducible loop of articulate → coalition → plan → momentum → hand-off.

Senior interview question on influence without authority

A staff interviewer might ask: "Tell me about a time you drove an important initiative that wasn't yours — where you had no authority over the people you needed, and someone could reasonably have said 'that's not your job.' How did you get it done, and how did you handle the people who resisted?" This is the definitive influence-without-authority probe. The interviewer wants the mechanism (how you created movement without power) and the conflict (how you converted resistance into commitment), not a happy-path story where everyone cooperated.

Solution Using the articulate-coalition-integrate-handoff influence loop

INFLUENCE-WITHOUT-AUTHORITY — worked behavioral answer
======================================================

SCOPE (the unowned, org-sized problem)
  "Data lineage was a black box. Four teams collectively lost ~Y
   hours/quarter debugging 'where did this number come from,' and no
   team owned lineage. It was explicitly no one's job — including mine."

ARTICULATE (claim by writing)
  "I wrote a 1-page problem statement quantifying the debugging cost and
   circulated it. Within a week I was the person people pinged about
   lineage, with no title change."

COALITION (their goals + air cover)
  "I pitched each team on its own pain: Platform's on-call load,
   Analytics' trust in numbers. I got a director as sponsor for air
   cover and monthly visibility."

THE RESISTANCE (the real test)
  "One team lead pushed back hard: 'this is scope creep, we don't have
   the cycles.' I steel-manned it — he was right that a heavy lineage
   tool would burden his team. So I integrated his constraint: lineage
   capture was auto-derived from existing dbt metadata, near-zero effort
   for his team. He went from blocker to advocate."

RESULT (multiplier + durability)
  "80% lineage coverage in 2 quarters across 4 teams; debugging time for
   'where did this come from' down ~65%. Each team owns its lineage; I
   own the standard. It runs without me now."

REFLECTION
  "The resistance improved the design. I now seek out the strongest
   objector early — their constraint is usually the one I missed."

Step-by-step trace.

Step	Move	Why it converts resistance
Scope	quantify the unowned problem	makes the case on merit, not mandate
Articulate	write it → de-facto owner	ownership without a title
Coalition	pitch each team's own pain	opt-in, not compliance
Resistance	steel-man + integrate the objection	blocker → co-owner
Result	coverage + durability	multiplier that runs without you

The answer leads with the unowned, quantified problem — establishing that it was explicitly no one's job, which is what makes it an influence-without-authority story rather than an assigned-project story. It shows ownership being claimed by articulation, the coalition being built on each team's own incentives, and — the crux — it does not dodge the resistance. The objecting lead is steel-manned, his real constraint (tooling burden) is discovered and integrated (auto-derived lineage from existing dbt metadata), and that integration converts him from blocker to advocate. The result is a durable multiplier that runs without the candidate, and the reflection generalizes the lesson (seek the strongest objector early), signaling a self-improving operator.

Output:

Grading signal	Weak answer	This answer
No-authority framing	"I was asked to lead it"	"it was explicitly no one's job"
Mechanism of influence	"I convinced everyone"	articulate → coalition → air cover
Handling resistance	"eventually they agreed"	steel-manned + integrated the constraint
Conflict → commitment	glossed over	blocker became an advocate
Durability	"we shipped it"	runs without me; teams own it

Why this works — concept by concept:

Unowned-problem framing — establishing that the initiative was explicitly no one's job (including yours) is what makes it a genuine influence-without-authority story. An assigned project tests execution; an unowned one tests whether you can manufacture org movement from nothing.
Claim-by-articulation — writing and circulating the quantified problem statement is the mechanism by which authority-free ownership is created. The author of the clearest problem statement becomes the de-facto owner, every time.
Steel-man-and-integrate — the resistance is the graded part. Steel-manning the objector's concern, discovering the real constraint (tooling burden), and integrating a fix (auto-derived lineage) converts a blocker into an advocate. This is the single most persuasive thing you can show: conflict turned into commitment via design, not politics.
Durability / hand-off — ending with "it runs without me; teams own it" proves force multiplication rather than heroics. A story where you remain the indispensable operator is a negative staff signal.
Cost — influence-without-authority is expensive in relationship and writing time — the 1:1s, the problem statement, the coalition-building, the steel-manning — all before any technical work. But it's the only mechanism that scales impact past your own two hands, and the coalition and standard it builds are O(1) to maintain once established, versus the O(N) cost of personally coordinating every cross-team effort forever.

Design
Topic — design
Cross-team leadership and design-review problems

Practice →

SQL
Topic — sql
SQL problems that keep your depth interview-sharp

Practice →

Cheat sheet — staff and principal interview recipes

Leveling in one line. Senior owns a system, staff owns a problem space across teams, principal owns a technical strategy. The senior→staff boundary is the "who else has this problem?" reflex; authority stays flat while scope grows. Aim each interview story at the level's unit of work — a solo, single-team story is senior evidence even when technically brilliant.
The four graded signals. Every staff loop scores Scope (org-sized vs task-sized), Impact (multiplier vs additive), Leadership (drove vs executed), and Judgment (owned the trade-off vs froze/hand-waved). Prepare at least one story that scores high on all four, and make sure your evidence spans 2–3 efforts, not one heroic project.
Additive vs multiplier — the reframe. Additive impact is your output ("I wrote 40 models," "cut my job 40%"); multiplier impact makes others faster ("12 teams onboard in a day"). Never state a technical win without the two extra sentences: who else it unblocked, and whether the technique spread. Multiplier framing raises the identical work a full level.
Scope-sizing gates. For staff evidence, an effort must pass all three: multiplier (not additive) AND blast radius ≥3 teams AND durability high (value persists without you). Score each story before the loop; lead behavioral rounds with multiplier stories, hold additive-but-deep stories for the deep-dive round.
Scope narrative template. One page per major effort: header with the headline multiplier, org-sized problem framing, drove-vs-did role, outcome in other teams' metrics plus the counterfactual, and links to artifacts (RFC, adoption dashboard, testimonials). Update it the week the outcome lands, while the numbers are fresh.
Defensible impact numbers. Show your work (per-team cost × teams − platform cost), anchor to a counterfactual (the bespoke systems never built), volunteer the error bars (±25%, extrapolated) before being asked, and restate the conclusion at the pessimistic end. A derived, falsifiable number reads as true; a round, un-derived one invites a takedown.
RFC hygiene. Drive decisions in writing: problem → goals/non-goals → options (with honest cons for every option, including yours) → recommendation → trade-offs → rollout/rollback → open questions. Non-goals and rejected-alternatives are mandatory — they stop scope creep and re-litigation. Classify each sub-decision as a one-way or two-way door.
Design-review facilitation. Frame the decision + door type + DRI in the first five minutes; spend the middle only on unresolved dissent (steel-manned and integrated live); close with an explicit decision, disagree-and-commit, recorded dissent, and dated owners. A review that ends without a decision and an owner was a status meeting.
One-way vs two-way doors. Sweat the irreversible decisions (partition key/storage layout, canonical schema/keys, tenancy isolation) with full RFC + review + ADR; move fast on the reversible ones (BI tool, streaming framework, retention). The reversed-priority anti-pattern — three meetings on the BI tool, a hallway call on the partition key — is the tell of weak judgment.
System-design altitude. Requirements first, NFRs second, design third, failure narratives last (and most important). Draw the NFR trade-off matrix before the architecture; set the cost/latency/reliability dial per consumer from the SLA; name at least one failure mode with its blast radius and graceful-degradation path; and pair every design with the query/metric that proves its SLA.
Force multiplication vs heroics. Mentoring, tooling, and standards that make others faster are positive leverage; being the indispensable hero who personally fixes everything is negative leverage (bottleneck + bus-factor-one). Staff engineers work themselves out of being the bottleneck; "it runs without me" is the durability signal.
Glue-work triage. Triage glue work on leverage × durability × visibility before agreeing to it. Own the high-leverage, durable glue (drive the RFC, own the cross-team plan) and frame it as leadership; delegate or automate the low-leverage toil (status-chasing). Indiscriminately absorbing all glue is the most common sub-staff stall.
Promo packet + staff loop. Structure the packet to map one-to-one onto the leveling rubric (summary, scope+impact, leadership, judgment, evidence), lead each section with a quantified multiplier, include at least one deliberate non-building decision as judgment evidence, and back every claim with a cross-team testimonial or artifact. The loop weights behavioral/cross-team rounds more than a senior loop; the promo packet and interview answers draw from the same running brag doc.

Frequently asked questions

What is a staff data engineer?

A staff data engineer is a senior individual-contributor role — typically one level above senior on the technical (non-management) ladder — where the unit of impact shifts from a system you own to a problem space you make an entire org better at. A staff data engineer still writes code and designs systems, but the defining work is leverage: building the shared platform many teams reuse, authoring the standard the org adopts, and driving cross-team architecture decisions without managing anyone. The role is measured in scope and impact (how many teams and future decisions your work touches) rather than in throughput, and its signature skill is producing org-level outcomes with only team-level authority. In the staff data engineer interview, this is why the questions probe scope selection, cross-team leverage, and judgment far more than they probe your ability to write correct pipelines.

Staff vs principal data engineer — what's the difference?

The staff vs principal distinction is another change in the unit of ownership. A staff data engineer owns a problem space across teams — a platform, a standard, a migration — and makes many teams faster. A principal data engineer owns technical strategy: which architectures the org standardizes on for the next several years, which bets it makes, which systems it sunsets, and which locally-good ideas to say "no" to in service of a coherent direction. Put simply, staff is measured in cross-team leverage and principal is measured in org- or company-level judgment and taste. Both are individual-contributor roles with little to no positional authority, so both depend on influence without authority; the difference is that a staff engineer drives decisions while a principal shapes the space of decisions the org even considers.

How is the staff interview different from the senior interview?

The senior data engineer interview questions center on whether you can build correct, performant, well-designed systems: model this warehouse, tune this query, design this pipeline, debug this failure. The staff engineer interview assumes all of that and instead grades scope, cross-team impact, leadership, and judgment. Concretely: the system-design round rises in altitude (requirements and trade-offs and failure narratives over technology names), the behavioral rounds carry more weight and probe influence-without-authority and dissent-handling, and there is usually a dedicated cross-team or leadership round with no analog in a senior loop. The depth rounds don't disappear — you still have to prove you can code and design — but they become a gate rather than the decision. The decision is made in the rounds that test whether you operate at org scope.

How do I show impact without a title or authority?

You show impact without authority through technical leadership expressed in artifacts and outcomes rather than positional power. Author the RFC that the org rallies around — the person who writes the clearest problem statement and proposal becomes the de-facto owner, no title required. Build the coalition by connecting your initiative to each team's own goals so they opt in rather than comply, and secure an executive sponsor for air cover. Handle resistance by steel-manning objections and integrating the real constraint behind them, which converts blockers into co-owners. Then quantify the result in other teams' metrics (their onboarding time, their reduced incidents) and make it durable by distributing ownership so the effort runs without you. Influence without authority is a reproducible loop — articulate, coalition, integrate, hand off — not an innate trait.

Do staff data engineers still code?

Yes — staff and even principal data engineers still write code, and the depth rounds of the interview verify it. What changes is the ratio and the purpose. A staff engineer codes less of the routine feature work and more of the high-leverage kind: the reference implementation of a platform, the tricky proof-of-concept that de-risks an architecture decision, the tooling that removes a bottleneck. The failure mode is a staff engineer who has stopped coding entirely and become a pure coordinator — that person loses the technical credibility that their influence depends on. So the honest answer is that staff DEs code strategically: enough to stay credible and to build the leveraged artifacts, but not so much that they become the indispensable bottleneck the role exists to eliminate. The interview reflects this by keeping a real deep-dive/coding round alongside the scope and leadership rounds.

How do I prepare for the staff interview loop?

Prepare on two tracks. First, keep the depth sharp — the staff loop still has a system-design round and a technical deep-dive, so drill design, SQL, and platform fundamentals so the gate rounds are non-events. Second, and more decisively, build a portfolio of quantified, cross-team, multiplier stories: maintain a running "brag doc" of scope narratives (org-sized problem, drove-vs-did, other-teams' metrics, artifacts), score each against the four graded signals and the scope-sizing gates, and rehearse them in the STAR-for-scope structure that weights scope and multiplier over the code you wrote. Prepare the specific staff scenarios — align three teams on a schema, drive an initiative you don't own, design a multi-tenant platform — because they recur. Finally, prepare to reason about levels out loud: being able to say why a given piece of work is staff-level rather than senior-level is itself the metacognition the loop is testing.

Practice on PipeCode

Drill the design practice library → for the system-design and cross-team architecture rounds that anchor every staff and principal loop.
Keep your depth interview-sharp on the SQL practice library → so the deep-dive gate rounds are non-events while you focus on scope and leadership.
Rehearse the platform muscle on the streaming practice library → for the real-time, multi-tenant, and event-schema scenarios staff designs are built on.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to keep your fundamentals graded and sharp while you build the cross-team, multiplier stories the staff loop actually decides on.

Level up from senior to staff

Docs describe the staff bar; PipeCode drills the skills the staff loop actually gates — the system-design altitude, the SQL and platform depth that has to be a non-event, and the trade-off judgment behind every cross-team decision. Pipecode.ai is Leetcode for Data Engineering — trade-off-first practice tuned for the scope, impact, and architecture-leadership questions senior and staff data engineers actually face.

Practice design problems →
Practice streaming problems →

Chaos Engineering for Data Pipelines: Fault Injection with LitmusChaos & Gremlin

Gowtham Potureddi — Fri, 31 Jul 2026 14:43:48 +0000

chaos engineering data is the discipline of deliberately injecting failures — pod kills, network partitions, broker crashes, disk fills — into a running data pipeline to prove the pipeline stays inside its freshness, correctness, and lag budgets when the underlying infrastructure misbehaves. It is not a stunt, it is not load testing, and it is not "let's yank a cable and see what happens." It is a hypothesis-driven engineering practice with a written steady state, a stated hypothesis, a blast-radius-controlled experiment, and a verification step that either confirms the pipeline's resilience or produces a concrete backlog item to fix it — and in 2026 it has moved from "SRE toolbox for stateless microservices" to a table-stakes practice for any data platform team that owns a real freshness SLO.

This guide is the senior-data-platform-engineer walkthrough you wished existed the first time you had to answer "how do we prove our Kafka + Airflow + Spark stack survives a broker crash without our warehouse silently losing an hour of events?" It walks through why chaos engineering pipelines is qualitatively different from chaos on stateless services (state, ordering, exactly-once contracts change everything), how litmuschaos on Kubernetes uses a ChaosEngine custom resource + probe guardrails to run pod-delete / pod-network-loss / disk-fill experiments with auto-abort, how gremlin layers scenarios + a blast-radius dial + halt-on-breach health checks on top of raw attack primitives, and how to compose data fault injection — kafka chaos (broker kill + ISR shrink), airflow chaos (scheduler kill mid-DAG), spark chaos (executor loss) — into a repeatable game-day loop. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the ETL practice library →, and sharpen the architecture axis with the design practice library →.

On this page

Why chaos engineering is different for data pipelines
LitmusChaos on Kubernetes — pod / network / node kill experiments
Gremlin — scenarios, blast-radius controls, workflows
Data-plane fault injection — Kafka broker kill, Airflow scheduler kill, Spark executor loss
Hypothesis-driven chaos loop — steady state → hypothesis → blast → verify
Cheat sheet — chaos engineering data recipes
Frequently asked questions
Practice on PipeCode

1. Why chaos engineering is different for data pipelines

Stateful streams, exactly-once contracts, and freshness SLOs — data chaos plays by different rules

The one-sentence invariant: chaos engineering data is the practice of deliberately injecting infrastructure failures into a running data pipeline while a codified data SLO (freshness, lag, correctness) acts as the steady state that decides whether the experiment passes or halts — and every design choice in the experiment (blast radius, probe cadence, rollback path) has to account for the fact that streams are ordered, tables are stateful, and downstream consumers hold exactly-once contracts that a naive chaos experiment can violate. A chaos experiment against a stateless HTTP microservice that returns "hello world" is easy: kill a pod, watch p99 latency, done. A chaos experiment against a Kafka broker that owns 300 partition leaders with min.insync.replicas=2 is a different physics problem — one wrong move and you have unshipped events sitting in producer buffers, ISR shrinks that trigger leader-election storms, or a downstream Spark job that fails mid-shuffle and re-reads gigabytes it already processed.

The four "must-answer" axes for a data chaos experiment.

State. Where does the pipeline hold state, and what happens if that state is unavailable for N seconds? Kafka partitions have leader + ISR replicas, Airflow schedulers own the metadata DB, Spark drivers own the DAG lineage, Iceberg tables own atomic commits. Every one of these is a distinct state-loss failure mode with distinct recovery semantics.
Ordering. Does the pipeline require in-order processing, and does the chaos experiment preserve or violate that? A network partition between two Kafka brokers can cause events to arrive at consumers out of order relative to source-time. A Spark executor loss forces stage retry, which re-emits shuffle blocks that downstream sinks may have already committed.
Exactly-once. Does the chaos experiment risk breaking the exactly-once contract? Kafka + idempotent producers + transactional consumers hold EOS via a coordinator; if you kill the coordinator broker at exactly the wrong moment during a commit, some records commit twice on retry. The chaos experiment must either prove EOS holds or clearly report where it broke.
Freshness SLO. Is the steady-state SLI a data metric (freshness p99, end-to-end lag, DAG success rate, tombstone count) rather than an infra metric (CPU, memory, pod-ready count)? Data chaos experiments verify against data SLIs; infra chaos experiments verify against infra SLIs. Confuse the two and you'll ship "the pod restarted but the pipeline is fine" when actually the pipeline is 40 minutes behind.

The 2026 reality — chaos has crossed from stateless SRE into data platform SRE.

LitmusChaos is the CNCF sandbox → incubating open-source chaos framework that runs entirely on Kubernetes. It exposes chaos experiments as ChaosEngine custom resources with a library of ready-made faults (pod-delete, pod-network-loss, node-taint, disk-fill, pod-cpu-hog, pod-memory-hog) plus a probe system that lets you attach httpProbe, promProbe, cmdProbe, k8sProbe guardrails that auto-abort on breach. It is the default choice when your data plane runs on Kubernetes and you want everything GitOps-managed.
Gremlin is the commercial SaaS chaos platform with a control plane in the cloud and lightweight agents (gremlind) on every target host, VM, container, or Kubernetes node. It exposes attack primitives (CPU, memory, disk, IO, latency, packet-loss, blackhole, DNS, shutdown) that stitch into "Scenarios" — multi-step attack workflows with explicit blast-radius sliders and halt-on-breach Health Checks tied to Datadog / New Relic / Prometheus. It is the choice when your data plane spans VMs, on-prem, cloud, and containers, and you want a single control plane with SOC-2-ready audit logs.
Kafka / Airflow / Spark chaos is the "data-plane" layer where you inject faults specific to the stateful data systems — broker kills, ISR shrinks, scheduler restarts, executor loss, shuffle-fetch failures. These are typically composed on top of LitmusChaos or Gremlin: the tool provides the primitive; you provide the guardrails that catch data-SLO breaches specific to your stack.
Game days are the human ceremony that wraps all of the above — a scheduled hour where the team runs a chaos experiment against a production-like environment (or a small % of prod), watches the SLI dashboards live, halts on breach, and writes a post-mortem doc that lands as a backlog item. This is the pattern that turns chaos from an SRE side project into a resilience-forcing function.

What interviewers listen for.

Do you name steady state, hypothesis, experiment, verify without prompting? — senior signal; this is the Netflix / PoP chaos framework in one line.
Do you say "my steady state is a data SLO, not CPU" — freshness_p99, e2e lag, DAG success rate — rather than infrastructure metrics? — required answer.
Do you name blast radius as the first control and explain the dev → staging → 1% prod → 10% prod graduation? — senior signal.
Do you distinguish LitmusChaos (K8s-native, open source, GitOps) vs Gremlin (SaaS, cross-platform, commercial) rather than treating them as interchangeable? — senior signal.
Do you name a halt condition — the automatic abort that fires when the steady-state SLI breaches — rather than describing chaos as an open-loop attack? — required answer.

Worked example — the four-axis data chaos comparison table

Detailed explanation. The single most useful artifact for a chaos-engineering interview is a memorised 4×N comparison table that maps each candidate chaos experiment against the four axes: state loss, ordering impact, exactly-once impact, freshness SLO impact. Every senior chaos discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical stack that has Kafka → Spark Structured Streaming → Iceberg → dbt → Snowflake.

Stack under test. Kafka (MSK, 6 brokers) → Spark Structured Streaming on EKS → Iceberg on S3 → dbt scheduled by Airflow → Snowflake.
Steady-state SLIs. end-to-end freshness p99 < 5 min from producer to Iceberg; DAG success rate > 99%; Kafka consumer group lag < 10k messages.
Candidate experiments. pod-delete on a broker, pod-network-loss on the Spark driver, disk-fill on the Airflow scheduler pod, blackhole to S3 for 60 s.
Question at hand. For each experiment, which axes are at risk and what is the halt condition?

Question. Build the four-axis chaos comparison table and derive the halt condition for each experiment.

Input.

Experiment	State loss	Ordering risk	EOS risk	Freshness risk
pod-delete on Kafka broker	leader for ~30 partitions	in-partition order preserved; cross-partition unaffected	low if `acks=all` + `min.insync.replicas=2`	~10-30 s bump in lag
pod-network-loss on Spark driver	driver metadata unreachable	streaming batch stuck; no re-order	medium — checkpoint may skip micro-batch	freshness p99 grows unboundedly
disk-fill on Airflow scheduler	scheduler pod OOMs on next heartbeat	irrelevant (DAGs are not ordered)	irrelevant	DAG start delayed; downstream freshness slips
blackhole to S3 (60 s)	Iceberg writes fail; retries	retries preserve order per Iceberg commit	high — commit + manifest write are atomic	freshness stalls for ~60 s

Code.

# ChaosEngine — pod-delete on Kafka broker with promProbe halt condition
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-chaos
  namespace: kafka
spec:
  engineState: 'active'
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: CHAOS_INTERVAL
              value: '30'
            - name: FORCE
              value: 'false'
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 10
              retry: 2
              probePollingInterval: 10
              stopOnFailure: true
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator:
                type: int
                criteria: '<'
                value: '50000'

Step-by-step explanation.

The ChaosEngine custom resource declares a chaos experiment scoped to applabel: app=kafka-broker in the kafka namespace. LitmusChaos discovers the matching pods and runs the referenced experiment (pod-delete) against a randomly-chosen one.
TOTAL_CHAOS_DURATION=60 bounds the chaos window to 60 s; CHAOS_INTERVAL=30 means the pod is deleted every 30 s within that window (so the broker will be killed once, then again 30 s later). FORCE=false means Kubernetes performs a graceful shutdown (SIGTERM → configured grace period → SIGKILL) rather than an immediate SIGKILL, so the broker can drain in-flight requests.
The promProbe attaches a Prometheus query as a continuous guardrail. Every 10 s during the chaos window, LitmusChaos runs max(kafka_consumergroup_lag{group="warehouse-sink"}) against Prometheus. If the returned value is not < 50000, stopOnFailure=true causes LitmusChaos to auto-abort the experiment and mark it Failed.
The < 50000 threshold is the halt condition — the point at which we've decided the experiment has produced too much data-SLO damage to continue. This is where infrastructure chaos and data chaos diverge: the halt condition is a data metric (consumer lag), not pod_ready.
When the experiment completes, LitmusChaos writes a ChaosResult CR with the verdict (Pass if all probes passed, Fail if any probe breached). The verdict + probe timeline is the artifact you attach to the GameDay doc.

Output.

Time (s)	State	Consumer lag	Probe verdict
0	broker-2 running	800	Pass
5	broker-2 killed (SIGTERM)	900	Pass
20	leader re-election underway	1800	Pass
35	broker-2 restarted; ISR healing	5200	Pass
60	broker-2 in ISR; lag draining	3400	Pass (final)

Rule of thumb. Never write a data chaos experiment without a continuous probe tied to a data SLI. The probe is the seatbelt; the SLI is what you're protecting; the halt condition is the crash sensor. Miss any one and you're doing chaos-as-a-stunt, not chaos engineering.

Worked example — steady state as a data SLO, not an infra metric

Detailed explanation. The steady state is the single most under-specified concept in most first-attempt chaos programs. Teams reach for pod_ready == true or cpu_percent < 80 because those metrics are easy to graph. But a Kafka broker can be "pod_ready" while its ISR is shrunk to a single replica and every consumer is 45 minutes behind — infra says green, data says on-fire. A senior data chaos program defines steady state as a data SLI first, then adds infra SLIs as secondary indicators. Walk through the four canonical data SLIs and how to instrument each.

Freshness p99. For any table or topic, the 99th percentile age of the newest committed record vs. wall clock time. The single most useful pipeline SLI.
End-to-end lag. Consumer group lag in messages, or commit_ts - source_ts in a streaming SQL. Not the same as freshness (a lag of 0 with a stale source is not fresh).
DAG success rate. For scheduled batch pipelines, successful_runs / (successful_runs + failed_runs) over a rolling 24h window.
Correctness invariants. Row-count parity vs source, no negative revenue values, foreign-key completeness — pipeline-specific business assertions.

Question. Define steady state as a data SLO for a Kafka → Iceberg pipeline and derive the halt conditions for a chaos experiment that kills a Spark executor.

Input.

SLI	Definition	Target	Halt threshold
freshness_p99_min	max(now() - iceberg.commit_ts) p99	< 5 min	> 15 min
lag_messages	sum(kafka_lag) across group	< 10k	> 100k
batch_success_rate	5-min rolling	> 99%	< 90% for 10 min
row_count_delta	source - iceberg over 1h	< 100	> 10000

Code.

# Prometheus recording rules — one file, four data SLIs
groups:
  - name: pipeline-slis
    interval: 30s
    rules:
      - record: pipeline:freshness_p99_min
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(iceberg_commit_age_seconds_bucket{table="events"}[5m])
            )
          ) / 60

      - record: pipeline:lag_messages
        expr: |
          sum(kafka_consumergroup_lag{group="warehouse-sink"})

      - record: pipeline:batch_success_rate
        expr: |
          sum(rate(spark_streaming_batch_success_total[5m]))
          /
          sum(rate(spark_streaming_batch_total[5m]))

      - record: pipeline:row_count_delta
        expr: |
          (source_events_total - iceberg_rows_total{table="events"})

# ChaosEngine — Spark executor kill with all four SLIs as halt probes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: spark-executor-chaos
  namespace: spark
spec:
  engineState: 'active'
  appinfo:
    appns: 'spark'
    applabel: 'app=spark-executor'
    appkind: 'pod'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '300'
            - name: PODS_AFFECTED_PERC
              value: '10'
        probe:
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:lag_messages'
              comparator: {type: int, criteria: '<', value: '100000'}
          - name: batch-success-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              query: 'pipeline:batch_success_rate'
              comparator: {type: float, criteria: '>', value: '0.90'}
          - name: correctness-guard
            type: promProbe
            mode: EOT   # end-of-test only
            promProbe/inputs:
              query: 'pipeline:row_count_delta'
              comparator: {type: int, criteria: '<', value: '10000'}

Step-by-step explanation.

The four recording rules define the pipeline's steady state in Prometheus terms. Every one is a data metric — freshness age, consumer lag, batch success ratio, source-vs-sink row delta. None of them mention pod_ready or cpu_percent; those become infra SLIs, tracked separately for triage but not gating the chaos verdict.
pipeline:freshness_p99_min derives from a histogram of iceberg_commit_age_seconds — the age (in seconds) of the newest committed Iceberg snapshot at scrape time. Dividing by 60 gives minutes. histogram_quantile(0.99, ...) gives the p99, so short spikes don't panic the probe.
The ChaosEngine attaches all four SLIs as probes. Three are Continuous (evaluated every N seconds during chaos); one (correctness-guard) is EOT (end-of-test only) — you only compute row-count parity after the chaos window has closed and things have settled.
The PODS_AFFECTED_PERC=10 sets the blast radius to 10% of matching pods. If you have 20 Spark executors, LitmusChaos kills 2. This is the "start small, graduate up" principle applied to a single experiment.
When any continuous probe fails, LitmusChaos aborts the experiment. When the EOT probe fails, the experiment is marked failed even if nothing tripped during the run. The distinction matters: correctness bugs often surface after the chaos, when the pipeline catches up and reveals a divergence.

Output.

SLI dashboard	Baseline	During chaos	Post-chaos	Verdict
freshness_p99_min	2.1	4.8	2.3	Pass
lag_messages	800	45,000	1,200	Pass
batch_success_rate	1.00	0.94	1.00	Pass
row_count_delta	12	(n/a)	34	Pass (EOT)

Rule of thumb. Steady state = data SLI. Halt condition = threshold on that SLI. Anything else is chaos theatre. If you can't articulate what "the pipeline is healthy" means in a single Prometheus query, you're not ready to inject chaos.

Worked example — what interviewers actually probe on data chaos

Detailed explanation. The senior data-platform chaos interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you validate your Kafka + Spark stack is resilient?"), then progressively narrows to test whether you understand the framework. Candidates who name the steady-state / hypothesis / experiment / verify loop in sentence one score highest; candidates who describe "we'd kill some pods and see" score lowest. Walk through the interview grading rubric.

Ambiguous opener. "How do you know your pipeline survives a broker crash?" — invites you to name the framework.
Follow-up 1. "What's your steady state?" — probes SLI axis.
Follow-up 2. "How do you decide the blast radius?" — probes blast-radius axis.
Follow-up 3. "What if the experiment breaks production?" — probes halt-condition axis.
Follow-up 4. "Where do you run this — dev, staging, or prod?" — probes graduation-of-trust axis.
Follow-up 5. "How is this different from load testing?" — probes methodology understanding.

Question. Draft a 5-minute senior chaos engineering answer that covers all axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Framework named	"we run chaos scripts"	"steady state → hypothesis → experiment → verify"
Steady state	"everything green in Grafana"	"freshness_p99_min < 5 and lag < 10k for 15 min"
Blast radius	"one broker"	"1 of 6 brokers in stage; 1 in prod only after 3 passing stage runs"
Halt condition	"we watch it"	"promProbe on freshness > 15 min stops the experiment"
Graduation	"prod eventually"	"dev → staging → 1% prod → 10% prod → 100% prod, one week between steps"
vs load testing	"it's like load testing"	"load tests break capacity; chaos breaks unavailability"

Code.

Senior chaos engineering answer template (5 minutes)
====================================================

Minute 1 — name the framework up front
  "I run chaos as a four-step loop: define the steady state as a
   data SLO, state a hypothesis about how the pipeline should
   behave under a specific failure, run the experiment inside a
   bounded blast radius with automatic halt on SLO breach, and
   verify by comparing SLI curves before/during/after."

Minute 2 — steady state
  "For our Kafka → Spark → Iceberg stack the steady state is:
   freshness_p99_min < 5, kafka_consumergroup_lag < 10000,
   batch_success_rate > 0.99, row_count_delta < 100 over 1 hour.
   Those four Prometheus queries are the SLI definition."

Minute 3 — hypothesis + experiment
  "Hypothesis: 'if we kill 1 of 6 Kafka brokers, freshness stays
   under 15 min and lag under 100k.' Experiment: LitmusChaos
   pod-delete on a broker labelled kafka-broker with
   PODS_AFFECTED_PERC=17, TOTAL_CHAOS_DURATION=60s, promProbes
   on all four SLIs, mode=Continuous, stopOnFailure=true."

Minute 4 — halt + blast radius
  "The blast radius is one broker out of six. Halt condition:
   any promProbe reads outside its threshold. Graduation:
   3 passing runs in staging over one week before promoting to
   a 1%-of-prod broker set, then 10%, then 100%. We never
   start with a bigger blast radius than we've already passed."

Minute 5 — verify + post-mortem
  "Verify: overlay before/during/after SLI curves in Grafana,
   write a GameDay doc with the hypothesis, actual outcome,
   probe timeline, and any backlog items. If the hypothesis
   failed, the backlog item is the fix; if it passed, the
   backlog item is the next larger blast radius."

Step-by-step explanation.

Minute 1 names the four-step loop. This is the Netflix Principles-of-Chaos / PoP framework in one sentence; naming it up front signals you know the discipline exists and isn't just "kill some pods."
Minute 2 states the steady state as four concrete Prometheus queries. This is the hardest thing for junior engineers to do — they'll say "the pipeline is healthy" but not codify what that means. Concrete numeric thresholds are senior signal.
Minute 3 fuses hypothesis + experiment. The hypothesis is a falsifiable prediction ("stays under 15 min lag"), not vague ("should be fine"). The experiment is the smallest fault that tests the hypothesis; anything bigger is a shotgun.
Minute 4 covers halt + blast radius — the safety controls. Naming stopOnFailure=true and the graduation ladder (dev → stage → 1% prod → 10% → 100%) shows you understand that chaos is a controlled experiment, not a demolition.
Minute 5 covers verify + post-mortem — the learning step. This is what separates chaos engineering from chaos monkey: every experiment produces either "resilience confirmed, graduate the blast radius" or "resilience broken, here's the backlog item."

Output.

Grading criterion	Weak score	Senior score
Names the four-step loop in minute 1	rare	mandatory
Steady state as SLI queries	occasional	mandatory
Hypothesis is falsifiable	rare	required
Halt condition + blast radius	rare	senior signal
Verify + post-mortem + backlog	rare	senior signal

Rule of thumb. The senior chaos answer is a 5-minute monologue that walks the steady-state → hypothesis → experiment → verify loop with concrete SLI thresholds and a graduation ladder. Rehearse it once; deploy it every time.

Senior interview question on chaos engineering for data pipelines

A senior interviewer often opens with: "Your team owns a Kafka + Spark Structured Streaming + Iceberg pipeline that promises 5-minute freshness to the warehouse. Design a chaos engineering program that proves the pipeline meets that SLO under broker crash, executor loss, and S3 latency spikes. Walk me through the steady-state definition, the hypothesis for each experiment, the blast-radius controls, the halt conditions, and how you graduate from staging to production."

Solution Using an SLI-first chaos program with graduated blast radius and halt-on-probe-failure

# 1. Prometheus recording rules — the steady-state SLI catalog
groups:
  - name: pipeline-steady-state
    interval: 30s
    rules:
      - record: pipeline:freshness_p99_min
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(iceberg_commit_age_seconds_bucket{table="events"}[5m]))
          ) / 60
      - record: pipeline:kafka_lag_messages
        expr: sum(kafka_consumergroup_lag{group="warehouse-sink"})
      - record: pipeline:batch_success_rate_5m
        expr: |
          sum(rate(spark_streaming_batch_success_total[5m]))
          / clamp_min(sum(rate(spark_streaming_batch_total[5m])), 1)
      - record: pipeline:row_count_delta_1h
        expr: |
          (source_events_1h_total - iceberg_rows_1h_total{table="events"})

# 2. Three chaos experiments, one file, shared halt conditions
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pipeline-chaos-suite
  namespace: chaos
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '120'
            - name: PODS_AFFECTED_PERC
              value: '17'                # 1 of 6 brokers
            - name: TARGET_PODS
              value: 'app=kafka-broker'
        probe: &shared_probes
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:kafka_lag_messages'
              comparator: {type: int, criteria: '<', value: '100000'}

    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '120'
            - name: PODS_AFFECTED_PERC
              value: '10'                # 1 of ~10 executors
            - name: TARGET_PODS
              value: 'app=spark-executor'
        probe: *shared_probes

    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: NETWORK_LATENCY
              value: '2000'              # +2s to S3 calls
            - name: DESTINATION_HOSTS
              value: 's3.us-east-1.amazonaws.com'
            - name: TARGET_PODS
              value: 'app=iceberg-writer'
        probe: *shared_probes

# 3. GitOps promotion — kustomize overlay per blast radius stage
# overlays/staging/kustomization.yaml
resources: [../../base]
patches:
  - target: {kind: ChaosEngine, name: pipeline-chaos-suite}
    patch: |-
      - op: replace
        path: /spec/appinfo/appns
        value: staging
      - op: replace
        path: /spec/experiments/0/spec/components/env
        value:
          - {name: PODS_AFFECTED_PERC, value: '17'}   # 1/6 brokers in stage
---
# overlays/prod-1pct/kustomization.yaml — promoted only after 3 passing stage runs
resources: [../../base]
patches:
  - target: {kind: ChaosEngine, name: pipeline-chaos-suite}
    patch: |-
      - op: replace
        path: /spec/appinfo/appns
        value: prod
      - op: replace
        path: /spec/experiments/0/spec/components/env
        value:
          - {name: PODS_AFFECTED_PERC, value: '17'}
          - {name: NODE_SELECTOR, value: 'blast=1pct'}

Step-by-step trace.

Step	Before (no chaos program)	After (SLI-first chaos suite)
Freshness SLO measurement	ad-hoc dashboards	4 recording rules; single-query steady state
Failure discovery	production incidents	scheduled staging chaos + game days
Halt behaviour	on-call reacts manually	promProbe auto-aborts on SLI breach
Blast radius control	none	PODS_AFFECTED_PERC per experiment
Graduation	none	dev → staging → 1% prod → 10% → 100% via kustomize overlays
Post-mortem	only after outages	after every experiment, pass or fail
Backlog outcome	vague "resilience" tickets	concrete fix or "graduate radius" ticket

After deployment, the chaos suite runs on a nightly cadence in staging, weekly in prod-1pct, monthly in prod-10pct. Every run writes a ChaosResult CR consumed by a Grafana dashboard that overlays the steady-state SLIs on the experiment timeline; deviations surface as either "resilience regression" alerts or "blast-radius graduation" tickets.

Output:

Metric	Before	After
Time-to-detect resilience regression	production incident (~hours)	staging chaos run (~30 min)
Halt latency on SLO breach	operator-driven (~minutes)	promProbe (~10 s)
Blast radius audit trail	none	ChaosResult CR + GameDay doc
Production chaos exposure	none or too much	1% → 10% → 100% graduation
Post-mortem coverage	outage-driven	every experiment

Why this works — concept by concept:

Steady state as SLI — the pipeline is "healthy" iff the four recording rules stay within their thresholds. This turns "how do I know if chaos broke things" from a debate into a Prometheus query. The four SLIs collectively cover freshness (age of the latest data), lag (queue depth), success rate (task health), and correctness (source-vs-sink parity) — the four dimensions of data-pipeline health.
Hypothesis-driven experiment — every ChaosEngine encodes a falsifiable prediction ("if I kill 1 of 6 brokers, freshness stays < 15 min"). Falsifiability is the difference between science and stunt; without a stated hypothesis, "we ran chaos" has no verdict.
Bounded blast radius — PODS_AFFECTED_PERC=17 (1 of 6 brokers) is the explicit dial. Every experiment starts with the smallest blast that still tests the hypothesis; graduation is an earned promotion, not a default.
Halt on probe failure — stopOnFailure: true on the promProbe gives you an automatic seatbelt. The experiment aborts the moment the SLI breaches; you don't need a human watching the dashboard.
GitOps promotion via kustomize overlays — the same base ChaosEngine ships to dev / staging / prod-1pct / prod-10pct via kustomize overlays. This is the audit trail — the CR you ran in prod is provably the same one that passed in staging, differing only in blast-radius and namespace.
Cost — one Prometheus (already deployed), LitmusChaos control plane (~1 pod, cluster-scoped), 4 recording rules (~ms/scrape), N ChaosEngine CRs per pipeline (~KB each). Compared to unmanaged production incidents, this is trading a small ops overhead for measurable resilience data. O(1) per experiment; O(scheduled cadence) per pipeline.

Design
Topic — design
Design problems on resilient data pipelines

Practice →

Streaming Topic — streaming Streaming resilience and back-pressure problems

Practice →

2. LitmusChaos on Kubernetes — pod / network / node kill experiments

`ChaosEngine` CR + probe guardrails — the K8s-native chaos framework built for GitOps

The mental model in one line: litmuschaos runs chaos experiments as Kubernetes custom resources (ChaosEngine, ChaosExperiment, ChaosResult) that target labelled workloads via a service account, execute a fault from a versioned experiment library (pod-delete, pod-network-loss, node-taint, disk-fill, pod-cpu-hog, pod-memory-hog, and dozens more), and gate the experiment behind probes (httpProbe, promProbe, cmdProbe, k8sProbe) that auto-abort on breach — the entire pattern is GitOps-friendly, CNCF-graduated, and free. Every senior K8s-native data platform team has evaluated LitmusChaos; most have shipped it because the CR-driven model composes naturally with existing GitOps + RBAC + Prometheus tooling.

The four axes for LitmusChaos.

Deployment surface. Native Kubernetes. Runs as a Helm-installable control plane (litmus-chaos-operator) plus a per-namespace litmus-admin service account with RBAC scoped to the target workloads. No SaaS control plane, no external agents.
Latency. Chaos starts within seconds of applying the ChaosEngine CR. Probe evaluation is per-probe interval (typically 10 s). Auto-abort fires within one probe interval after breach.
Fault library. ~60 pre-built experiments across pod, node, network, disk, CPU, memory, and cloud (AWS EC2, EBS, RDS, Azure VM, GCP VM). Custom experiments are defined as ChartServiceVersion CRs — you can ship your own fault.
Cost. Free (Apache-2.0). Optional paid observability + control plane via chaos-center. Operational cost is one operator pod per cluster plus one chaos-runner pod per active experiment.

The four probe kinds — the guardrails that make chaos safe.

httpProbe. Hits an HTTP endpoint on the target or a nearby service; checks response body / status / latency. Use for "is my /health endpoint returning 200?"
promProbe. Runs a PromQL query against Prometheus; compares the result against a threshold. Use for every data SLI — freshness, lag, success rate, correctness.
cmdProbe. Executes a shell command inside a chaos-runner pod (or a sidecar); checks the exit code or output. Use for "kafka-console-consumer sees a message" style checks.
k8sProbe. Reads a Kubernetes API object; compares its status field. Use for "is my StatefulSet Ready", "is my PVC Bound", "is my Deployment at the expected replica count."

Probe modes — when in the chaos lifecycle the probe runs.

SOT (start-of-test). Once, before chaos begins. Used to validate the pre-condition — "is the system healthy enough to inject chaos?"
EOT (end-of-test). Once, after chaos ends. Used to validate the recovery — "did the pipeline catch up within N minutes?"
Edge. Both SOT and EOT.
Continuous. Every N seconds throughout the chaos window. Used with stopOnFailure: true to enforce halt conditions.
OnChaos. Only during the chaos period (skips before/after). Used for "verify a specific behaviour appears while the fault is active."

Common interview probes on LitmusChaos.

"How does LitmusChaos abort an experiment?" — required answer: probe stopOnFailure: true + mode: Continuous; the operator writes a ChaosResult verdict of Fail.
"How do you scope which pods get killed?" — applabel selector + PODS_AFFECTED_PERC; both are enforced by the operator.
"How do you version chaos experiments?" — ChartServiceVersion CRs in a Git repo; chaos-hub is the community index.
"How is LitmusChaos different from chaos-mesh?" — LitmusChaos is CR + probe + GitOps; chaos-mesh is CR + Web UI + dashboard. Both are CNCF; pick based on team's GitOps preference.

Worked example — `pod-delete` on a Kafka broker with a `promProbe` guardrail

Detailed explanation. The canonical LitmusChaos data-pipeline experiment: delete a Kafka broker pod, watch consumer lag stay under a threshold, verify the broker rejoins the ISR within a bounded time. Build the ChaosEngine, install the experiment chart, and apply it against a stage cluster.

Target. A StatefulSet/kafka-broker in the kafka namespace with 6 replicas (kafka-broker-0 through kafka-broker-5).
Fault. pod-delete from the generic chart on chaos-hub.
Guardrail. promProbe on max(kafka_consumergroup_lag) with a < 50000 threshold.
Blast radius. 1 pod at a time (PODS_AFFECTED_PERC=17).

Question. Write the ChaosEngine CR, the RBAC for the chaos service account, and the kubectl apply that runs the experiment.

Input.

Object	Purpose
`ServiceAccount litmus-admin`	scoped identity for the chaos runner
`Role/RoleBinding`	RBAC on pods, chaosengines, chaosresults
`ChaosExperiment pod-delete`	fault library chart (installed once per namespace)
`ChaosEngine kafka-broker-chaos`	the actual experiment definition
`promProbe consumer-lag-guard`	halt on lag > 50k

Code.

# 1. RBAC — the chaos runner needs to delete pods in the kafka namespace
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litmus-admin
  namespace: kafka
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: litmus-admin
  namespace: kafka
rules:
  - apiGroups: [""]
    resources: [pods, events, configmaps, secrets, services]
    verbs: [get, list, watch, create, delete, update, patch, deletecollection]
  - apiGroups: [apps]
    resources: [statefulsets, deployments, replicasets]
    verbs: [get, list, watch]
  - apiGroups: [litmuschaos.io]
    resources: [chaosengines, chaosexperiments, chaosresults]
    verbs: [get, list, create, update, patch, delete]
  - apiGroups: [batch]
    resources: [jobs]
    verbs: [get, list, create, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: litmus-admin
  namespace: kafka
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: litmus-admin}
subjects: [{kind: ServiceAccount, name: litmus-admin, namespace: kafka}]

---
# 2. Experiment chart — installed once per namespace
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
  namespace: kafka
spec:
  definition:
    scope: Namespaced
    permissions: [{apiGroups: [""], resources: [pods], verbs: [create, delete, get, list, patch, update]}]
    image: litmuschaos/go-runner:3.10.0
    imagePullPolicy: Always
    args: [-c, ./experiments -name pod-delete]
    command: [/bin/bash]
    env:
      - {name: TOTAL_CHAOS_DURATION, value: '60'}
      - {name: RAMP_TIME, value: ''}
      - {name: FORCE, value: 'false'}
      - {name: CHAOS_INTERVAL, value: '30'}
      - {name: PODS_AFFECTED_PERC, value: ''}
      - {name: TARGET_CONTAINER, value: ''}

---
# 3. ChaosEngine — the experiment execution
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-chaos
  namespace: kafka
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '60'}
            - {name: CHAOS_INTERVAL, value: '30'}
            - {name: FORCE, value: 'false'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 10
              retry: 2
              probePollingInterval: 10
              stopOnFailure: true
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator: {type: int, criteria: '<', value: '50000'}

# 4. Apply and watch
kubectl apply -f rbac.yaml -f experiment.yaml -f chaosengine.yaml

# 5. Watch the ChaosResult verdict
kubectl -n kafka get chaosresult kafka-broker-chaos-pod-delete -w
# → phase: Running          → phase: Completed
# → verdict: (blank)         → verdict: Pass | Fail

# 6. Inspect the probe timeline
kubectl -n kafka describe chaosresult kafka-broker-chaos-pod-delete

Step-by-step explanation.

The RBAC block creates a litmus-admin service account in the kafka namespace with just-enough permission: delete pods, read statefulsets, manage its own chaos CRs, create runner jobs. Nothing cluster-wide, nothing outside kafka.
The ChaosExperiment chart is installed once per namespace. It declares the fault image (litmuschaos/go-runner), the entry-point, and the default env vars. Think of it as a Helm chart for a chaos primitive.
The ChaosEngine is the actual experiment execution. appinfo scopes the fault to pods labelled app=kafka-broker (which selects kafka-broker-0 through kafka-broker-5 in the StatefulSet). PODS_AFFECTED_PERC=17 = ceil(6 × 0.17) = 1 pod.
The promProbe runs max(kafka_consumergroup_lag{group="warehouse-sink"}) against Prometheus every 10 s. If the query returns anything not < 50000, stopOnFailure: true halts the experiment, the chaos-runner exits, and the ChaosResult records a Fail verdict with the probe timeline.
After the experiment completes, kubectl describe chaosresult shows the phase (Completed), verdict (Pass / Fail), and each probe's evaluation history. This is the CI-friendly artifact you check into GitOps or feed into your GameDay dashboard.

Output.

Time (s)	Event	Probe: consumer-lag	Verdict
0	ChaosEngine applied	800 (< 50k)	Pass
5	chaos-runner spawned	900	Pass
10	kafka-broker-3 deleted	1200	Pass
20	leader re-election	4800	Pass
30	kafka-broker-3 restarted	12000	Pass
45	broker rejoined ISR	8000	Pass
60	chaos window closed	3200	Pass (final)

Rule of thumb. For any Kubernetes-native data platform, model chaos as ChaosEngine CRs alongside your workload manifests. RBAC-scope the runner to the target namespace, ship the CR through GitOps, and gate every experiment behind a promProbe bound to a data SLI.

Worked example — `pod-network-loss` between Airflow scheduler and metadata DB

Detailed explanation. A subtler chaos experiment: introduce packet loss between the Airflow scheduler pod and the metadata Postgres, verify the scheduler survives without corrupting DAG state, and confirm task instances resume cleanly. This is a stateful chaos scenario — the scheduler is the sole writer to dag_run / task_instance tables and a partial write is much scarier than a clean pod kill.

Target. Deployment/airflow-scheduler, 1 pod.
Fault. pod-network-loss — introduces packet loss on the pod's network interface to a specific destination.
Destination. The Airflow metadata Postgres service (postgres-airflow.airflow.svc).
Guardrail. k8sProbe on the scheduler Deployment ready status + cmdProbe that runs airflow db check.

Question. Write the ChaosEngine that injects 50% packet loss to Postgres for 60 s and verify the scheduler recovers without a corrupted metadata write.

Input.

Setting	Value
Fault	pod-network-loss
NETWORK_PACKET_LOSS_PERCENTAGE	50
DESTINATION_HOSTS	postgres-airflow.airflow.svc
DESTINATION_PORTS	5432
TOTAL_CHAOS_DURATION	60
Post-chaos verification	airflow db check + no `stale` `task_instance` rows

Code.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-netloss
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'app=airflow-scheduler'
    appkind: 'deployment'
  experiments:
    - name: pod-network-loss
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '60'}
            - {name: NETWORK_PACKET_LOSS_PERCENTAGE, value: '50'}
            - {name: DESTINATION_HOSTS, value: 'postgres-airflow.airflow.svc'}
            - {name: DESTINATION_PORTS, value: '5432'}
            - {name: CONTAINER_RUNTIME, value: 'containerd'}
            - {name: SOCKET_PATH, value: '/run/containerd/containerd.sock'}
        probe:
          - name: scheduler-ready-guard
            type: k8sProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 2, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'
          - name: metadata-consistency-check
            type: cmdProbe
            mode: EOT
            runProperties: {probeTimeout: 20, interval: 5, retry: 1, stopOnFailure: true}
            cmdProbe/inputs:
              command: |
                psql -h postgres-airflow.airflow.svc -U airflow -d airflow \
                     -tAc "SELECT count(*) FROM task_instance WHERE state='running' AND queued_dttm < NOW() - INTERVAL '10 minutes'"
              comparator: {type: int, criteria: '=', value: '0'}
              source:
                image: 'postgres:16-alpine'
                imagePullPolicy: IfNotPresent
                inheritInputs: true

Step-by-step explanation.

pod-network-loss uses tc netem (kernel traffic control) inside the target pod's network namespace to drop 50% of packets destined for postgres-airflow.airflow.svc:5432. The fault is bounded by IP + port so only Postgres traffic is affected; the scheduler can still reach the k8s API, the DAGs Git-sync sidecar, etc.
The k8sProbe runs continuously and checks the Airflow scheduler Deployment stays in a valid state. If the Deployment goes to zero ready replicas (i.e. the scheduler crashed and can't restart), stopOnFailure aborts the experiment.
The cmdProbe runs at end-of-test, launching a temporary postgres:16-alpine pod that connects to the Airflow metadata DB and asks: "are there any task_instances in state 'running' with a queue time older than 10 minutes?" If the answer is not zero, we have stuck rows — the network loss corrupted a state transition and the experiment fails.
The cmdProbe pattern is powerful because it lets you assert data-shape invariants that Prometheus can't natively express. "No stuck task instances" is a domain-specific invariant of Airflow's operational state; a PromQL query can approximate it but the cmdProbe gives you the ground-truth SQL.
The fault duration (60 s) is deliberately short — enough to force Postgres retries but not so long that scheduled DAG runs will start missing SLAs. Longer network-loss experiments should have PODS_AFFECTED_PERC reduced accordingly and require the game-day escalation.

Output.

Phase	Event	Probe verdict
Setup	ChaosEngine applied	—
0-60 s	50% packet loss to Postgres	scheduler-ready-guard: Pass (deployment still Ready)
60-70 s	packet loss removed; connection retries drain	—
EOT	`SELECT count(*) FROM task_instance WHERE stuck`	metadata-consistency-check: Pass (0 stuck)
Result	ChaosResult verdict	Pass

Rule of thumb. For stateful-writer chaos (scheduler, metadata DB, coordinator), pair a k8sProbe (liveness during chaos) with an EOT cmdProbe (data consistency after chaos). One catches the crash, the other catches the corruption.

Worked example — `disk-fill` on the Airflow scheduler pod

Detailed explanation. A resource-exhaustion chaos experiment: fill the Airflow scheduler pod's ephemeral disk to 95% and observe how the scheduler handles the disk pressure. Kubernetes may evict the pod under DiskPressure, the scheduler may fail to write logs, or the dags/ folder Git-sync sidecar may fail to pull new DAGs. Every one of these is a real production failure mode; making it a scheduled chaos experiment turns it from a surprise into a learning.

Target. Deployment/airflow-scheduler, 1 pod.
Fault. disk-fill — writes a large file to the pod's ephemeral storage.
Fill target. 95% of the pod's ephemeral-storage limit.
Guardrail. promProbe on pipeline:batch_success_rate_5m + k8sProbe on scheduler deployment.

Question. Write the ChaosEngine that fills the scheduler's ephemeral disk to 95% for 90 s and confirm DAG success rate stays above 90%.

Input.

Setting	Value
Fault	disk-fill
FILL_PERCENTAGE	95
EPHEMERAL_STORAGE_MEBIBYTES	4096
TOTAL_CHAOS_DURATION	90
Halt SLI	batch_success_rate_5m > 0.90

Code.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-diskfill
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'app=airflow-scheduler'
    appkind: 'deployment'
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '90'}
            - {name: FILL_PERCENTAGE, value: '95'}
            - {name: EPHEMERAL_STORAGE_MEBIBYTES, value: '4096'}
            - {name: CONTAINER_RUNTIME, value: 'containerd'}
            - {name: SOCKET_PATH, value: '/run/containerd/containerd.sock'}
            - {name: DATA_BLOCK_SIZE, value: '256'}
        probe:
          - name: batch-success-rate-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:batch_success_rate_5m'
              comparator: {type: float, criteria: '>', value: '0.90'}
          - name: scheduler-alive-guard
            type: k8sProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 2, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'

Step-by-step explanation.

disk-fill writes a chaos-fill file inside the target pod's writable filesystem using dd if=/dev/zero of=... bs=... count=.... It writes until the pod's ephemeral-storage usage hits FILL_PERCENTAGE (95%) of the declared limit.
If the pod has an ephemeral-storage request/limit set in its spec, Kubernetes may trigger DiskPressure eviction. If not, the pod's writable layer just fills up and any component that writes (Airflow's log rotator, the SQLite session store, the Git-sync sidecar) starts erroring. Both failure modes are interesting.
The promProbe on batch_success_rate_5m catches the case where the scheduler survives but DAG runs start failing (e.g. because the scheduler can no longer write task logs). The k8sProbe catches the eviction case.
DATA_BLOCK_SIZE=256 tunes the dd block size for reasonable I/O behaviour — small enough not to lock the disk, large enough to fill quickly.
The 90-second window is long enough for the disk-pressure symptom to propagate through the scheduler's next-heartbeat, DAG-parse, and task-launch cycles. Shorter windows may not exercise the full failure envelope.

Output.

Phase	Event	Probe: success rate	Probe: scheduler alive
0 s	Chaos applied	0.99	Pass
30 s	Disk 90% full	0.98	Pass
60 s	Disk 95% full; log writes failing	0.95	Pass
90 s	Chaos removed; disk freed	0.96	Pass
Post	Log-writer catches up	0.99	Pass (final)

Rule of thumb. Every stateful pod deserves a disk-fill chaos test. The failure modes are legion (log rotator crashes, sqlite locks, git-sync stalls, eviction storms) and every one of them surfaces only under actual disk pressure — no unit test finds them.

Senior interview question on LitmusChaos

A senior interviewer might ask: "You've adopted LitmusChaos as the K8s-native chaos framework for your data platform. Design the end-to-end pattern for shipping a new chaos experiment — from a Prometheus SLI definition through the ChaosEngine CR, the RBAC, the GitOps promotion, and the ChaosResult post-processing that lands in the on-call GameDay doc. Include how you'd version experiments and how you'd deprecate a broken one."

Solution Using a GitOps chaos pipeline with SLI-first probes and PR-gated promotion

# repo layout
# chaos/
#   base/
#     _slis.yaml               # Prometheus recording rules (shared)
#     _rbac.yaml               # litmus-admin ServiceAccount + Role
#     experiments/
#       pod-delete.yaml        # ChaosExperiment chart (installed once)
#       pod-network-loss.yaml
#       disk-fill.yaml
#     engines/
#       kafka-broker-kill.yaml
#       airflow-net-loss.yaml
#       spark-executor-kill.yaml
#   overlays/
#     dev/kustomization.yaml
#     staging/kustomization.yaml
#     prod-1pct/kustomization.yaml
#     prod-10pct/kustomization.yaml
#     prod-100pct/kustomization.yaml

# base/engines/kafka-broker-kill.yaml — the source of truth
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-kill
  labels:
    chaos.pipecode.ai/owner: 'data-platform-sre'
    chaos.pipecode.ai/tier: 'staging'
    chaos.pipecode.ai/version: 'v3'
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
        probe:
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}

# post_chaos.py — a small daemon that watches ChaosResult CRs and pipes to Slack + Jira
import subprocess, json, time, requests

SLACK_URL = "https://hooks.slack.com/services/xxx"
JIRA_URL  = "https://acme.atlassian.net/rest/api/2/issue"

def watch_chaosresults():
    proc = subprocess.Popen(
        ["kubectl", "get", "chaosresults", "-A", "-w", "-o", "json"],
        stdout=subprocess.PIPE, bufsize=1, text=True,
    )
    for line in proc.stdout:
        try:
            evt = json.loads(line)
        except json.JSONDecodeError:
            continue

        name  = evt["metadata"]["name"]
        phase = evt.get("status", {}).get("experimentStatus", {}).get("phase")
        verdict = evt.get("status", {}).get("experimentStatus", {}).get("verdict", "")
        probe_status = evt.get("status", {}).get("probeStatuses", [])

        if phase != "Completed":
            continue

        if verdict == "Pass":
            requests.post(SLACK_URL, json={
                "text": f":white_check_mark: chaos {name} passed. Consider graduating blast radius."
            })
        else:
            requests.post(SLACK_URL, json={
                "text": f":rotating_light: chaos {name} FAILED. Probe timeline: {probe_status}"
            })
            requests.post(JIRA_URL, json={
                "fields": {
                    "project": {"key": "DATA"},
                    "summary": f"Chaos regression: {name}",
                    "description": json.dumps(probe_status, indent=2),
                    "issuetype": {"name": "Bug"},
                    "labels": ["chaos-regression"],
                }
            }, auth=("bot@acme.com", "***"))

if __name__ == "__main__":
    while True:
        try:
            watch_chaosresults()
        except Exception as e:
            print("watch died:", e)
            time.sleep(30)

Step-by-step trace.

Step	Component	Action
1	Engineer opens PR	edit `base/engines/kafka-broker-kill.yaml` — bump `PODS_AFFECTED_PERC`
2	CI	kustomize build; kubeconform schema check; OPA policy check
3	Argo CD	sync PR branch to `dev` cluster; nightly chaos run
4	ChaosResult	verdict recorded; posted to Slack + optional Jira
5	PR gets 3 green nightly runs	reviewer approves promotion to `staging` overlay
6	Same PR extended	`overlays/prod-1pct/kustomization.yaml` bumped
7	Argo CD	sync to prod-1pct cluster; monthly chaos run during business hours
8	3 green monthly runs	promote to prod-10pct
9	Deprecation	old engine `engineState: 'stop'` + PR labelled `chaos-retirement`

After deployment, every chaos experiment flows through a PR → dev → staging → prod-1pct → prod-10pct → prod-100pct GitOps pipeline. The ChaosResult CR is the receipt; the Slack + Jira integration is the human interface; the RBAC is the safety boundary. New experiments cannot skip stages; failed experiments cannot self-graduate.

Output:

Artifact	Purpose	Consumer
`ChaosEngine` CR in Git	source of truth for the experiment	Argo CD / reviewer / audit
`ChaosResult` CR in cluster	verdict + probe timeline	Slack bot / Grafana / GameDay doc
kustomize overlays	blast-radius graduation ladder	PR reviewer / audit
Slack alert	live notification	on-call rotation
Jira ticket	failure follow-up	data-platform-sre backlog

Why this works — concept by concept:

ChaosEngine as CR — the experiment is a Kubernetes object, versionable, reviewable, RBAC-scoped. Not a Terraform script, not a Jenkins job — a first-class cluster object. The GitOps toolchain treats it the same as any Deployment.
Shared SLI recording rules — a single _slis.yaml file defines the four data SLIs; every experiment's probes reference pipeline:* metrics. New SLIs land in one place; new experiments reuse them for free.
RBAC-scoped runner — litmus-admin has permission to delete pods only in the target namespace, and only for chaos CRs it created. A compromised chaos runner cannot escalate; the blast radius is bounded by the RBAC.
kustomize overlays as the graduation ladder — the same base CR ships to five stages differing only in PODS_AFFECTED_PERC, node selectors, and namespace. The diff between dev and prod-100pct is auditable in a single git diff output.
PR-gated promotion — you cannot promote an experiment to a bigger blast radius without three green runs at the previous stage and a peer-reviewed PR. This is the human safety net that makes chaos in production defensible.
ChaosResult post-processing — the verdict + probe timeline pipes to Slack (visibility) and Jira (accountability). Every failed experiment produces a ticket; every passed experiment produces a "consider graduating" signal.
Cost — one operator pod, one runner pod per active experiment, a few KB per CR, one Slack webhook, one Jira token. Compared to the incident-driven alternative (learn resilience during real outages), this is a 100× reduction in mean-time-to-learn. O(1) per experiment; O(N experiments × N stages) per pipeline.

Design
Topic — design
Design problems on Kubernetes-native reliability

Practice →

ETL Topic — etl ETL problems on scheduled DAG failure recovery

Practice →

3. Gremlin — scenarios, blast-radius controls, workflows

Attack primitives → Scenarios → Halt-on-breach — the SaaS control plane for cross-platform chaos

The mental model in one line: gremlin is a commercial SaaS chaos platform whose control plane runs in the vendor's cloud and whose lightweight agent (gremlind) runs on every VM, container, ECS/Fargate task, or Kubernetes node you want to attack — it exposes low-level attack primitives (CPU, memory, disk, IO, latency, packet-loss, blackhole, DNS, shutdown, time-travel) that stitch into higher-level Scenarios with explicit blast-radius sliders and Halt-on-Breach Health Checks tied to your observability platform (Datadog, New Relic, Prometheus, custom webhooks). Every senior data platform team evaluating chaos tooling faces the LitmusChaos-vs-Gremlin decision; the deciding factor is almost always deployment surface (K8s-only vs cross-platform) and audit posture (self-hosted OSS vs SaaS with SOC-2).

The four axes for Gremlin.

Deployment surface. Cross-platform. Runs on Linux, Windows, container, K8s, ECS, Fargate, on-prem, EC2, GCE. The gremlind agent is a single ~50MB Go binary that phones home to the SaaS control plane; attacks are triggered from the web UI, Terraform provider, or API.
Fault library. ~15 attack primitives, richer than LitmusChaos on node-level and network-level primitives (DNS, packet-loss, time-travel, IO throttle). Weaker on Kubernetes-object-aware faults (no direct k8sProbe analog).
Blast-radius model. Explicit sliders: "attack N clients matching tag=X." Multi-dimensional selectors: tag:cluster=prod, tag:service=kafka, container:image=confluent-kafka. Percentage sliders per selector.
Halt behaviour. "Halt-on-Breach" scenarios — a scenario references a Datadog / New Relic / Prometheus / webhook Health Check; the check runs on a cadence; if any check reports unhealthy the scenario aborts and the attack rolls back.

Attack primitives — the low-level faults Gremlin ships.

Resource. cpu (peg N cores at X%), memory (allocate X GB), disk (fill X% of a volume), io (throttle read/write bandwidth).
Network. latency (add X ms to selected traffic), packet-loss (drop X% of packets), blackhole (drop all packets to/from a destination), dns (return NXDOMAIN or spoofed IP), bandwidth (throttle Mbps).
State. shutdown (halt the host), process-killer (SIGTERM/SIGKILL specific PIDs/regex), time-travel (shift the system clock forward/backward).

Scenarios — the workflow layer.

Definition. A Scenario is an ordered list of Steps. Each Step is either an Attack (an attack primitive with parameters and blast radius) or a Delay. Scenarios can also declare Health Checks that run continuously.
Halt-on-Breach. If any Health Check reports unhealthy during scenario execution, all remaining Steps are cancelled and any active attack is halted (reverting tc netem, freeing memory, releasing CPU pegs). This is Gremlin's equivalent of LitmusChaos stopOnFailure.
Attack rollback. Every attack primitive has an idempotent rollback path — killing the attack ID via API restores the pre-attack state. This is the safety property that makes production chaos possible.

Common interview probes on Gremlin.

"How does Gremlin differ from LitmusChaos?" — required answer: cross-platform SaaS with agents vs K8s-native OSS with CRs; Gremlin has richer network/state primitives, LitmusChaos has richer K8s-object probes.
"How do you halt a Gremlin attack on SLO breach?" — required answer: Halt-on-Breach Scenario with a Health Check tied to your observability platform.
"How do you audit who ran what chaos?" — Gremlin control plane logs every attack with user, timestamp, blast-radius, targets; export via API to your SIEM.
"How do you version Gremlin scenarios?" — Terraform gremlin_scenario resource or the REST API; check the JSON into Git alongside your infrastructure.

Worked example — `blackhole` attack on Spark executors with halt on SLA breach

Detailed explanation. A stateful chaos experiment: blackhole all network traffic on 10% of Spark executor nodes for 60 s, forcing shuffle-fetch failures on running Spark jobs. Verify the job SLA (batch_processing_seconds_p99) stays under threshold and the driver reprovisions executors. Model the experiment as a Gremlin Scenario with three steps: a latency warm-up, the main blackhole, and a Datadog Halt-on-Breach check.

Target. EC2 instances tagged service=spark-executor, env=stage.
Blast radius. 10% of matched clients.
Steps. 30 s of +200 ms latency to warm up, then 60 s of blackhole to a specific S3 endpoint, then 30 s of cooldown.
Halt. Datadog metric spark.batch.processing.p99 < 30s (per 1-min tumble).

Question. Write the Gremlin Scenario JSON, the Terraform resource that manages it, and the Datadog synthetic used as the Halt-on-Breach check.

Input.

Setting	Value
Target selector	`tag:service=spark-executor tag:env=stage`
Blast radius	10%
Step 1	latency +200 ms, 30 s
Step 2	blackhole s3.us-east-1.amazonaws.com, 60 s
Step 3	delay 30 s
Health check	Datadog: `spark.batch.processing.p99 < 30` for 5 min

Code.

// Gremlin Scenario — POST /v1/scenarios
{
  "name": "spark-executor-blackhole-stage",
  "description": "Blackhole S3 traffic on 10% of Spark executors; halt on p99 batch time breach.",
  "hypothesis": "If 10% of Spark executors lose S3 connectivity for 60s, batch processing p99 stays under 30s and the driver reprovisions.",
  "graph": {
    "nodes": {
      "start": {"type": "InfraAttack", "next": "warmup"},
      "warmup": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Latency",
          "attackType": "latency",
          "ms": 200,
          "length": 30,
          "targetType": "Random",
          "percent": 10,
          "targets": {
            "type": "Exact",
            "tags": {"service": "spark-executor", "env": "stage"}
          }
        },
        "next": "main"
      },
      "main": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Blackhole",
          "attackType": "blackhole",
          "hostnames": "s3.us-east-1.amazonaws.com",
          "length": 60,
          "targetType": "Random",
          "percent": 10,
          "targets": {
            "type": "Exact",
            "tags": {"service": "spark-executor", "env": "stage"}
          }
        },
        "next": "cooldown"
      },
      "cooldown": {"type": "Delay", "durationSeconds": 30, "next": null}
    }
  },
  "healthChecks": [
    {
      "id": "hc-spark-batch-p99",
      "haltOnUnhealthy": true,
      "provider": "datadog",
      "config": {
        "monitorId": 12345678,
        "state": "OK"
      }
    }
  ]
}

# Terraform — manages the same scenario as code
resource "gremlin_scenario" "spark_blackhole_stage" {
  name        = "spark-executor-blackhole-stage"
  description = "Blackhole S3 traffic on 10% of Spark executors; halt on p99 batch time breach."
  hypothesis  = "If 10% of Spark executors lose S3 connectivity for 60s, batch p99 stays < 30s."

  step {
    type = "latency"
    attack_definition = jsonencode({
      ms      = 200
      length  = 30
      percent = 10
      tags    = { service = "spark-executor", env = "stage" }
    })
  }

  step {
    type = "blackhole"
    attack_definition = jsonencode({
      hostnames = "s3.us-east-1.amazonaws.com"
      length    = 60
      percent   = 10
      tags      = { service = "spark-executor", env = "stage" }
    })
  }

  step { type = "delay"; duration_seconds = 30 }

  health_check {
    provider          = "datadog"
    monitor_id        = 12345678
    halt_on_unhealthy = true
  }
}

# Datadog monitor definition — the halt criterion
# monitor.yaml
type: metric alert
query: "avg(last_5m):p99:spark.batch.processing.seconds{env:stage} < 30"
message: |
  Spark batch processing p99 breached during chaos experiment.
  Gremlin will halt this scenario if this monitor goes OK -> Alert.
tags: [chaos, spark, stage]
options:
  notify_no_data: false
  evaluation_delay: 60

Step-by-step explanation.

The Scenario graph declares three ordered steps: warmup (latency), main (blackhole), cooldown (delay). Gremlin executes them in graph order; the next pointers describe the DAG.
Each InfraAttack node has an impactDefinition with targetType: Random and percent: 10, meaning Gremlin will randomly pick 10% of the clients matching the tag selector service=spark-executor env=stage. If you have 20 executors, 2 get blackholed.
The blackhole attack drops all IP traffic from the target host to s3.us-east-1.amazonaws.com (which resolves to a set of IPs; the attack tracks the resolution). Spark's S3A committer will retry with exponential back-off; after N failed retries, the executor's task fails and the driver reschedules it.
The healthChecks[] array references a Datadog monitor. Gremlin polls the monitor state every ~30 s; if it transitions from OK to Alert, haltOnUnhealthy: true causes the scenario to abort — all remaining steps cancelled, active attacks rolled back (blackhole removed, latency removed).
The Terraform resource wraps the same underlying REST API; checking the resource into Git gives you an audit trail (who bumped the percent from 5 → 10, when, with what justification). The Datadog monitor is defined outside the Gremlin resource because it's shared with other alerting; the Gremlin scenario only references its ID.

Output.

Time (s)	Step	Health check	Verdict
0	warmup start	OK (batch p99: 8s)	continue
30	main start (blackhole)	OK (batch p99: 12s)	continue
60	shuffle-fetch failures begin	OK (batch p99: 22s)	continue
75	driver reprovisioning	OK (batch p99: 28s)	continue
90	main end; blackhole rolled back	OK (batch p99: 18s)	continue
120	cooldown end	OK (batch p99: 10s)	scenario Passed

Rule of thumb. For cross-platform (mixed VM + container + K8s) data planes, model chaos as Gremlin Scenarios with explicit percent blast radius and a Datadog / New Relic Halt-on-Breach Health Check tied to a data SLI. The scenario graph is your chaos-as-code artifact; the halt behaviour is your safety net.

Worked example — `latency` attack on S3 for Iceberg writes

Detailed explanation. A subtler chaos experiment: add 500 ms of latency to all S3 API calls from the Iceberg writer nodes for 5 minutes. Verify the writer doesn't lose commits, doesn't corrupt metadata (Iceberg manifest atomicity), and the p99 commit latency stays within the SLO. Latency chaos surfaces retry storms, connection-pool exhaustion, and back-pressure paths that no functional test hits.

Target. EC2 instances tagged service=iceberg-writer, env=stage.
Blast radius. 25% of matched clients.
Attack. latency +500 ms to s3.us-east-1.amazonaws.com and s3.dualstack.us-east-1.amazonaws.com.
Duration. 5 minutes.
Halt. Prometheus: iceberg_commit_seconds_p99 < 30.

Question. Write the Gremlin Scenario, and add a Prometheus-backed Halt-on-Breach using a webhook health check.

Input.

Setting	Value
Attack	latency +500 ms
Destinations	s3.us-east-1.amazonaws.com, s3.dualstack.us-east-1.amazonaws.com
Duration	300 s
Blast radius	25% of `iceberg-writer` nodes
Halt	Prom: `iceberg_commit_seconds_p99 < 30` for 3 min

Code.

{
  "name": "iceberg-s3-latency-stage",
  "hypothesis": "If S3 gains 500ms of latency on 25% of writer nodes, commit p99 stays under 30s and no manifest corruption occurs.",
  "graph": {
    "nodes": {
      "attack": {
        "type": "InfraAttack",
        "impactDefinition": {
          "kind": "Latency",
          "attackType": "latency",
          "ms": 500,
          "hostnames": "s3.us-east-1.amazonaws.com,s3.dualstack.us-east-1.amazonaws.com",
          "length": 300,
          "targetType": "Random",
          "percent": 25,
          "targets": {
            "type": "Exact",
            "tags": {"service": "iceberg-writer", "env": "stage"}
          }
        },
        "next": null
      }
    }
  },
  "healthChecks": [
    {
      "id": "hc-iceberg-commit-p99",
      "haltOnUnhealthy": true,
      "provider": "webhook",
      "config": {
        "url": "https://chaos-webhooks.internal/gremlin/iceberg-commit-p99",
        "expectedStatus": 200,
        "pollIntervalSeconds": 30
      }
    }
  ]
}

# chaos-webhooks/iceberg-commit-p99 handler
# Small FastAPI service that turns a PromQL query into a webhook Gremlin can poll.
from fastapi import FastAPI, HTTPException
import httpx

app = FastAPI()

@app.get("/gremlin/iceberg-commit-p99")
async def iceberg_commit_p99():
    async with httpx.AsyncClient() as c:
        r = await c.get("http://prometheus.monitoring:9090/api/v1/query",
                        params={"query":
                                "histogram_quantile(0.99, sum by(le)("
                                "rate(iceberg_commit_seconds_bucket{env=\"stage\"}[3m])))"})
    val = float(r.json()["data"]["result"][0]["value"][1])
    if val > 30.0:
        # Report unhealthy — Gremlin halts the scenario
        raise HTTPException(status_code=503, detail=f"p99 {val:.2f} > 30s")
    return {"p99_seconds": val}

Step-by-step explanation.

The latency attack uses tc netem on the target host to add 500 ms delay to any packet destined for the listed hostnames. Both the standard and dualstack S3 endpoints are covered so IPv4/IPv6 traffic both see the injected latency.
targetType: Random, percent: 25 picks a random quarter of the matching hosts. If you have 8 iceberg-writer nodes, 2 get the latency injection; the other 6 write to S3 normally, so the pipeline as a whole should self-balance if writer selection is uniform.
The webhook health check is Gremlin's escape hatch when your observability platform isn't natively integrated. Any HTTP endpoint that returns 200 for "healthy" and non-200 for "unhealthy" works. Here we wrap a PromQL query into a FastAPI endpoint.
pollIntervalSeconds: 30 sets the cadence — Gremlin hits the webhook every 30 s. If the endpoint returns 503, Gremlin halts the scenario. The webhook itself is stateless and can be co-located with any Prometheus-adjacent service.
The 5-minute duration is chosen because Iceberg commit p99 is a lagging indicator — you need a few commit cycles for the latency to propagate into the SLI. Shorter windows may pass with false-negative "everything's fine" verdicts.

Output.

Time (min)	Attack	webhook status	Iceberg p99 (s)	Verdict
0	applied	200 (2.1)	2.1	continue
1	~2 writers slow	200 (6.4)	6.4	continue
2	retries exhaust pool	200 (14.9)	14.9	continue
3	commit p99 spike	200 (22.3)	22.3	continue
4	reprovisioning	200 (18.7)	18.7	continue
5	attack ended	200 (7.2)	7.2	Passed

Rule of thumb. For every external dependency (S3, DynamoDB, Snowflake, external HTTP) inject a latency chaos experiment. Latency chaos is the single highest-yield chaos primitive — it surfaces retry storms, connection-pool exhaustion, timeout misconfigurations, and back-pressure paths that no functional test finds.

Worked example — the blast-radius graduation matrix

Detailed explanation. Every Gremlin scenario is a candidate for graduation from dev → staging → 1% prod → 10% prod → 100% prod. The graduation matrix is the artifact that codifies the promotion rules: how many green runs at each stage before the next, who approves, what the rollback plan is. Walk through the matrix for the Kafka broker kill scenario.

Stage 1 (dev). Nightly automation, no on-call, no approval.
Stage 2 (staging). Weekly automation, tag-approved by data platform SRE.
Stage 3 (prod-1pct). Monthly ceremony, game-day format, senior SRE + platform lead required.
Stage 4 (prod-10pct). Quarterly, executive sponsor + full on-call bridge.
Stage 5 (prod-100pct). Reserved for annual DR drills; formal DR runbook + customer comms.

Question. Codify the graduation matrix as a decision table, and derive the CI check that gates promotion.

Input.

Stage	Cadence	Approvals	Green runs required to promote	Rollback plan
dev	nightly	automation	5	halt-on-breach
staging	weekly	1 SRE	3	halt-on-breach
prod-1pct	monthly	2 SREs + lead	3	halt + on-call bridge
prod-10pct	quarterly	lead + exec	2	halt + on-call bridge + customer comms
prod-100pct	annual (DR drill)	DR sponsor	N/A (single ceremony)	formal DR runbook

Code.

# ci/gate_chaos_promotion.py — enforces the graduation matrix on PR merge
import argparse, requests, sys

MATRIX = {
    "dev":         {"required_green": 5, "prev_stage": None},
    "staging":     {"required_green": 3, "prev_stage": "dev"},
    "prod-1pct":   {"required_green": 3, "prev_stage": "staging"},
    "prod-10pct":  {"required_green": 2, "prev_stage": "prod-1pct"},
    "prod-100pct": {"required_green": 1, "prev_stage": "prod-10pct"},
}

def gremlin_recent_runs(scenario_id, tag_env, n=10):
    r = requests.get(f"https://api.gremlin.com/v1/scenarios/{scenario_id}/runs",
                     params={"limit": n, "tag:env": tag_env},
                     headers={"Authorization": f"Bearer {TOKEN}"})
    return r.json()["runs"]

def can_promote(scenario_id, from_stage, to_stage):
    prev_required = MATRIX[to_stage]["required_green"]
    runs = gremlin_recent_runs(scenario_id, from_stage, n=prev_required)
    greens = [r for r in runs if r["state"] == "Successful"]
    if len(greens) < prev_required:
        return False, f"Only {len(greens)} green runs at {from_stage}; need {prev_required}"
    return True, f"OK: {len(greens)} green runs at {from_stage}"

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--scenario-id", required=True)
    p.add_argument("--to-stage", required=True, choices=MATRIX.keys())
    a = p.parse_args()
    prev = MATRIX[a.to_stage]["prev_stage"]
    if not prev:
        print("Stage has no prerequisite; auto-allow.")
        sys.exit(0)
    ok, msg = can_promote(a.scenario_id, prev, a.to_stage)
    print(msg)
    sys.exit(0 if ok else 1)

Step-by-step explanation.

The MATRIX codifies "how many green runs at stage N do you need before you can promote to stage N+1." Dev requires 5 green runs (nightly for a week); staging requires 3; prod-1pct requires 3; prod-10pct requires 2; prod-100pct is a single-ceremony DR drill.
gremlin_recent_runs calls the Gremlin API to fetch the last N runs of a scenario tagged with the previous stage's environment. The tag is your foreign key between the CI check and the historical run data.
The CI script is invoked from your PR pipeline: gate_chaos_promotion.py --scenario-id ... --to-stage prod-1pct. If insufficient green runs exist at the previous stage, the CI check fails and the PR cannot merge.
The design deliberately reads run history from Gremlin (source of truth) rather than tracking green-run counts in the repo. This prevents the "someone updated the counter without an actual run" failure mode.
Executive approval for prod-10pct is enforced via GitHub CODEOWNERS + branch protection — the platform/* codeowner (VP Data Platform) must approve any PR touching overlays/prod-10pct/. The matrix is codified in both machine-checkable (CI) and human-checkable (CODEOWNERS) forms.

Output.

PR event	Stage promotion	CI verdict	Human approval
Create engine	dev	auto-allow	1 SRE reviewer
Promote dev → staging	staging	required: 5 green dev runs	1 SRE reviewer
Promote staging → prod-1pct	prod-1pct	required: 3 green staging runs	2 SREs + team lead
Promote prod-1pct → prod-10pct	prod-10pct	required: 3 green prod-1pct runs	team lead + exec sponsor
Promote to prod-100pct	prod-100pct (DR)	required: 2 green prod-10pct runs	DR sponsor + formal runbook

Rule of thumb. Every chaos scenario in production had to earn its way there by passing N green runs at each prior stage. Codify the graduation matrix in both CI (machine-checkable) and CODEOWNERS (human-checkable). The matrix is the artifact your auditor will ask to see.

Senior interview question on Gremlin

A senior interviewer might ask: "You're bringing chaos engineering to a multi-cloud data platform (Kafka on-prem, Airflow on EKS, Spark on Databricks, Snowflake). Design the Gremlin-based chaos program: which primitives, which scenarios, which halt criteria, and how you manage the tag hierarchy so on-call knows what's being attacked at any moment."

Solution Using a tag-hierarchy-driven scenario library with SIEM-integrated audit trail

// 1. Tag hierarchy — every gremlind agent inherits these tags
// gremlind config at /etc/gremlin/config.yaml
{
  "team": "data-platform",
  "identifier": "auto",
  "tags": {
    "cluster": "kafka-onprem-1",
    "service": "kafka-broker",
    "env": "prod",
    "criticality": "tier-1",
    "chaos-eligible": "true",
    "chaos-tier": "1pct"
  }
}

# 2. Terraform module — reusable scenario definitions per stack
module "kafka_broker_kill_scenarios" {
  source = "./modules/gremlin-broker-kill"

  for_each = {
    "kafka-onprem-1-stage" = { env = "stage", percent = 17 }
    "kafka-onprem-1-1pct"  = { env = "prod", percent = 17, chaos_tier = "1pct" }
    "kafka-onprem-1-10pct" = { env = "prod", percent = 17, chaos_tier = "10pct" }
  }

  scenario_name = each.key
  env           = each.value.env
  percent       = each.value.percent
  chaos_tier    = try(each.value.chaos_tier, null)

  health_check_monitor_id = data.datadog_monitor.kafka_lag_slo.id
  halt_on_unhealthy       = true
}

# 3. SIEM exporter — mirror every Gremlin attack to Splunk for audit
# runs as a scheduled job every 5 minutes
import os, requests, json, hmac, hashlib
from datetime import datetime, timedelta, timezone

GREMLIN_TOKEN = os.environ["GREMLIN_TOKEN"]
SPLUNK_HEC    = os.environ["SPLUNK_HEC_URL"]
SPLUNK_HEC_TOKEN = os.environ["SPLUNK_HEC_TOKEN"]

def poll_gremlin_since(since_iso):
    r = requests.get("https://api.gremlin.com/v1/attacks",
                     params={"since": since_iso},
                     headers={"Authorization": f"Bearer {GREMLIN_TOKEN}"})
    return r.json()["attacks"]

def to_splunk_event(attack):
    return {
        "time":  attack["created_at"],
        "sourcetype": "gremlin:attack",
        "event": {
            "attack_id":   attack["id"],
            "scenario":    attack.get("scenarioId"),
            "user":        attack["createdBy"]["email"],
            "attack_type": attack["type"],
            "targets":     attack["targets"],
            "state":       attack["state"],
            "started_at":  attack["startedAt"],
            "ended_at":    attack.get("endedAt"),
            "halted":      attack.get("halted", False),
        }
    }

def push_to_splunk(events):
    for e in events:
        requests.post(SPLUNK_HEC,
                      headers={"Authorization": f"Splunk {SPLUNK_HEC_TOKEN}"},
                      json=e, timeout=5)

def main():
    since = (datetime.now(timezone.utc) - timedelta(minutes=10)).isoformat()
    attacks = poll_gremlin_since(since)
    push_to_splunk(to_splunk_event(a) for a in attacks)

if __name__ == "__main__":
    main()

Step-by-step trace.

Layer	Component	Responsibility
Agents	`gremlind` on every VM/container	inherit tag hierarchy; execute attack; report health
Control plane	Gremlin SaaS	scenario execution, blast-radius, halt
Scenario definitions	Terraform module	one module per fault type; per-env instances via for_each
Health checks	Datadog monitors	data SLI thresholds; Gremlin polls these
Halt behaviour	Gremlin `halt_on_unhealthy`	Datadog Alert → scenario abort
Audit trail	SIEM exporter → Splunk	every attack ID, user, targets, timing mirrored to SIEM
Approval matrix	GitHub CODEOWNERS + CI	machine-checked green-run count + human sign-off
On-call visibility	Slack `#chaos-live` channel	scenario start / end / halt posts via Gremlin webhook

After deployment, every chaos attack against the data platform is: (a) defined in Terraform (audit trail), (b) executed via the Gremlin API (blast-radius enforced), (c) gated by Datadog Halt-on-Breach (data-SLO seatbelt), (d) mirrored to Splunk (compliance receipt), (e) announced to #chaos-live (operator awareness). On-call always knows what's running; auditors always have the receipt.

Output:

Artifact	Purpose	Consumer
Terraform state	scenario definitions	infra reviewer / audit
Gremlin control plane	live scenario execution	data-platform SRE
Datadog monitor	halt criterion	Gremlin health-check poller
Splunk `gremlin:attack` events	audit trail	compliance / SOC-2
Slack `#chaos-live` post	live operator awareness	on-call
Post-mortem doc (Confluence)	scenario outcome + backlog	data-platform team

Why this works — concept by concept:

Tag hierarchy on every agent — the service / env / criticality / chaos-tier tags define the selector language for every scenario. A well-designed tag hierarchy makes "blast radius = 10% of service=kafka-broker in env=prod chaos-tier=1pct" a one-line selector.
Terraform module per fault type — a Terraform module encodes the fault + its parameters + its halt criterion. for_each over {stage, 1pct, 10pct} instantiates the same scenario at each blast tier with the right selectors. Chaos-as-code becomes chaos-in-Terraform.
Datadog Halt-on-Breach — the halt criterion is a Datadog monitor, not a Gremlin-internal check. This decouples the SLI definition from the chaos tool — the same SLI monitor pages on-call and halts chaos. One source of truth.
SIEM audit trail — every attack ID + user + targets + timing mirrors to Splunk. When a security auditor asks "who ran chaos in prod on 2026-07-30 at 14:00 and what did it hit," you point at a Splunk search. This is what makes prod chaos defensible in a SOC-2 environment.
Slack live channel — #chaos-live gets a post at scenario start (with hypothesis + expected duration + rollback path), during halts, and at scenario end (with verdict). Operators can silence chaos-triggered alerts because they know a chaos window is active.
CI-gated promotion — the same graduation-matrix pattern from LitmusChaos applies: dev → staging → 1% prod → 10% → 100%. CI counts green runs; CODEOWNERS enforces human approval. Chaos in prod is earned, not granted.
Cost — Gremlin SaaS licence (per-agent per-month), gremlind footprint (~50 MB per host), one Terraform module per fault type, one Datadog monitor per SLI, one Splunk index for chaos events. Compared to the DIY equivalent (write attack primitives, build a control plane, build audit, build halt), Gremlin trades money for time-to-value. O(1) per attack; O(N scenarios × N stages) per pipeline.

Design
Topic — design
Design problems on multi-cloud data platform resilience

Practice →

API Topic — api-integration API integration problems on webhook and halt endpoints

Practice →

4. Data-plane fault injection — Kafka broker kill, Airflow scheduler kill, Spark executor loss

Stateful chaos — the three canonical data-plane experiments and their guardrails

The mental model in one line: data fault injection is the practice of applying chaos primitives (pod-delete, network-loss, blackhole, latency) to the stateful data systems — Kafka brokers with partition leadership + ISR + transactional coordinators, Airflow schedulers with sole write-access to a metadata DB, and Spark executors with in-memory shuffle blocks and driver-tracked lineage — where the correctness contract binds the experiment far more tightly than for stateless HTTP services. Every senior data platform team runs these three experiments; the difference between amateur and senior is whether the experiment ships with the specific data-system guardrail that makes the chaos survivable — min.insync.replicas=2 + acks=all for Kafka, catchup=False + max_active_runs=1 for Airflow, dynamicAllocation.enabled=true + blacklist.enabled=true for Spark.

The three canonical data-plane chaos experiments.

Kafka broker kill. Kill a broker; verify the pipeline stays inside its freshness + lag SLO. Failure modes: unacked writes if acks!=all, ISR shrink storms, coordinator loss (transactional-EOS blast). Guardrails: min.insync.replicas=2, acks=all, producer.idempotence=true, retention.ms sized to slot lag.
Airflow scheduler kill. Kill the scheduler mid-DAG; verify running tasks continue and new tasks are scheduled after the restart. Failure modes: mid-parse crashes leave dag_run in queued forever, task_instance rows stuck in running when the executor is a Celery worker, catchup floods the queue on restart. Guardrails: catchup=False, max_active_runs=1, retries>=2, dagrun_timeout set.
Spark executor loss. Kill a Spark executor; verify the driver reprovisions and the batch/streaming job continues. Failure modes: shuffle-fetch failures cascade into stage restarts, driver OOM if too many executors lost, dynamic-allocation flap. Guardrails: spark.dynamicAllocation.enabled=true, spark.blacklist.enabled=true, spark.task.maxFailures=4, spark.stage.maxConsecutiveAttempts=4.

The stateful chaos multiplier — why data-plane chaos is different.

In-flight state. A stateless HTTP service can be killed mid-request and the client just retries. A Kafka broker killed mid-transaction leaves the coordinator's state machine mid-way; recovery requires either a coordinator failover or a transactional abort. The chaos experiment must know which.
Ordering contracts. Kafka partitions guarantee in-partition order; a broker kill preserves it. Spark shuffle blocks are not ordered but they are addressed by (mapId, reduceId); executor loss forces the driver to re-fetch from surviving executors, which the shuffle service handles transparently if configured. Airflow DAG runs are logical ordering only; the scheduler kill risks orphaning a mid-run task_instance if the executor is out-of-process.
SLI selection. For each experiment the correct SLI differs. Kafka: consumer group lag + producer error rate. Airflow: DAG success rate + scheduler heartbeat age. Spark: batch processing time p99 + stage retry count. Getting the SLI wrong makes the experiment blind.
Recovery topology. Kafka has a metadata quorum (ZooKeeper or KRaft) and a partition leader per topic-partition — recovery is per-partition. Airflow has a single scheduler + N executors — recovery is scheduler-restart-plus-heartbeat. Spark has a driver + N executors — recovery is executor-reprovision.

The three guardrails you cannot ship without.

Kafka: acks=all on the producer + min.insync.replicas=2 on the topic. Without both, a broker kill can lose an unacknowledged write. This is the single most-cited Kafka correctness invariant in interviews.
Airflow: catchup=False on the DAG + max_active_runs=1. Without both, a scheduler restart after a downtime replays every missed logical date and can OOM the queue.
Spark: spark.dynamicAllocation.enabled=true + spark.blacklist.enabled=true. Without both, an executor loss either leaves the cluster under-provisioned or repeatedly retries onto the same bad node.

Common interview probes on data-plane chaos.

"How do you verify a Kafka broker kill didn't lose writes?" — required answer: acks=all + min.insync.replicas=2 + producer-error-rate SLI + downstream row-count reconciliation.
"What breaks when you kill an Airflow scheduler mid-DAG?" — orphaned task_instance rows in running state if executor is separate; dag_run stuck in queued if the parse loop crashed.
"What's the Spark executor-loss failure mode?" — shuffle-fetch failures on downstream stages; without dynamicAllocation, the cluster shrinks and the job hangs.
"How do you pick which broker to kill?" — random over applabel=kafka-broker avoiding the controller; the controller loss is a separate experiment (controller-kill) because it exercises different code paths.

Worked example — Kafka broker kill with `min.insync.replicas` guard

Detailed explanation. The canonical Kafka data-plane chaos experiment: kill one broker in a 6-broker cluster (replication.factor=3, min.insync.replicas=2) and verify no committed messages are lost and the producer's record-error-rate stays at zero. This is the interview go-to; every senior Kafka engineer has run it.

Cluster. 6 brokers, replication.factor=3, min.insync.replicas=2, KRaft-mode metadata quorum.
Producer. acks=all, enable.idempotence=true, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5.
Fault. LitmusChaos pod-delete on 1 of 6 brokers (17% blast radius).
Guardrails. promProbe on kafka_consumergroup_lag < 100000 + promProbe on rate(kafka_producer_record_error_total[1m]) < 0.1.

Question. Write the ChaosEngine + the producer / topic config that makes this experiment survivable, and quantify what happens if you drop the min.insync.replicas guard.

Input.

Setting	With guard	Without guard
replication.factor	3	3
min.insync.replicas	2	1
acks (producer)	all	1
enable.idempotence	true	false
Broker kill outcome	0 lost writes	~250 lost writes

Code.

# 1. Topic configuration — enforced by kafka-topics.sh at creation
# kafka-topics.sh --create \
#   --topic events \
#   --partitions 24 \
#   --replication-factor 3 \
#   --config min.insync.replicas=2 \
#   --config unclean.leader.election.enable=false \
#   --bootstrap-server localhost:9092

// 2. Producer configuration — the two lines that make the chaos survivable
Properties p = new Properties();
p.put("bootstrap.servers", "kafka-0:9092,kafka-1:9092,kafka-2:9092");
p.put("acks",                                 "all");
p.put("enable.idempotence",                   "true");
p.put("retries",                              Integer.MAX_VALUE);
p.put("max.in.flight.requests.per.connection", "5");
p.put("delivery.timeout.ms",                  "120000");
p.put("request.timeout.ms",                    "30000");
p.put("linger.ms",                                "20");
p.put("compression.type",                      "lz4");

# 3. ChaosEngine — kill 1 of 6 brokers with two data-SLI probes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-broker-kill
  namespace: kafka
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'kafka'
    applabel: 'app=kafka-broker,role!=controller'
    appkind: 'statefulset'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: CHAOS_INTERVAL, value: '60'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
            - {name: FORCE, value: 'false'}
        probe:
          - name: consumer-lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(kafka_consumergroup_lag{group="warehouse-sink"})'
              comparator: {type: int, criteria: '<', value: '100000'}
          - name: producer-error-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'sum(rate(kafka_producer_record_error_total[1m]))'
              comparator: {type: float, criteria: '<', value: '0.1'}
          - name: isr-integrity-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'min(kafka_topic_partition_in_sync_replica{topic="events"})'
              comparator: {type: int, criteria: '>=', value: '2'}

Step-by-step explanation.

The topic is created with replication.factor=3, min.insync.replicas=2, unclean.leader.election.enable=false. When a broker holding a partition leader dies, controller elects a new leader from the ISR; if the ISR has < 2 in-sync replicas the producer receives NotEnoughReplicasException and retries. With unclean-leader-election disabled, no out-of-sync replica is ever elected, so no committed writes vanish.
The producer sets acks=all (waits for the leader + all in-sync replicas to acknowledge) and enable.idempotence=true (attaches a producer-id + sequence, so retries don't duplicate). Together these give exactly-once-per-producer semantics — the exact property the chaos experiment verifies.
applabel: 'app=kafka-broker,role!=controller' explicitly excludes the controller pod. Controller loss is a different chaos experiment because the controller drives partition-leader election — testing controller loss requires different halt criteria (controller-fail-over time SLO instead of consumer lag).
The three probes cover three orthogonal SLIs: (a) consumer lag stays bounded (freshness); (b) producer error rate stays low (durability); (c) minimum ISR stays >= 2 (correctness contract). If any breaches, the experiment aborts.
FORCE: 'false' means the broker gets a graceful shutdown (SIGTERM → configured shutdown.timeout.ms → SIGKILL). Kafka brokers flush their memory-mapped log segments on graceful shutdown; a forced kill would test crash-recovery instead. Both are valuable; label them as different experiments.

Output.

Time (s)	Event	Consumer lag	Producer error rate	Min ISR
0	broker-3 alive	800	0.00	3
10	broker-3 SIGTERM	900	0.00	3
15	broker-3 gone; ISR shrinks	1400	0.02	2
30	new leaders elected	4200	0.01	2
60	broker-3 restarted	12000	0.00	2
90	ISR healing	8000	0.00	3
120	full ISR restored	3200	0.00	3

Rule of thumb. Kafka broker chaos without acks=all + min.insync.replicas=2 is theatre — you're not testing the resilient system, you're testing an unsafe one. Enforce both at the topic + producer config level before the chaos experiment ships. Add an isr-integrity-guard probe to make the min-ISR contract auditable during chaos.

Worked example — Airflow scheduler kill with `catchup=False` guard

Detailed explanation. The canonical Airflow chaos experiment: kill the scheduler pod while a DAG has tasks in-flight, verify running tasks complete (executor is Celery or Kubernetes, out-of-process), verify the scheduler restarts within HEALTHCHECK_MAX_AGE and doesn't flood the queue with catchup runs on restart, verify no task_instance rows are stuck in running state after the scheduler is healthy again.

Setup. Airflow 2.9 with KubernetesExecutor, scheduler HA disabled (single scheduler), HEALTHCHECK_MAX_AGE=30s.
DAG. Hourly ETL with catchup=False, max_active_runs=1, retries=2.
Fault. pod-delete on scheduler pod.
Guardrails. k8sProbe on scheduler Deployment + cmdProbe on select count(*) from task_instance where state='running' and queued_dttm < now() - interval '10 minutes'.

Question. Write the ChaosEngine and demonstrate what breaks if catchup=True is misconfigured.

Input.

Setting	With guard	Without guard
catchup	False	True
max_active_runs	1	unbounded
Scheduler downtime	30 s	30 s
Post-restart runs	1	~24 (one per missed hour × 24 hrs)

Code.

# The DAG with the correct guardrails
from airflow import DAG
from airflow.decorators import task
from datetime import datetime, timedelta

with DAG(
    dag_id="hourly_etl",
    start_date=datetime(2026, 1, 1),
    schedule="@hourly",
    catchup=False,                 # <-- guard 1: don't backfill missed runs
    max_active_runs=1,             # <-- guard 2: never more than one instance
    dagrun_timeout=timedelta(minutes=45),
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "execution_timeout": timedelta(minutes=30),
    },
) as dag:

    @task
    def extract(): ...
    @task
    def transform(x): ...
    @task
    def load(y): ...

    load(transform(extract()))

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: airflow-scheduler-kill
  namespace: airflow
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'airflow'
    applabel: 'component=scheduler'
    appkind: 'deployment'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '90'}
            - {name: CHAOS_INTERVAL, value: '45'}
            - {name: PODS_AFFECTED_PERC, value: '100'}
            - {name: FORCE, value: 'false'}
        probe:
          - name: scheduler-restart-guard
            type: k8sProbe
            mode: EOT
            runProperties: {probeTimeout: 30, interval: 5, retry: 3, stopOnFailure: true}
            k8sProbe/inputs:
              group: 'apps'
              version: 'v1'
              resource: 'deployments'
              namespace: 'airflow'
              fieldSelector: 'metadata.name=airflow-scheduler'
              operation: 'present'
          - name: no-stuck-tasks
            type: cmdProbe
            mode: EOT
            runProperties: {probeTimeout: 30, interval: 5, retry: 1, stopOnFailure: true}
            cmdProbe/inputs:
              command: |
                psql -h postgres-airflow.airflow.svc -U airflow -d airflow -tAc \
                  "SELECT COUNT(*) FROM task_instance
                   WHERE  state='running'
                     AND  queued_dttm < NOW() - INTERVAL '10 minutes'"
              comparator: {type: int, criteria: '=', value: '0'}
              source:
                image: 'postgres:16-alpine'
                imagePullPolicy: IfNotPresent
                inheritInputs: true
          - name: no-catchup-flood
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 5, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'airflow_dagrun_active_count{dag_id="hourly_etl"}'
              comparator: {type: int, criteria: '<=', value: '1'}

Step-by-step explanation.

PODS_AFFECTED_PERC: 100 on a Deployment with 1 replica means the entire scheduler dies. Kubernetes will reschedule the pod immediately; the scheduler restarts, re-parses DAGs, resumes its scheduling loop. Executor tasks (Kubernetes pods) continue running because they're separate pods; the scheduler only manages state transitions.
catchup=False is the load-bearing invariant. On restart, the scheduler asks "what's the last logical date I scheduled for this DAG?" Without catchup, it schedules only the next logical date. With catchup=True and 24h of downtime, it would try to schedule 24 backfill runs immediately, potentially overwhelming the executor.
max_active_runs=1 is the second guard: even if the scheduler somehow tries to schedule multiple runs, only 1 is allowed active. This bounds the blast radius of a scheduler bug.
The EOT cmdProbe runs SQL against the metadata DB after chaos ends: "any task_instances stuck in 'running' state where their queue time is > 10 minutes ago?" If the scheduler died mid-transition on a task_instance (e.g. between queued and running), the row can be orphaned. The probe catches this.
The EOT promProbe on airflow_dagrun_active_count <= 1 verifies no catchup flood. If it fires, either the DAG has catchup=True (config bug) or the scheduler restart triggered multiple runs (Airflow bug).

Output.

Time (s)	Event	Scheduler ready	Stuck tasks	Active DAG runs
0	Chaos applied	1	0	1
5	Scheduler pod killed	0	0 (in-flight preserved)	1
20	Pod restarted; parsing DAGs	0	0	1
40	Scheduler resuming loop	1	0	1
60	Chaos window ends	1	0	1
EOT	Verification	1	0	1

Rule of thumb. For any Airflow chaos experiment, verify the DAG has catchup=False + max_active_runs=1 before running the experiment. Ship the two-line SQL probe that hunts orphaned task_instance rows as an EOT guard — it's the single most under-instrumented Airflow failure mode.

Worked example — Spark executor loss with dynamic allocation

Detailed explanation. The canonical Spark chaos experiment: kill 2 of 10 Spark executors mid-batch, verify the driver reprovisions replacements (dynamic allocation), verify shuffle-fetch failures don't cascade into unbounded stage retries, verify the job SLA (batch processing p99) stays under threshold. This tests both Spark's own fault tolerance and the surrounding YARN / K8s resource-manager's ability to reprovision.

Cluster. Spark 3.5 on K8s, 1 driver + 10 executor pods, spark.dynamicAllocation.enabled=true.
Guards. spark.blacklist.enabled=true, spark.task.maxFailures=4, spark.stage.maxConsecutiveAttempts=4, spark.dynamicAllocation.minExecutors=8.
Fault. LitmusChaos pod-delete on 20% of executors (2 of 10).
Halt. spark_streaming_batch_processing_seconds_p99 < 30.

Question. Write the ChaosEngine and the Spark submit config that makes executor loss survivable.

Input.

Setting	Value
spark.dynamicAllocation.enabled	true
spark.dynamicAllocation.minExecutors	8
spark.dynamicAllocation.maxExecutors	20
spark.blacklist.enabled	true
spark.task.maxFailures	4
spark.stage.maxConsecutiveAttempts	4
spark.shuffle.service.enabled	true (external shuffle service)

Code.

# 1. Spark submit config that survives executor loss
spark-submit \
  --master k8s://https://kubernetes.default.svc \
  --deploy-mode cluster \
  --conf spark.executor.instances=10 \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=8 \
  --conf spark.dynamicAllocation.maxExecutors=20 \
  --conf spark.dynamicAllocation.executorIdleTimeout=60s \
  --conf spark.blacklist.enabled=true \
  --conf spark.blacklist.timeout=1h \
  --conf spark.task.maxFailures=4 \
  --conf spark.stage.maxConsecutiveAttempts=4 \
  --conf spark.shuffle.service.enabled=true \
  --conf spark.kubernetes.executor.label.app=spark-executor \
  --conf spark.kubernetes.executor.deleteOnTermination=true \
  --conf spark.streaming.stopGracefullyOnShutdown=true \
  --conf spark.sql.streaming.gracefulShutdown.timeout=60s \
  --class com.acme.EventStreamJob \
  local:///opt/spark/jobs/event-stream.jar

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: spark-executor-kill
  namespace: spark
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: 'spark'
    applabel: 'app=spark-executor'
    appkind: 'pod'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '180'}
            - {name: CHAOS_INTERVAL, value: '60'}
            - {name: PODS_AFFECTED_PERC, value: '20'}
            - {name: FORCE, value: 'true'}
        probe:
          - name: batch-p99-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: |
                histogram_quantile(0.99,
                  sum by(le)(rate(spark_streaming_batch_processing_seconds_bucket{job="event-stream"}[3m])))
              comparator: {type: int, criteria: '<', value: '30'}
          - name: executor-reprovision-guard
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'spark_executor_active_count{job="event-stream"}'
              comparator: {type: int, criteria: '>=', value: '8'}
          - name: stage-retry-guard
            type: promProbe
            mode: EOT
            runProperties: {probeTimeout: 5, interval: 15, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'max(spark_stage_consecutive_attempts{job="event-stream"})'
              comparator: {type: int, criteria: '<', value: '4'}

Step-by-step explanation.

spark.dynamicAllocation.enabled=true lets the driver request new executors from the resource manager when load requires. minExecutors=8 sets the floor — even after two are killed, the driver will request replacements to get back above 8.
spark.blacklist.enabled=true + spark.blacklist.timeout=1h prevents Spark from repeatedly retrying tasks onto a node that just crashed an executor. Without this, a bad node produces a cascade of retries as the driver keeps assigning tasks that keep failing.
spark.shuffle.service.enabled=true runs the external shuffle service (as a DaemonSet on K8s or a NodeManager auxiliary on YARN) which persists shuffle blocks outside the executor lifetime. Without it, executor death loses shuffle data and forces a re-compute; with it, surviving executors keep serving shuffle blocks.
spark.streaming.stopGracefullyOnShutdown=true on the streaming job ensures the driver drains active batches before exiting on a SIGTERM. Combined with spark.kubernetes.executor.deleteOnTermination=true, executor pods clean up their K8s resources on exit.
The three probes cover three orthogonal invariants: (a) batch processing p99 stays under 30 s (SLA); (b) executor count recovers to >= 8 within the chaos window (reprovisioning works); (c) no stage hits max consecutive attempts (retry storm didn't happen).

Output.

Time (s)	Event	Batch p99 (s)	Active executors	Max stage attempts
0	Chaos start	8	10	1
30	2 executors killed	12	8	1
60	Shuffle-fetch failures	22	8	2
90	Driver reprovisions	18	10	2
120	Job stable	10	10	2
EOT	Verification	9	10	2

Rule of thumb. Every Spark chaos experiment must ship with dynamicAllocation.enabled=true + blacklist.enabled=true + shuffle.service.enabled=true. Without the trio, executor loss produces either an under-provisioned job (no reprovisioning) or a retry storm (no blacklist) or a shuffle re-compute (no shuffle service).

Senior interview question on data-plane chaos

A senior interviewer might ask: "Design a game-day chaos scenario that covers your entire ingest stack — Kafka → Spark Structured Streaming → Iceberg — with one composed experiment that kills a broker, then loses an executor, then latency-injects S3. Include the halt criteria, the ordering, and how you'd staff the game day."

Solution Using a composed multi-step scenario with per-step data-SLI halts and a game-day runbook

# ChaosEngine — three sequential experiments, each with its own halt SLI
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: ingest-stack-game-day
  namespace: chaos
spec:
  engineState: 'active'
  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - {name: TOTAL_CHAOS_DURATION, value: '120'}
            - {name: PODS_AFFECTED_PERC, value: '17'}
            - {name: TARGET_PODS, value: 'app=kafka-broker,role!=controller'}
        probe: &data_sli_probes
          - name: freshness-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:freshness_p99_min'
              comparator: {type: int, criteria: '<', value: '15'}
          - name: lag-guard
            type: promProbe
            mode: Continuous
            runProperties: {probeTimeout: 5, interval: 10, retry: 1, stopOnFailure: true}
            promProbe/inputs:
              endpoint: 'http://prometheus.monitoring:9090'
              query: 'pipeline:kafka_lag_messages'
              comparator: {type: int, criteria: '<', value: '150000'}

    # 2-minute gap between experiments — let the pipeline settle
    - name: pod-delete
      spec:
        components:
          env:
            - {name: RAMP_TIME, value: '120'}     # 2-min delay before this exp starts
            - {name: TOTAL_CHAOS_DURATION, value: '180'}
            - {name: PODS_AFFECTED_PERC, value: '20'}
            - {name: TARGET_PODS, value: 'app=spark-executor'}
        probe: *data_sli_probes

    - name: pod-network-latency
      spec:
        components:
          env:
            - {name: RAMP_TIME, value: '120'}
            - {name: TOTAL_CHAOS_DURATION, value: '300'}
            - {name: NETWORK_LATENCY, value: '500'}
            - {name: DESTINATION_HOSTS, value: 's3.us-east-1.amazonaws.com'}
            - {name: TARGET_PODS, value: 'app=iceberg-writer'}
        probe: *data_sli_probes

<!-- game-day-runbook.md — the human ceremony -->
# Ingest Stack Game Day — Runbook

**Duration.** 90 minutes (30 min setup + 30 min chaos + 30 min post-mortem)

**Roles.**
- Game Master (GM): drives the ceremony; runs the chaos apply command
- Observer 1 (Kafka): watches Kafka SLI dashboard; calls out anomalies
- Observer 2 (Spark): watches Spark SLI dashboard
- Observer 3 (Iceberg/S3): watches Iceberg + S3 dashboards
- Scribe: takes running notes into the post-mortem doc
- On-Call: shadows; can halt if real customer impact

**Pre-flight.**
- [ ] All observers connected to Zoom bridge
- [ ] All SLI dashboards open (freshness, lag, batch p99, iceberg-commit-p99)
- [ ] `#chaos-live` Slack channel posted with hypothesis
- [ ] Halt authority confirmed (any observer can shout "HALT" to trigger `kubectl delete chaosengine`)

**Sequence.**
- T+0 : GM applies `ChaosEngine`; observers narrate SLIs
- T+2 : Kafka broker killed (probes gate lag < 150k)
- T+4 : Kafka window ends; 2-min settle
- T+6 : Spark executors killed (probes gate batch p99)
- T+9 : Spark window ends; 2-min settle
- T+11: S3 latency injected (probes gate iceberg-commit-p99)
- T+16: S3 window ends
- T+18: Chaos complete; observers verify all SLIs recovered
- T+30: Scribe finalises post-mortem doc; GM assigns action items

Step-by-step trace.

Step	Component	Action	Halt criterion
1	ChaosEngine applied	kafka-broker-kill starts	lag < 150k, freshness < 15 min
2	120s chaos + settle	broker recovers	ISR heals
3	spark-executor-kill starts	2 executors killed	batch p99 < 30s
4	180s chaos + settle	executors reprovision	dynamic allocation
5	s3-latency starts	+500ms to S3	iceberg-commit-p99 < 30s
6	300s chaos ends	latency removed	commits catch up
7	GameDay doc	scribe finalises	action items filed

After the ceremony, the game-day doc lists every observed deviation, every halted probe (with reason), and every backlog item filed in Jira. The doc becomes the reference for the next ceremony; every recurring ceremony either graduates the blast radius or logs a resilience regression.

Output:

Artifact	Purpose	Consumer
ChaosEngine CRs (3 experiments)	scripted chaos	LitmusChaos operator
Data-SLI probes (2 per exp)	halt criteria	LitmusChaos runner
game-day-runbook.md	human ceremony script	GM + observers
Live SLI dashboards	real-time verdict	observers + on-call
Post-mortem doc	learning artifact	data-platform team
Jira backlog items	fix commitments	product + engineering

Why this works — concept by concept:

Composed sequential experiments — three chaos primitives run in sequence with 2-min settle gaps. Each experiment exercises a different subsystem (broker, executor, external dependency) while the others get to recover. Sequence-with-gaps is safer than parallel — the pipeline never faces two novel failure modes at once.
Shared data-SLI probe set (&data_sli_probes YAML anchor) — the freshness + lag guards are defined once and referenced by all three experiments. This keeps the halt criteria consistent across the ceremony; every experiment is judged by the same yardstick.
Ramp time between experiments — RAMP_TIME: '120' delays each experiment start so the previous one has 2 minutes to fully settle. Without ramp gaps, back-to-back failures mask each other's SLI signal.
Explicit game-day roles — GM, observers, scribe, on-call are named roles. This is the missing piece in most chaos programs: the ceremony has to have humans in the loop, or the automation looks like an unmanaged accident.
Halt authority = any observer — any observer can shout "HALT" and trigger kubectl delete chaosengine ingest-stack-game-day. This is the human escape hatch that the automated probes can't provide (e.g. if a customer bug report comes in mid-ceremony).
Post-mortem doc as source of truth — the doc lands in a Confluence space with a stable URL. Every game-day ceremony produces one; every subsequent ceremony references the prior; the compounding effect is a written history of your pipeline's resilience posture.
Cost — 90 minutes of 5 engineers = ~7.5 engineer-hours per ceremony; quarterly cadence = 30 engineer-hours per year. Compared to the cost of one unplanned incident (~40+ engineer-hours + customer impact), the game day is a massive positive-ROI trade. O(1) per ceremony; O(quarterly cadence) per pipeline.

Streaming
Topic — streaming
Streaming problems on Kafka consumer resilience

Practice →

Event Topic — event-processing Event processing problems on exactly-once contracts

Practice →

5. Hypothesis-driven chaos loop — steady state → hypothesis → blast → verify

The four-step loop that turns chaos from a stunt into a repeatable engineering practice

The mental model in one line: blast radius data chaos becomes a practice (rather than a stunt) when every experiment cycles through four explicit stages — codify the steady state as a data SLI, state a falsifiable hypothesis about how the pipeline should behave under a specific failure, run the experiment inside a bounded blast radius with an auto-halt on SLI breach, and verify by comparing SLI curves before / during / after with a written post-mortem that either graduates the blast radius or produces a fix backlog item. Every mature chaos program lives inside this loop; every immature one skips one or more stages and produces "we killed something and it seemed fine" as its output.

The four stages, mapped onto concrete data-pipeline artifacts.

Steady state. One or more Prometheus recording rules that define "the pipeline is healthy." For a Kafka → Spark → Iceberg stack: pipeline:freshness_p99_min, pipeline:kafka_lag_messages, pipeline:batch_success_rate_5m, pipeline:row_count_delta_1h. The SLI is data-first, infra-second.
Hypothesis. A written, falsifiable prediction of the form "if X breaks, Y stays within Z." Example: "If we kill 1 of 6 Kafka brokers, freshness_p99_min stays under 15 and kafka_lag_messages stays under 100k for the 120s chaos window."
Experiment. The chaos configuration that tests exactly the hypothesis — the smallest fault that exercises the failure mode, run inside the smallest blast radius that produces a signal, with halt-on-SLI-breach probes. The experiment is the ChaosEngine CR + probe set.
Verify. Compare SLI curves before / during / after (Grafana overlay). Write the post-mortem doc: hypothesis, actual outcome, probe timeline, root cause of any deviation, decision — graduate blast radius, or file a fix backlog item, or repeat the experiment next cycle.

The graduation ladder — how blast radius escalates over calendar time.

Stage 1 — dev. Nightly automated runs against a dev cluster. Blast radius = 100% of dev workloads (they're disposable). Failure = auto-file a Jira; no on-call impact. Goal: catch config errors before they touch shared environments.
Stage 2 — staging. Weekly automated runs against staging. Blast radius = same percent as prod-target (17% of brokers, 20% of executors) so staging teaches you what prod will experience. Failure = Slack ping to data-platform-sre; no customer impact. Goal: validate the experiment's halt criteria against realistic-scale infra.
Stage 3 — prod-1pct. Monthly ceremony in prod during business hours. Blast radius = 1 broker of 6, or 2 executors of 20, or 1% of writers. GM + observers required; formal game-day format. Goal: prove the SLO holds under real production load.
Stage 4 — prod-10pct. Quarterly ceremony. Blast radius scales to 10% of tier-1 workloads. Executive sponsor required; on-call bridge open. Goal: verify the pipeline handles a "one AZ down" magnitude failure.
Stage 5 — prod-100pct. Annual DR drill. Blast radius = full region outage or full-cluster loss. DR runbook + customer comms. Goal: annual DR compliance and executive-level confidence.

The written post-mortem template — what "verify" produces.

Hypothesis (stated before the experiment). "If X breaks, Y stays within Z."
Actual outcome. Screenshot of SLI dashboard with the chaos window highlighted; probe timeline extracted from ChaosResult.
Deviation analysis. For each SLI, was the actual within predicted? If not, why? Was it a config bug, a code bug, or a genuine resilience gap?
Decision. Graduate blast radius / repeat at same radius / halt this experiment class pending fix.
Action items. Each deviation gets a Jira ticket with owner + ETA.

Common interview probes on the chaos loop.

"How do you decide when to graduate blast radius?" — required answer: N green runs at the current stage (matrix), no red runs in the last M runs, peer-reviewed PR to promote.
"What if a chaos experiment reveals a bug in production?" — halt-on-breach immediately, on-call takes over, file a P1 bug with the SLI evidence, roll back any partially-applied fault.
"How do you write a falsifiable hypothesis?" — required answer: "if X breaks, Y stays within Z" with X, Y, Z all quantitative. Weak: "the pipeline is resilient." Strong: "freshness_p99_min stays < 15 for 120 s of pod-delete on 1 of 6 brokers."
"What separates a chaos experiment from a load test?" — chaos tests unavailability failure modes; load tests capacity. Chaos gives you an SLO verdict under fault; load gives you a throughput ceiling under peak.

Worked example — writing a falsifiable chaos hypothesis

Detailed explanation. The hypothesis is the single most-skipped step in first-attempt chaos programs. Teams reach for a fault ("let's kill a broker") without writing down what should happen. Without a written hypothesis, "we ran chaos" has no verdict — you either observe "it seemed fine" (which is not verifiable) or "it broke" (which is not attributable). Every mature chaos ceremony writes the hypothesis in advance and pastes it into the game-day doc before the experiment runs.

Falsifiable. The hypothesis must be a statement that concrete data can prove wrong. "The pipeline is resilient" is not falsifiable. "freshness_p99_min stays < 15 min for 120 s of pod-delete on 1 of 6 brokers" is.
Quantitative. Every noun in the hypothesis must have a number. Not "some brokers" — "1 of 6." Not "low latency" — "< 15 min p99." Not "briefly" — "120 s."
Bounded. The hypothesis includes an explicit time window and blast radius. Without bounds, "stays < 15 min" is trivially violable at some future time.
Named SLIs. The hypothesis references specific Prometheus recording rules (pipeline:freshness_p99_min, pipeline:kafka_lag_messages) — not vague concepts.

Question. Write three chaos hypotheses for the ingest stack, and derive the halt criterion from each.

Input.

Hypothesis	Failure mode	SLI	Threshold	Duration	Blast radius
Kafka broker survives without lag spike	pod-delete	pipeline:kafka_lag_messages	< 100k	120s	1/6 brokers
Spark stays inside p99 under executor loss	pod-delete	spark_streaming_batch_p99_sec	< 30s	180s	2/10 executors
Iceberg commits queue but don't fail under S3 latency	pod-network-latency	iceberg_commit_seconds_p99	< 30s	300s	25% writers

Code.

# Hypothesis card — checked into Git alongside every ChaosEngine
# hypothesis-cards/kafka-broker-kill.yaml
apiVersion: pipecode.ai/v1
kind: ChaosHypothesis
metadata:
  name: kafka-broker-kill
spec:
  hypothesis: |
    IF one Kafka broker (of six) is deleted via pod-delete,
    THEN pipeline:kafka_lag_messages stays below 100,000
     AND pipeline:freshness_p99_min stays below 15 minutes
     AND min(kafka_topic_partition_in_sync_replica{topic="events"}) stays >= 2
    FOR the duration of the 120-second chaos window.

  falsifiable_by:
    - metric: 'pipeline:kafka_lag_messages'
      condition: 'exceeds 100000 at any 10-second sample'
    - metric: 'pipeline:freshness_p99_min'
      condition: 'exceeds 15 at any 10-second sample'
    - metric: 'min(kafka_topic_partition_in_sync_replica{topic="events"})'
      condition: 'falls below 2 at any 15-second sample'

  chaosengine_ref: chaos/base/engines/kafka-broker-kill.yaml

  owner: data-platform-sre
  reviewers: [alice, bob]

# hypothesis_check.py — a CI script that verifies every ChaosEngine has a hypothesis card
import glob, yaml, sys

engines = glob.glob("chaos/base/engines/*.yaml")
cards   = glob.glob("chaos/hypothesis-cards/*.yaml")

engine_names = {yaml.safe_load(open(f))["metadata"]["name"] for f in engines}
card_refs = {yaml.safe_load(open(c))["spec"]["chaosengine_ref"].split("/")[-1].split(".")[0]
             for c in cards}

missing = engine_names - card_refs
if missing:
    print(f"ERROR: ChaosEngines without hypothesis cards: {missing}")
    sys.exit(1)

# Verify each card has quantitative falsifiability
for c in cards:
    card = yaml.safe_load(open(c))
    fbs = card["spec"]["falsifiable_by"]
    for fb in fbs:
        cond = fb["condition"]
        if "some" in cond.lower() or "few" in cond.lower():
            print(f"ERROR: non-quantitative condition in {c}: {cond}")
            sys.exit(1)

print("All ChaosEngines have quantitative hypothesis cards.")

Step-by-step explanation.

The ChaosHypothesis CR is a documentation artifact — it doesn't drive execution, it drives review. Every ChaosEngine in Git must have a corresponding hypothesis card; the CI script enforces this.
The hypothesis text uses IF ... THEN ... AND ... FOR ... grammar. This isn't arbitrary — it forces the author to name the failure, name the SLI(s), name the threshold(s), and name the duration. Any missing piece becomes obviously wrong on review.
The falsifiable_by block enumerates the exact conditions that would falsify the hypothesis. Each metric + condition pair maps 1:1 to a promProbe in the ChaosEngine — if the two get out of sync, the experiment can pass while the hypothesis is falsified (a silent bug).
The CI check catches two failure modes: (a) ChaosEngine without a hypothesis card (execution without prediction), (b) hypothesis card with vague language ("some", "few", "briefly") — quantitative discipline enforced at commit time.
The reviewers field is not just administrative — chaos hypotheses need domain review. Killing a Kafka broker without ISR knowledge, or restarting the Airflow scheduler without catchup knowledge, is where teams ship dangerous experiments. Two reviewers is the norm; the more senior one should have shipped the specific system before.

Output.

Artifact	Purpose	Gate
`ChaosHypothesis` CR	falsifiable prediction	code review
CI `hypothesis_check.py`	every engine has a card	pre-merge
Grammar `IF ... THEN ... FOR ...`	forces quantitative discipline	review-time
`falsifiable_by` block	maps 1:1 to probes	consistency check
Two reviewers	domain expertise	governance

Rule of thumb. No ChaosEngine ships to any environment without a hypothesis card. The hypothesis is the falsifiable prediction; the probes enforce the halt; the post-mortem records the outcome. Skip the hypothesis and you're not doing chaos engineering — you're doing chaos.

Worked example — the quarterly ingest-stack game day

Detailed explanation. The quarterly game day is where the chaos loop scales beyond one experiment. Teams gather for 90 minutes, run a pre-planned sequence of experiments against the whole ingest stack, watch the SLI dashboards live, and produce one post-mortem covering the ceremony. The ceremony has a fixed rhythm: 30-min pre-flight, 30-min chaos, 30-min post-mortem. Walk through the artifacts and cadence.

Attendees. GM (drives), 3 Observers (Kafka / Spark / Iceberg), Scribe, On-Call (shadow with halt authority), optional Product owner (learns).
Environment. Prod-1pct (early quarters) → Prod-10pct (later quarters). Never dev — dev doesn't produce actionable learning.
Pre-flight. SLI dashboards open, #chaos-live Slack post, hypothesis cards printed, halt authority confirmed.
Sequence. Three experiments (broker kill, executor loss, S3 latency) with 2-min settle between.
Post-mortem. Written same day; action items assigned; blast-radius graduation decision recorded.

Question. Design the quarterly ceremony agenda, the responsibilities of each role, and the post-mortem template.

Input.

Role	Responsibility
GM	apply ChaosEngine; drive tempo; call "next experiment"
Observer 1 (Kafka)	watch broker + lag + ISR panels; call anomalies
Observer 2 (Spark)	watch executor count + batch p99 + stage retries
Observer 3 (Iceberg)	watch commit p99 + write error rate + manifest count
Scribe	live-take notes into post-mortem doc
On-Call	shadow; halt authority; watches customer impact
Product	learns; represents customer perspective

Code.

<!-- game-day/2026-Q3-ingest-stack.md -->
# Ingest Stack Game Day — 2026-Q3

**Date.** 2026-07-30 14:00 - 15:30 UTC
**Environment.** prod-10pct
**GM.** Alice Chen (data-platform-sre)
**Halt authority.** Any observer via `#chaos-live` "HALT" message → GM runs `kubectl delete chaosengine`

## Pre-flight checklist (14:00 - 14:30)

- [ ] All observers on Zoom bridge
- [ ] SLI dashboards open in tabs:
  - [https://grafana/pipeline-freshness]
  - [https://grafana/kafka-brokers]
  - [https://grafana/spark-executors]
  - [https://grafana/iceberg-commits]
- [ ] `#chaos-live` posted with ceremony agenda + hypothesis cards
- [ ] `kubectl` context confirmed = `prod-10pct`
- [ ] Halt authority explicitly re-confirmed on the bridge
- [ ] Customer status page monitored (green expected)

## Experiment 1 — Kafka broker kill (14:30 - 14:35)

**Hypothesis.** IF one broker of six is deleted, THEN kafka_lag_messages stays < 100k AND freshness_p99_min stays < 15 AND min ISR stays >= 2 FOR 120s.

**Apply.** `kubectl apply -f chaos/overlays/prod-10pct/kafka-broker-kill.yaml`

**Observations.** (Scribe fills in live)
- Lag peak: ___
- Freshness peak: ___
- ISR minimum: ___
- Halt triggered? ___

**Verdict.** [ ] Pass  [ ] Fail  [ ] Halted

## Experiment 2 — Spark executor loss (14:38 - 14:45)

**Hypothesis.** IF 2 of 10 executors are deleted, THEN batch_p99 stays < 30s AND executor_count recovers to >= 8 within 60s AND max stage attempts stays < 4 FOR 180s.

...

## Experiment 3 — S3 latency (14:48 - 14:58)

...

## Post-mortem (15:00 - 15:30)

**Overall verdict.** ___ / 3 experiments Pass

**Deviations.**
| Experiment | Predicted | Actual | Root cause | Action item |
|---|---|---|---|---|
| ... | ... | ... | ... | ... |

**Blast-radius decisions.**
- Kafka broker kill: [ ] graduate to prod-100pct  [ ] repeat at prod-10pct  [ ] rollback to prod-1pct
- Spark executor loss: [ ] ...
- S3 latency: [ ] ...

**Action items (owner + ETA).**
1. ...
2. ...

**Follow-up date.** 2026-10-30 (next quarterly)

Step-by-step explanation.

The 90-min ceremony has a fixed shape: 30-min pre-flight, 30-min chaos, 30-min post-mortem. Deviating from the timing produces either "not enough setup" or "not enough learning" — the timings are load-bearing.
Each experiment has a printed hypothesis card that pastes verbatim into the doc; the observations table is filled in live by the scribe. This makes the ceremony reproducible — anyone reading the doc a year later can retrace what was tried and what happened.
The halt authority is deliberately human ("any observer via #chaos-live HALT message"). The automated probes catch SLI breaches; the humans catch business-impact signals that no probe covers (a support ticket, a Twitter mention, a customer email).
The post-mortem table forces attribution: for each deviation, name the root cause (config bug / code bug / genuine resilience gap) and file an action item. Without the table, deviations get forgotten.
The blast-radius decisions are the promotion signal for the next quarter's ceremony. Each experiment either graduates (next quarter runs at bigger radius), repeats (same radius, verify stability), or rolls back (fix a bug, retry at smaller radius). This is what makes chaos a compounding practice.

Output.

Artifact	Purpose	Cadence
Pre-flight checklist	operator readiness	per ceremony
Per-experiment hypothesis + observations	falsifiable outcome	per experiment
Post-mortem deviation table	attributed learning	per ceremony
Blast-radius decision matrix	graduation signal	per ceremony
Action items with owner + ETA	fix commitments	per deviation
Follow-up date	compounding cadence	per ceremony

Rule of thumb. Every game day produces exactly one written post-mortem, filed in a stable location, with one row per experiment and one action item per deviation. The doc is the compounding artifact — after 8 quarters you have 8 post-mortems that collectively tell the story of your pipeline's resilience trajectory.

Worked example — the "verify" step — SLI-before / SLI-during / SLI-after overlay

Detailed explanation. The verify step is where the hypothesis meets the data. Grafana overlays the SLI curves from a baseline (24h prior to chaos), the chaos window, and the post-chaos period. Deviations pop visually; agreement is boring (which is what you want). Every game-day doc includes screenshots of the overlays as the "here's what actually happened" evidence.

Baseline. Last 24h of the SLI, same window-of-day, same day-of-week.
Chaos. The exact chaos window with the ChaosEngine start / end times marked as vertical lines.
Post-chaos. 30 min after chaos ends — verify recovery is complete.
Overlay. Grafana time_range templating shows all three periods on the same panel.

Question. Design the Grafana panel that provides the verify-step evidence for a Kafka broker kill.

Input.

Panel	Metric	Overlay	Purpose
Consumer lag	`sum(kafka_consumergroup_lag)`	baseline vs chaos vs post	freshness verdict
Producer errors	`sum(rate(kafka_producer_record_error_total[1m]))`	baseline vs chaos	durability verdict
Min ISR	`min(kafka_topic_partition_in_sync_replica)`	chaos-only, annotated	correctness verdict
Broker count ready	`count(kube_pod_status_ready{pod=~"kafka-broker.*",condition="true"})`	chaos-only	infra recovery

Code.

// Grafana panel — Consumer lag with baseline overlay
{
  "type": "timeseries",
  "title": "Consumer lag — chaos overlay",
  "targets": [
    {
      "refId": "A",
      "expr": "sum(kafka_consumergroup_lag{group=\"warehouse-sink\"})",
      "legendFormat": "chaos",
      "datasource": "prometheus"
    },
    {
      "refId": "B",
      "expr": "sum(kafka_consumergroup_lag{group=\"warehouse-sink\"} offset 24h)",
      "legendFormat": "baseline (24h ago)",
      "datasource": "prometheus"
    }
  ],
  "options": {
    "legend": {"placement": "bottom"},
    "tooltip": {"mode": "multi"}
  },
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green",  "value": null},
          {"color": "yellow", "value": 50000},
          {"color": "red",    "value": 100000}
        ]
      }
    }
  },
  "annotations": {
    "list": [
      {
        "name": "chaos-window",
        "datasource": "prometheus",
        "expr": "ALERTS{alertname=\"ChaosEngineActive\", chaosengine=\"kafka-broker-kill\"}",
        "iconColor": "purple",
        "titleFormat": "chaos active",
        "tagKeys": "chaosengine"
      }
    ]
  }
}

# verify_generate.py — auto-generate the Grafana-URL for the verify step
# runs at ChaosResult completion; posts the URL to Slack + post-mortem doc
import subprocess, json, urllib.parse, requests

def get_chaosresult(name, ns):
    r = subprocess.check_output(["kubectl", "-n", ns, "get", "chaosresult",
                                  name, "-o", "json"])
    return json.loads(r)

def build_grafana_url(dashboard_uid, start_epoch_ms, end_epoch_ms):
    base = "https://grafana.internal/d/" + dashboard_uid
    # 30 min pre-chaos, chaos window, 30 min post-chaos
    from_ms = start_epoch_ms - 30*60*1000
    to_ms   = end_epoch_ms   + 30*60*1000
    qs = urllib.parse.urlencode({
        "from": from_ms,
        "to":   to_ms,
        "var-chaosengine": "kafka-broker-kill",
    })
    return f"{base}?{qs}"

def post_verify_link(chaosresult):
    exp   = chaosresult["status"]["experimentStatus"]
    name  = chaosresult["metadata"]["name"]
    start = int(exp["startTimestamp"])
    end   = int(exp["endTimestamp"])
    url = build_grafana_url("ingest-chaos-overlay", start*1000, end*1000)
    requests.post("https://hooks.slack.com/services/xxx", json={
        "text": f":mag: Verify-step overlay for {name}: {url}"
    })

if __name__ == "__main__":
    cr = get_chaosresult("kafka-broker-kill-pod-delete", "kafka")
    post_verify_link(cr)

Step-by-step explanation.

The Grafana panel plots two Prometheus series on the same y-axis: kafka_consumergroup_lag (chaos) and ... offset 24h (baseline). The offset shifts the query 24h back so we compare "now" to "same time yesterday" — same window-of-day traffic pattern.
The chaos-window annotation is driven by a Prometheus alert ChaosEngineActive fired by an ancillary exporter that watches for active ChaosEngine CRs. When the CR is active, the alert is firing, and Grafana overlays a shaded region on the panel.
The verify_generate.py script watches ChaosResult CRs; when one completes, it extracts start/end timestamps and generates a Grafana URL scoped to the chaos window ± 30 min. The URL posts to #chaos-live and lands in the post-mortem doc.
The 30-min windows before/after the chaos give visual context — you can see the baseline SLI curve, the chaos-induced deviation, and the recovery back to baseline. The recovery segment is often more informative than the chaos segment; a slow recovery means the pipeline was more fragile than the point-in-time SLI suggested.
The three-color threshold (green/yellow/red) on the y-axis gives an at-a-glance verdict — if the chaos curve stays in green throughout, hypothesis confirmed; if it dips into red, hypothesis falsified. The threshold values (50k warn, 100k halt) match the promProbe values.

Output.

Panel component	Purpose
Chaos series (query A)	actual behaviour under fault
Baseline series (offset 24h)	what "normal" looks like
Chaos-window annotation	visual scope of the experiment
Threshold bands	SLI verdict at a glance
Auto-generated URL	shareable evidence for post-mortem
Slack post	live operator awareness

Rule of thumb. Every chaos experiment ships with a paired Grafana panel that overlays chaos-vs-baseline SLIs with the chaos window annotated. The panel URL lands in the post-mortem doc as the "verify" evidence. Without the overlay, "verify" degenerates into "we watched the dashboard and it seemed fine."

Senior interview question on the chaos loop

A senior interviewer might ask: "Design the chaos-engineering practice for a new data platform team from scratch. Include the SLI catalog, the hypothesis format, the graduation ladder, the game-day ceremony, the tooling choice (LitmusChaos vs Gremlin), and how you'd measure the practice's own maturity over time."

Solution Using an SLI-first hypothesis-driven program with quarterly ceremonies and a maturity ladder

# chaos_maturity.py — score the practice itself, quarterly
# runs against the chaos GitOps repo + ChaosResult history

CRITERIA = [
    ("SLI catalog exists",           "grep -r 'record: pipeline:' chaos/base/_slis.yaml"),
    ("Every engine has a hypothesis card", "python hypothesis_check.py"),
    ("Every engine has continuous probes", "python probe_check.py"),
    ("Every hypothesis card is falsifiable", "python falsifiable_check.py"),
    ("Nightly dev chaos runs",       "kubectl get cronjob chaos-dev -n chaos"),
    ("Weekly staging chaos runs",    "kubectl get cronjob chaos-staging -n chaos"),
    ("Monthly prod-1pct ceremony",   "gh api repos/acme/chaos/actions/workflows/prod-1pct.yml"),
    ("Quarterly prod-10pct ceremony", "gh api repos/acme/chaos/actions/workflows/prod-10pct.yml"),
    ("Post-mortem doc per ceremony", "ls game-day/*.md | wc -l"),
    ("Action-item follow-through > 80%", "python action_item_check.py --window 90d"),
    ("Blast-radius graduation events > 0", "python graduation_check.py --window 90d"),
    ("SIEM audit trail",             "curl -s splunk-hec/health"),
]

def score():
    passed, total = 0, len(CRITERIA)
    for name, cmd in CRITERIA:
        try:
            subprocess.check_output(cmd, shell=True, stderr=subprocess.DEVNULL)
            passed += 1
            print(f"  [x] {name}")
        except subprocess.CalledProcessError:
            print(f"  [ ] {name}")
    print(f"\nMaturity: {passed}/{total}")
    return passed, total

if __name__ == "__main__":
    score()

# CronJob — the practice's own heartbeat
apiVersion: batch/v1
kind: CronJob
metadata:
  name: chaos-maturity-score
  namespace: chaos
spec:
  schedule: '0 9 1 */3 *'   # quarterly, 9am on the 1st
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: score
              image: acme/chaos-maturity:latest
              command: ['python', '/app/chaos_maturity.py']
          restartPolicy: OnFailure

Step-by-step trace.

Layer	Component	Responsibility
SLI catalog	`_slis.yaml`	one recording rule per pipeline SLI
Hypothesis cards	`hypothesis-cards/*.yaml`	one per ChaosEngine; enforced by CI
ChaosEngine CRs	`engines/*.yaml`	one per experiment; probes reference SLIs
Blast-radius overlays	`overlays/{dev,staging,prod-*}/`	per-stage promotion
Game-day doc	`game-day/*.md`	one per ceremony; scribe live-writes
CI checks	`hypothesis_check.py`, `probe_check.py`, `falsifiable_check.py`	pre-merge gates
Maturity CronJob	quarterly	scores the practice itself
SIEM export	`siem_exporter.py`	attack audit trail

After deployment, the practice grades itself quarterly. A team scoring 12/12 has a mature chaos practice: SLI catalog, hypothesis-driven experiments, probe-gated halt, graduated blast radius, written post-mortems, follow-through on action items, audit trail. A team scoring 4/12 has the tools but not the discipline — usually because they skipped the hypothesis step and can't produce a verdict.

Output:

Practice level	Score	Meaning
0-3	Absent	no chaos program
4-6	Tooling-only	ran experiments; no discipline
7-9	Hypothesis-driven	SLIs + hypotheses + halts; no ceremony
10-11	Ceremony-driven	quarterly game days + post-mortems
12	Compounding	maturity score itself is monitored

Why this works — concept by concept:

SLI catalog before any experiment — the practice starts by defining what "healthy" means. The catalog is the single source of truth referenced by every probe and every hypothesis card. Without it, each experiment invents its own criteria and comparability dies.
Hypothesis-driven experiments — every experiment ships with a falsifiable prediction. CI enforces the pairing (hypothesis_check.py), the grammar (falsifiable_check.py), and the probe consistency (probe_check.py). Chaos becomes engineering, not stunts.
Graduation ladder with matrix-gated promotion — dev → staging → prod-1pct → prod-10pct → prod-100pct, with N green runs required at each stage. CI enforces the counts; CODEOWNERS enforces the human approval. Prod chaos is earned, not granted.
Quarterly game-day ceremony — the human ritual that reads the SLI evidence, discusses the deviations, and decides the next quarter's blast-radius adjustments. Ceremony frequency is the practice's heartbeat; skipping ceremonies collapses the compounding.
Written post-mortem per ceremony — the doc is the compounding artifact. After 8 quarters you have 8 post-mortems that collectively tell the story of your pipeline's resilience trajectory. New team members read the docs to onboard.
Action-item follow-through as an SLI — the practice measures whether its own action items are being closed. A team that files 20 action items and closes 3 is doing theatre; a team that files 8 and closes 7 is doing engineering. The 80% threshold in chaos_maturity.py is the guard.
Maturity score as a monitored SLI — the practice grades itself. When maturity drops (a ceremony was skipped, a check regressed), the CronJob output triggers a Slack alert. The practice's own health becomes a metric on the same dashboard as pipeline health.
Cost — one quarterly ceremony (7.5 engineer-hours) + nightly/weekly automation (~0 marginal ops) + one CI check bundle (~5 minutes per PR) + one CronJob (~1 minute per quarter). Compared to the cost of one unplanned incident, this is a 100× positive-ROI trade. O(1) per ceremony; O(quarterly) per pipeline; O(annual) per program review.

Real-time
Topic — real-time-analytics
Real-time analytics problems on SLI-first design

Practice →

Data-Validation
Topic — data-validation
Data validation problems on post-chaos correctness

Practice →

Cheat sheet — chaos engineering data recipes

Which tool when. LitmusChaos is the 2026 default when your data plane is Kubernetes-native and you want everything GitOps-managed — free, CNCF, ChaosEngine CR + probe guardrails + ChaosResult verdict. Gremlin is the choice when your data plane spans VMs / on-prem / ECS / K8s and you want one SaaS control plane with SOC-2 audit, richer network/state primitives (DNS, time-travel, blackhole), and a Terraform provider. Pick both if your platform has both surfaces — LitmusChaos for K8s workloads, Gremlin for cross-platform infra chaos.
Steady-state SLI catalog template. Define four Prometheus recording rules per pipeline: pipeline:freshness_p99_min (histogram_quantile of commit_age_seconds / 60), pipeline:kafka_lag_messages (sum of consumer group lag), pipeline:batch_success_rate_5m (success / total ratio over 5-min window), pipeline:row_count_delta_1h (source − sink parity). Every probe on every ChaosEngine references these — one source of truth for "the pipeline is healthy."
Halt-condition template. Halt criterion = promProbe in Continuous mode with stopOnFailure: true, threshold set 2-3× tighter than the customer-facing SLO (so chaos aborts before customers notice), probeTimeout: 5, interval: 10, retry: 1 (short retry — one flaky query shouldn't abort). Pair with an EOT cmdProbe for correctness invariants Prometheus can't express (row counts, orphaned rows, manifest integrity).
LitmusChaos ChaosEngine template. apiVersion: litmuschaos.io/v1alpha1, spec.chaosServiceAccount: litmus-admin, spec.appinfo.applabel selector, experiments[0].name: pod-delete (or pod-network-loss, disk-fill, pod-cpu-hog), env TOTAL_CHAOS_DURATION + PODS_AFFECTED_PERC + FORCE: 'false' (graceful), probe[] with 2-3 promProbe + optional k8sProbe + optional cmdProbe. RBAC-scope the runner service account to the target namespace only.
Gremlin scenario YAML template. scenario.name + hypothesis (falsifiable), graph.nodes.{start,attack,cooldown} with InfraAttack type + impactDefinition (kind, attackType, params, percent blast radius, targets.tags selector), healthChecks[] with provider: datadog|webhook|newrelic|prometheus and haltOnUnhealthy: true. Version via Terraform gremlin_scenario resource; mirror every attack to SIEM via the API for audit.
Kafka broker chaos guardrails. Topic: replication.factor >= 3, min.insync.replicas = 2, unclean.leader.election.enable = false. Producer: acks = all, enable.idempotence = true, retries = Integer.MAX_VALUE, max.in.flight.requests.per.connection <= 5. Chaos: PODS_AFFECTED_PERC <= 17 (1 of 6), FORCE: 'false', halt on kafka_consumergroup_lag > 100000 OR min(in_sync_replica) < 2 OR rate(kafka_producer_record_error_total[1m]) > 0.1.
Airflow scheduler chaos guardrails. DAG: catchup=False, max_active_runs=1, retries >= 2, dagrun_timeout set. Chaos: pod-delete on component=scheduler with FORCE: 'false'. Halt on k8sProbe on scheduler Deployment ready + EOT cmdProbe SQL SELECT count(*) FROM task_instance WHERE state='running' AND queued_dttm < now() - INTERVAL '10 minutes' = 0 (orphan check) + EOT promProbe on airflow_dagrun_active_count <= max_active_runs (no catchup flood).
Spark executor chaos guardrails. Spark submit: dynamicAllocation.enabled=true, dynamicAllocation.minExecutors=<floor>, blacklist.enabled=true, blacklist.timeout=1h, task.maxFailures=4, stage.maxConsecutiveAttempts=4, shuffle.service.enabled=true (external shuffle service DaemonSet or NodeManager aux). Chaos: PODS_AFFECTED_PERC <= 20 on app=spark-executor. Halt on batch_processing_p99 > 30s + EOT check active_executor_count >= minExecutors + max_stage_consecutive_attempts < 4.
Blast-radius graduation matrix. Dev = 100% (disposable), 5 green runs to promote; Staging = same percent as prod-target (17% brokers, 20% executors, 25% writers), 3 green weekly runs to promote; Prod-1pct = 1 broker of 6 / 2 executors of 20 / 1% writers, monthly, 3 green ceremonies to promote; Prod-10pct = 10% of tier-1, quarterly, 2 green ceremonies to promote; Prod-100pct = annual DR drill only. CI script counts green runs; CODEOWNERS enforces human approval per stage.
Game-day ceremony script. 90 minutes total: 30-min pre-flight (SLI dashboards open, #chaos-live posted, halt authority re-confirmed, kubectl context verified) → 30-min chaos (three composed experiments with 2-min settle gaps, scribe live-writes observations, any observer can shout HALT) → 30-min post-mortem (deviation table, root cause per deviation, blast-radius decisions, action items with owner + ETA, follow-up date). Doc lives in Confluence at a stable URL.
Hypothesis card template. IF <fault at blast radius X> THEN <SLI Y> stays <comparator threshold Z> AND <SLI Y'> stays <comparator threshold Z'> FOR <duration>. Every noun quantitative. Every SLI a Prometheus rule reference. Every falsifiable_by entry maps 1:1 to a promProbe. Two named reviewers, one with production experience on the specific system.
SIEM audit trail. Every chaos attack (LitmusChaos ChaosResult, Gremlin attack event) mirrors to your SIEM (Splunk, Datadog, Elastic) with fields: {attack_id, scenario, user, targets, tags, started_at, ended_at, halted, verdict, probe_timeline}. Retention: 1 year minimum (audit typical); 7 years for SOC-2 environments. Alert on any prod-scope chaos with no matching approval PR — that's an unauthorised attack.
Post-mortem action-item follow-through. Track action-item close-rate as a chaos-practice SLI: (closed_within_90d / total_filed) > 80% is healthy; < 50% is theatre. Report the ratio in the quarterly maturity score; regress the ratio into the team's OKRs. An unclosed action item is a resilience gap that will surface as a real incident.
Chaos maturity ladder. 0-3 criteria = absent (no chaos program); 4-6 = tooling-only (running experiments, no discipline); 7-9 = hypothesis-driven (SLIs + hypotheses + halts, no ceremony); 10-11 = ceremony-driven (quarterly game days + written post-mortems); 12 = compounding (maturity score itself monitored, action-item follow-through > 80%, blast radius graduating quarter over quarter). Score the practice quarterly; target one level per year.

Frequently asked questions

What is chaos engineering for data pipelines in one sentence?

Chaos engineering for data pipelines is the discipline of deliberately injecting infrastructure failures — Kafka broker kills, Airflow scheduler restarts, Spark executor loss, S3 latency spikes, disk fills, network partitions — into a running pipeline while a codified data SLO (freshness_p99, end-to-end lag, DAG success rate, source-to-sink row parity) acts as the steady state that decides whether the experiment passes or halts, all inside a bounded blast radius that graduates from dev → staging → 1% prod → 10% prod → 100% prod. The four canonical tools in 2026 — litmuschaos (K8s-native, open source, ChaosEngine + probe CRs), gremlin (SaaS, cross-platform, scenarios + blast-radius dial + halt-on-breach), plus the data-plane primitives (kafka chaos broker kill, airflow chaos scheduler kill, spark chaos executor loss) — differ in deployment surface and fault library but converge on the same four-step loop: steady state → hypothesis → experiment → verify. Getting the axes right (data-first SLI, falsifiable hypothesis, halt-on-breach probe, written post-mortem) is what separates a mature practice from a stunt, and it's the load-bearing skill senior data platform interviewers probe.

LitmusChaos vs Gremlin — when do I pick each?

Default to LitmusChaos when your data plane runs entirely on Kubernetes and you want everything GitOps-managed — it's CNCF-graduated, Apache-2.0, and models chaos as first-class Kubernetes custom resources (ChaosEngine, ChaosExperiment, ChaosResult) with a probe system (httpProbe, promProbe, cmdProbe, k8sProbe) that auto-aborts on breach. Ship it via Argo CD, RBAC-scope the runner to the target namespace, and the entire experiment lives in Git. Pick Gremlin when your data plane spans VMs, on-prem, ECS, Databricks, mixed cloud — a single SaaS control plane with lightweight gremlind agents on every target host, richer network/state attack primitives (DNS, blackhole, time-travel), and SOC-2-friendly SIEM-exportable audit logs. Gremlin is commercial (per-agent-per-month licence), so the trade is money-for-time-to-value vs LitmusChaos's ops-cost-for-flexibility. Larger platforms often run both — LitmusChaos for the K8s data plane, Gremlin for the cross-platform infra chaos — with shared Datadog / Prometheus SLIs as the common halt criteria.

How do I define steady state for a data pipeline?

Steady state must be a data SLO, not an infra metric. For a Kafka → Spark Structured Streaming → Iceberg → warehouse stack, four Prometheus recording rules define steady state: pipeline:freshness_p99_min (99th percentile age in minutes of the newest committed record vs wall clock), pipeline:kafka_lag_messages (sum of consumer group lag across all partitions), pipeline:batch_success_rate_5m (successful streaming batches / total batches over a 5-min window), and pipeline:row_count_delta_1h (source events count − sink rows count over the last hour). Every chaos probe references these — no pod_ready, no cpu_percent < 80, because a pod can be Ready while the pipeline is 40 minutes behind. Set halt thresholds 2-3× tighter than the customer-facing SLO (chaos aborts before customers notice) and pair Continuous promProbe guards with EOT cmdProbe checks for correctness invariants Prometheus can't express (orphaned task_instance rows, Iceberg manifest integrity, tombstone parity). If you can't articulate "the pipeline is healthy" in a single Prometheus query, you're not ready to inject chaos.

What's a safe blast radius for a Kafka broker kill?

The 2026 answer is one broker at a time, scoped via PODS_AFFECTED_PERC: '17' on a 6-broker cluster (or the equivalent selector on your topology), with a mandatory topic contract of replication.factor >= 3 + min.insync.replicas = 2 + unclean.leader.election.enable = false and a producer contract of acks = all + enable.idempotence = true. Under those settings, killing one broker leaves each affected partition with 2 in-sync replicas — enough to satisfy min.insync.replicas and continue accepting writes; the controller re-elects a new leader from the remaining ISR within seconds and the producer's retry loop absorbs the transient errors. Halt on kafka_consumergroup_lag > 100000 OR min(in_sync_replica) < 2 OR rate(kafka_producer_record_error_total[1m]) > 0.1 — any of these means the guardrails failed and further chaos will lose writes. Never kill the controller in the same experiment (label the controller pod separately and exclude via applabel: 'app=kafka-broker,role!=controller'); controller loss is a distinct experiment with different SLIs (controller failover time SLO).

Should I run chaos engineering in production?

Yes, but only after the graduation ladder is climbed and the halt criteria are proven. The mature 2026 pattern runs chaos through dev → staging → prod-1pct → prod-10pct → prod-100pct with N green runs required at each stage before promotion (5, 3, 3, 2, 1 respectively) and CI-gated + CODEOWNERS-approved PRs promoting each experiment between stages. Prod-1pct chaos is a monthly ceremony in a game-day format (GM, observers, scribe, on-call with halt authority); prod-10pct is quarterly with an executive sponsor; prod-100pct is reserved for annual DR drills with a formal runbook and customer comms. Every prod experiment ships with stopOnFailure: true promProbe guards tied to data SLIs 2-3× tighter than customer-facing SLOs and mirrors every attack to your SIEM (Splunk, Datadog, Elastic) for audit. The alternative to production chaos is not zero chaos — it's unplanned chaos when a real incident hits an untested failure mode. Scheduled, bounded, halted chaos is quantifiably safer than the incident lottery.

How is chaos engineering different from load testing?

Load testing measures capacity — how many requests per second, how many concurrent users, how big a batch — the pipeline can handle before latency/throughput degrades. Chaos engineering measures availability — whether the pipeline stays inside its SLO when a specific infrastructure component fails. Both are essential and both are complementary, but they answer different questions: load testing answers "how much load survives?" and chaos answers "which failure modes survive?" Load tests typically run in a dedicated environment with synthetic traffic; chaos experiments typically run in real environments with real traffic and inject unavailability rather than volume. A pipeline can pass a load test at 10× peak and still fail a chaos experiment because a Kafka broker kill exposes a min.insync.replicas misconfig that only surfaces during failure. In the 2026 senior interview, describing chaos as "load testing but for failures" is a weak answer; describing it as "hypothesis-driven falsification of resilience assumptions under bounded infrastructure faults with SLO-tied halt criteria" is a senior signal.

Practice on PipeCode

Drill the streaming practice library → for the Kafka broker chaos, ISR contract, and consumer-lag-under-failure problems senior interviewers love.
Rehearse on the ETL practice library → for the Airflow scheduler chaos, DAG catchup, and orphaned task_instance recovery patterns.
Sharpen the architecture axis with the design practice library → for graduated blast radius, game-day ceremony design, and SLO-first steady-state selection.
Practise the event-processing library → for exactly-once contracts, coordinator loss, and idempotent consumer patterns under chaos.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-step chaos loop against real graded inputs.

Lock in chaos engineering muscle memory

Docs explain chaos tools. PipeCode drills explain the decision — when a `promProbe` should halt, when `min.insync.replicas` earns its config line, when Airflow's `catchup=False` prevents a queue flood, when Spark's `dynamicAllocation` saves an executor-loss experiment, when a hypothesis is falsifiable versus vague. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data platform engineers and SREs actually face.

Practice streaming problems →
Practice design problems →

CI/CD for dbt + Airflow + Spark: Slim CI, State Comparison, Preview Environments

Gowtham Potureddi — Fri, 31 Jul 2026 14:41:26 +0000

dbt ci cd is the load-bearing engineering discipline that decides whether every pull request to an analytics-engineering monorepo is a 90-second confidence signal or a 45-minute warehouse-burning re-run — and it is the single subsystem senior analytics engineers most often ship half-done, because "we already have GitHub Actions" is not the same as CI that understands manifest.json, deferral, and zero-copy clones. A modern data platform ships three separately-versioned artifacts on every merge — a dbt project, a set of Airflow DAGs, and one or more Spark jobs — each of which mutates warehouse state, and each of which has its own axis of "what does it mean for this PR to be safe to merge?" that generic web-app CI/CD (unit tests + docker build + push) simply does not model. The engineering trade-off does not live in "should we run CI on data pipelines" — every team past two engineers needs it — but in how you scope each run so the fast path is fast, the slow path is honest, and the warehouse bill in November does not triple because CI decided to rebuild the entire mart on every whitespace-only PR.

This guide is the senior data-platform walkthrough you wished existed the first time an interviewer asked "walk me through slim CI for dbt and why state:modified+ matters", or "your PR only edits one model — why did the CI job just spend $180 on Snowflake compute?", or "how do you give analysts a preview URL for the dashboard they're editing without cloning the entire warehouse?". It covers the four canonical CI axes (correctness, cost, latency-to-merge, blast radius), the slim CI recipe with dbt build --select state:modified+ --defer --state prod-manifest/ plus dbt clone for zero-copy previews, the Airflow DAG-parse gate + operator unit-test matrix that catches import-time breaks before the scheduler ever sees them, the Spark packaged-JAR + local pytest + dry-run submit sequence, and the preview-environment pattern where every open pull request maps to its own pr_<PR>_<branch> schema on Snowflake or Databricks Unity Catalog that is auto-created on PR open and auto-dropped on PR close. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the SQL practice library →, and sharpen the systems axis with the design practice library →.

On this page

Why generic CI/CD breaks on data pipelines
Slim CI for dbt — state:modified, defer, and clone
Airflow CI — DAG parse, integration tests, deployable artifacts
Spark CI — packaged jobs, unit + integration tests, dry-run submits
Preview environments — one branch, one warehouse schema
Cheat sheet — CI/CD recipes for dbt + Airflow + Spark
Frequently asked questions
Practice on PipeCode

1. Why generic CI/CD breaks on data pipelines — the four axes

Four systems, four different notions of "safe to merge" — the invariant every senior architect encodes

The one-sentence invariant: ci for data pipelines is a picking exercise across four systems (dbt, Airflow, Spark, and the warehouse itself), each with its own definition of "the PR is safe" — dbt needs state:modified+ deferral so the CI job rebuilds only changed models against parent tables in prod, Airflow needs a DAG-parse gate so import-time errors never reach the scheduler, Spark needs a packaged job + local unit tests + a dry-run submit so the cluster is only touched after config is validated, and the warehouse needs a per-PR preview schema so blast radius stays bounded — and shipping any of the four without its native gate turns CI into either "45-minute full rebuild theatre" or "green build, broken prod". The default you pick in month one becomes the default you fight to migrate away from in year three, because every analyst dashboard, every Airflow DAG, and every Spark job hard-codes assumptions about where the "safe to merge" line lives.

The four axes generic CI/CD misses.

Correctness. Web-app CI answers "did the unit tests pass?" Data-pipeline CI has to answer and "does the new dbt model actually produce the same PK count as the old one?", "does the DAG still parse under the current Airflow version?", "does the Spark job's schema still match the upstream Kafka topic?". Each system's correctness gate is different; encoding all three is the senior-signal answer.
Cost. A full dbt build against Snowflake for a 200-model warehouse can cost $50-$500 per CI run. Multiply by 30 PRs/day and the CI compute bill dwarfs the actual production bill. Slim CI + preview schemas + XS warehouses are how you keep the November invoice under control. Interviewers open with this question because it separates people who have paid the bill from those who have not.
Latency-to-merge. A CI run that takes 45 minutes turns PR review into a next-day cycle. A CI run that takes 90 seconds keeps analysts in flow. The dbt slim-CI + Airflow parse-gate + Spark local-unit-test sequence is designed for the 90-second floor; skipping any layer pushes latency back into the 20-45 minute regime.
Blast radius. Web-app CI writes to a docker layer cache and a test bucket. Data-pipeline CI writes to actual warehouse schemas — if you don't isolate per PR, one CI job's dbt seed overwrites another CI job's fixture and both fail. The preview-schema pattern gives every PR its own pr_<PR>_<branch> namespace so blast radius is exactly one PR.

The 2026 reality — slim CI everywhere, state comparison, preview environments.

Slim CI is the default for dbt. dbt build --select state:modified+ --defer --state prod-manifest/ is the four-word answer. Every senior interview lists it in the first minute. Rebuilding only changed models against prod parents cuts CI from 45 minutes to 2 minutes on a typical monorepo.
State comparison is the default for artifact promotion. The manifest.json from the last prod run is downloaded as an artifact; the CI job compares its own manifest against it and computes the state:modified set. Without prod manifest, slim CI does not work.
Preview environments are the default for warehouse UX. Every PR gets a pr_<PR>_<branch> schema (Snowflake CREATE SCHEMA ... CLONE, Databricks CREATE SCHEMA ... CLONE). Analysts hit the preview URL; when the PR merges or closes, the schema is dropped. Zero-copy clones make this cheap.
Airflow parse gates are non-negotiable. A DAG that fails at import time takes down every DAG in the DagBag. The parse gate is a one-line CI check that catches it before the scheduler ever tries to load it.
Spark dry-run submits are the last-mile gate. spark-submit --deploy-mode client --num-executors 0 (or the equivalent Databricks JOBS API dry-run) validates config, JAR resolution, and cluster reachability without actually spinning up executors — the cheapest possible "does this job actually work in this cluster" gate.

What interviewers listen for.

Do you name all four systems (dbt, Airflow, Spark, warehouse) and their per-system gates without prompting? — senior signal.
Do you say "slim CI with state:modified+ and deferral" in the first minute when dbt CI comes up? — required answer.
Do you push back on "just run the full test suite" with the cost argument — "that's a $180 CI run per PR"? — senior signal.
Do you name the preview-environment pattern as the answer to "how do analysts review a PR"? — senior signal.
Do you describe CI as "gates that shrink blast radius" rather than as vague "running tests"? — required answer.

Worked example — the four-axis comparison table across dbt, Airflow, Spark

Detailed explanation. The single most useful artifact for a data-pipeline CI/CD interview is a memorised 4×3 comparison table: four axes (correctness, cost, latency, blast radius) crossed with three systems (dbt, Airflow, Spark). Every senior CI/CD discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical analytics-engineering monorepo.

Repo shape. One monorepo containing dbt/, airflow/dags/, spark/jobs/. Deploys to Snowflake (warehouse), MWAA (Airflow), and EMR-on-EKS (Spark).
PR volume. ~30 pull requests per day; ~5 touch dbt, ~5 touch Airflow, ~2 touch Spark, the rest touch shared libraries.
Target CI budget. Under 5 minutes p95 for a dbt-only PR; under 8 minutes for a mixed PR; under $10 per CI run.

Question. Build the four-axis CI comparison for dbt, Airflow, and Spark and pick the gate each system needs.

Input.

Axis	dbt	Airflow	Spark
Correctness gate	`state:modified+` rebuild + `dbt test`	DAG parse + operator unit tests	pytest with local SparkSession
Cost gate	slim CI + XS warehouse	container reuse + shared runner	local mode + `--num-executors 0` dry-run
Latency gate	2 min for slim, 45 min for full	30 s parse, 3 min unit tests	4 min local pytest, 30 s dry-run
Blast radius gate	`pr_<PR>_<branch>` schema	separate MWAA environment	temp S3 prefix + separate EMR cluster

Code.

# .github/workflows/data-pipeline-ci.yml
name: data-pipeline-ci
on:
  pull_request:
    branches: [ main ]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      dbt:     ${{ steps.filter.outputs.dbt }}
      airflow: ${{ steps.filter.outputs.airflow }}
      spark:   ${{ steps.filter.outputs.spark }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            dbt:
              - 'dbt/**'
            airflow:
              - 'airflow/**'
            spark:
              - 'spark/**'

  dbt-slim-ci:
    needs: detect-changes
    if: needs.detect-changes.outputs.dbt == 'true'
    uses: ./.github/workflows/dbt-slim.yml

  airflow-parse-and-test:
    needs: detect-changes
    if: needs.detect-changes.outputs.airflow == 'true'
    uses: ./.github/workflows/airflow.yml

  spark-unit-and-dry-run:
    needs: detect-changes
    if: needs.detect-changes.outputs.spark == 'true'
    uses: ./.github/workflows/spark.yml

Step-by-step explanation.

The top-level workflow gates every downstream job on a paths-filter step so a whitespace-only PR to README.md does not trigger a dbt build. Path-based short-circuiting is the cheapest CI optimisation and pays for itself the first day.
Each system gets its own reusable workflow file. Isolating dbt, Airflow, and Spark into separate files keeps the top-level data-pipeline-ci.yml readable and lets each team own its own gate logic without cross-editing.
The dbt workflow (covered in section 2) runs slim CI against a preview schema. It downloads the prod manifest.json, computes state:modified+, defers parents to prod, and rebuilds only what changed.
The Airflow workflow (covered in section 3) runs the DAG parse gate first (fastest fail), then the operator unit tests, then optionally an integration test against a LocalExecutor.
The Spark workflow (covered in section 4) runs local unit tests with pyspark.testing, packages the job into a wheel or JAR, and runs a dry-run submit against a small dev cluster to validate config.

Output.

PR touches	Jobs that run	p95 CI time	Approximate cost
dbt only	dbt-slim-ci	2 min	$0.30
Airflow only	airflow-parse-and-test	3 min	$0.05
Spark only	spark-unit-and-dry-run	4 min	$0.08
Mixed (dbt + Airflow)	both	4 min (parallel)	$0.35
README only	none	0 s	$0.00

Rule of thumb. Never run a dbt build on a PR that does not touch dbt. Path-based gating is the single most impactful CI optimisation for a data-pipeline monorepo. Combine it with slim CI, DAG parse gates, and Spark dry-runs and a mixed-PR run finishes in under 5 minutes.

Worked example — what interviewers actually probe on data-pipeline CI/CD

Detailed explanation. The senior data-platform CI/CD interview has a predictable structure: the interviewer opens with an ambiguous question ("how do you set up CI for a dbt project?"), then progressively narrows to test whether you know the axes. The candidates who name slim CI and state comparison in sentence one score highest; the candidates who describe "GitHub Actions running dbt run" score lowest. Walk through the interview grading rubric.

Ambiguous opener. "How would you set up CI for our dbt project?" — invites you to name slim CI.
Follow-up 1. "What happens when the PR only edits one model?" — probes state:modified.
Follow-up 2. "How do you avoid rebuilding all parent tables?" — probes deferral.
Follow-up 3. "How much does a CI run cost?" — probes cost axis.
Follow-up 4. "How do analysts review the changed dashboard?" — probes preview environments.
Follow-up 5. "What about Airflow and Spark in the same repo?" — probes multi-system gates.

Question. Draft a 5-minute senior CI/CD answer that covers all three systems and all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
dbt gate named	"we run dbt run in CI"	"slim CI with `dbt build --select state:modified+ --defer --state prod-manifest/`"
Airflow gate named	"we run pytest"	"DAG parse gate first, then operator unit tests, then integration test with LocalExecutor"
Spark gate named	"we submit the job"	"local pytest + packaged JAR + dry-run submit against dev cluster"
Cost bound	"not sure"	"under $1/run, XS warehouse with AUTO_SUSPEND=60"
Preview UX	"analysts wait for merge"	"`pr_<PR>_<branch>` schema created on PR open, dropped on close"

Code.

Senior data-pipeline CI/CD answer template (5 minutes)
======================================================

Minute 1 — name the gates up front
  "For each of the three systems in a data monorepo, CI has a native
   gate. dbt: slim CI with state:modified+ and deferral. Airflow:
   DAG parse gate + operator unit tests. Spark: local pytest +
   dry-run submit."

Minute 2 — dbt slim CI
  "The command is: dbt build --select state:modified+ --defer
   --state prod-manifest/ --target ci. The prod manifest.json is
   downloaded from S3 (uploaded on every prod run). state:modified+
   picks the changed models and their descendants; --defer routes
   any unbuilt parent refs to the prod schema instead of the CI
   schema. This turns a 45-minute full rebuild into a 2-minute
   incremental."

Minute 3 — Airflow CI
  "First gate is a DagBag import: python -c 'from airflow.models
   import DagBag; assert not DagBag(dag_folder=\"airflow/dags\",
   include_examples=False).import_errors'. Fails in 15 seconds if
   any DAG file breaks at import. Second gate is unit tests on
   custom operators and hooks. Third gate is an integration test
   with LocalExecutor for the DAGs that touch shared resources."

Minute 4 — Spark CI + preview envs
  "Spark: pytest against a local SparkSession fixture for logic,
   docker-compose Spark mini-cluster for schema and shuffle behaviour,
   and spark-submit --deploy-mode client --num-executors 0 as a
   dry-run gate that validates cluster config without spinning up
   executors. Preview envs: every PR gets a pr_<PR>_<branch> schema
   via Snowflake CREATE SCHEMA ... CLONE, with AUTO_SUSPEND=60 on the
   compute; dropped on PR close via GitHub webhook."

Minute 5 — cost + blast radius
  "Cost: slim CI + XS warehouse + AUTO_SUSPEND=60 keeps a dbt CI run
   under $0.30 typically. Blast radius: per-PR schema means one CI
   job cannot corrupt another. On PR close, the cleanup workflow
   drops the schema and de-registers the compute. November invoice
   is bounded because ephemeral compute suspends within 60 s of the
   last query."

Step-by-step explanation.

Minute 1 is the crucial framing. Naming the three system-specific gates immediately signals you understand the axes; weak candidates dive into a single tool ("we use GitHub Actions and…") before naming any gate.
Minute 2 addresses the dbt cost axis with the exact state:modified+ --defer command. This preempts the "how do you avoid rebuilding all parents?" follow-up by answering it before it is asked.
Minute 3 addresses the Airflow correctness axis: the DAG parse gate is the fast-fail; the operator unit tests are the coverage layer; the integration test is the "does the whole DAG wire together?" layer. Naming all three shows you have shipped Airflow CI, not just read the docs.
Minute 4 combines Spark and preview environments into one minute because they share a design theme: cheap ephemeral compute that suspends when idle. --num-executors 0 for Spark and AUTO_SUSPEND=60 for Snowflake are both "compute suspends when unused" patterns.
Minute 5 covers the cost + blast-radius axes explicitly. Naming the November invoice concern signals you have paid the bill; naming the auto-cleanup on PR close signals you have on-called the workflow.

Output.

Grading criterion	Weak score	Senior score
Names all four axes	rare	mandatory
Names slim CI in minute 1	rare	required
Names DAG parse gate	occasional	mandatory
Names dry-run Spark submit	rare	senior signal
Names preview env + cleanup	rare	senior signal
Names cost budget in dollars	rare	senior signal

Rule of thumb. The senior data-pipeline CI/CD answer is a 5-minute monologue that covers dbt, Airflow, and Spark plus the four axes (correctness, cost, latency, blast radius) without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the pick-the-gate decision tree per PR type

Detailed explanation. Given a new pull request, the senior architect runs a fast decision tree in their head to pick which gates to run. Codifying the tree makes the CI system answer reproducible: any change fires only the gates that could possibly detect a regression from that change, and skips the rest. Walk through the tree for four canonical PR types.

Q1. Which files did the PR touch? → path-filter output.
Q2. If dbt files → does the PR add/modify a .sql model, a .yml schema test, a macro, or a seed? → each triggers different subsets of slim CI.
Q3. If Airflow files → is it a DAG, an operator, a hook, or a plugin? → each has a different test target.
Q4. If Spark files → is it a job entry point, a UDF, or shared library code? → each has different local + dry-run coverage.
Q5. If shared libraries → run every downstream system's smoke test.

Question. Walk the decision tree for four scenarios and record the gates that fire.

Input.

Scenario	Files touched	Q2/3/4	Gates that fire
Edit one dbt model	`dbt/models/marts/orders.sql`	model	slim CI (state:modified+), dbt test
Add an Airflow operator	`airflow/plugins/operators/reverse_etl.py`	operator	operator unit tests, DAG parse
Fix a Spark UDF	`spark/jobs/enrich/udf/geo.py`	UDF	local pytest, dry-run submit
Bump a shared library	`shared/pyutils/dates.py`	shared	all three systems' smoke tests

Code.

# scripts/pick_ci_gates.py — illustrative
def pick_gates(files_touched: list[str]) -> set[str]:
    """Return the set of CI gates that must run for a given file diff."""
    gates: set[str] = set()

    if any(f.startswith("dbt/") for f in files_touched):
        gates |= {"dbt-slim-ci", "dbt-test", "preview-schema"}

    if any(f.startswith("airflow/") for f in files_touched):
        gates |= {"airflow-parse", "airflow-unit"}
        if any("dags/" in f for f in files_touched):
            gates.add("airflow-integration")

    if any(f.startswith("spark/") for f in files_touched):
        gates |= {"spark-unit"}
        if any("jobs/" in f for f in files_touched):
            gates |= {"spark-package", "spark-dry-run"}

    if any(f.startswith("shared/") for f in files_touched):
        # Shared code — smoke-test every downstream system
        gates |= {"dbt-slim-ci", "airflow-parse", "spark-unit"}

    return gates


# Walk the four scenarios
print(pick_gates(["dbt/models/marts/orders.sql"]))
# → {'dbt-slim-ci', 'dbt-test', 'preview-schema'}

print(pick_gates(["airflow/plugins/operators/reverse_etl.py"]))
# → {'airflow-parse', 'airflow-unit'}

print(pick_gates(["spark/jobs/enrich/udf/geo.py"]))
# → {'spark-unit', 'spark-package', 'spark-dry-run'}

print(pick_gates(["shared/pyutils/dates.py"]))
# → {'dbt-slim-ci', 'airflow-parse', 'spark-unit'}

Step-by-step explanation.

Scenario 1 — a single dbt model edit. Slim CI computes state:modified+, rebuilds orders and its descendants against the preview schema, runs the model's .yml tests, and posts pass/fail to the PR. Total: ~2 minutes.
Scenario 2 — an operator edit. The operator is not imported by any DAG at file-scan time, so the parse gate is fast (~15 s). The operator unit tests exercise the operator directly with a mock hook. No integration test fires because no DAG file changed.
Scenario 3 — a Spark UDF edit. Local pytest exercises the UDF against a SparkSession.builder.master("local[2]") fixture. The job entry point that uses the UDF is repackaged and dry-run submitted to validate the cluster still accepts it.
Scenario 4 — a shared library change. Because we don't know which downstream systems use pyutils.dates, we run one smoke gate for each: dbt slim CI (which will trip if any macro imports the library), Airflow parse (which will trip if any DAG imports it), Spark unit tests (which will trip if any job imports it).
The decision tree is not a heuristic — it is a contract. Every gate in the tree has to run when the tree says it should; skipping any gate breaks the invariant that CI-green means merge-safe.

Output.

Scenario	Gates fired	p95 latency
Single dbt model edit	slim CI + dbt test + preview schema	2 min
New operator	parse + operator unit	30 s
Spark UDF fix	unit + package + dry-run	4 min
Shared library bump	slim CI + parse + spark unit (parallel)	4 min

Rule of thumb. The four-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any PR shape and get a gate list in under 30 seconds.

Senior interview question on data-pipeline CI/CD architecture

A senior interviewer often opens with: "You inherit a data-pipeline monorepo (dbt + Airflow + Spark) whose CI takes 45 minutes and costs $180 per PR. Walk me through the redesign — the per-system gates, the state comparison, the preview environments, the cost budget, and the on-call runbook when CI is red on prod-blocking PRs."

Solution Using slim CI + parse gate + dry-run submit + per-PR preview schemas

# .github/workflows/pipeline-ci.yml — the whole redesign
name: pipeline-ci
on:
  pull_request:
    branches: [ main ]
  pull_request_target:
    types: [ closed ]

jobs:
  # 1. Path-based short-circuit — never run a system that did not change
  detect:
    runs-on: ubuntu-latest
    outputs:
      dbt:     ${{ steps.f.outputs.dbt }}
      airflow: ${{ steps.f.outputs.airflow }}
      spark:   ${{ steps.f.outputs.spark }}
    steps:
      - uses: actions/checkout@v4
      - id: f
        uses: dorny/paths-filter@v3
        with:
          filters: |
            dbt:     [ 'dbt/**' ]
            airflow: [ 'airflow/**' ]
            spark:   [ 'spark/**' ]

  # 2. Preview schema — created per PR, dropped on close (see section 5)
  preview-schema:
    needs: detect
    if: needs.detect.outputs.dbt == 'true' && github.event.action != 'closed'
    uses: ./.github/workflows/preview-schema-create.yml

  # 3. dbt slim CI (see section 2)
  dbt-ci:
    needs: [ detect, preview-schema ]
    if: needs.detect.outputs.dbt == 'true'
    uses: ./.github/workflows/dbt-slim.yml

  # 4. Airflow gates (see section 3)
  airflow-ci:
    needs: detect
    if: needs.detect.outputs.airflow == 'true'
    uses: ./.github/workflows/airflow.yml

  # 5. Spark gates (see section 4)
  spark-ci:
    needs: detect
    if: needs.detect.outputs.spark == 'true'
    uses: ./.github/workflows/spark.yml

  # 6. On PR close — clean up preview schema regardless of merge/reject
  cleanup:
    if: github.event.action == 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Drop preview schema
        run: python scripts/drop_preview_schema.py --pr ${{ github.event.number }}

# CI budget contract — one line per gate
Gate                    p95 latency   Cost budget   Owner
────────────────────────────────────────────────────────────
dbt-slim-ci             120 s         $0.30         analytics-eng
airflow-parse           15 s          $0.02         platform
airflow-unit            180 s         $0.05         platform
spark-unit              240 s         $0.08         data-eng
spark-dry-run           30 s          $0.02         data-eng
preview-schema-create   20 s          $0.01         platform
preview-schema-drop     10 s          $0.00         platform
────────────────────────────────────────────────────────────
Mixed PR (all systems)  ~5 min        ~$0.50        —

Step-by-step trace.

Concern	Before	After
dbt strategy	full `dbt run --target ci`	`dbt build --select state:modified+ --defer`
CI duration (dbt-only PR)	45 min	2 min
CI cost (dbt-only PR)	$180	$0.30
Airflow gate	none	parse gate + operator unit tests
Spark gate	submit-to-prod-cluster	local pytest + dry-run submit
Warehouse blast radius	shared `analytics_ci` schema	per-PR `pr_<PR>_<branch>` schema
Preview UX for analysts	wait for merge	preview URL on PR open
Cleanup	manual quarterly purge	auto-drop on PR close

After the migration, a dbt-only PR completes CI in about 2 minutes and costs roughly $0.30; a mixed PR touching all three systems completes in about 5 minutes because gates run in parallel; the November warehouse invoice drops from about $9,000 in CI compute to about $200. Analysts open the preview URL from the PR description and see the dashboard rendered against the changed models within 3 minutes of PR open.

Output:

Metric	Before	After
Median CI time	45 min	2 min
p95 CI time (mixed PR)	55 min	5 min
Median CI cost	$180	$0.30
Preview environment	none	one per PR
Blast radius	one shared schema	per-PR isolation
November warehouse CI bill	~$9,000	~$200

Why this works — concept by concept:

Path-based short-circuit — the cheapest optimisation available. A whitespace-only PR to README.md fires zero gates; the CI pipeline is essentially free for docs-only work. This is the "detect" job at the top of the workflow.
Per-system gates — dbt, Airflow, and Spark each get the native gate their runtime actually cares about. Slim CI for dbt, parse gate for Airflow, dry-run submit for Spark. Running only the gates that could possibly detect a regression is the definition of proportionate CI.
Preview schemas — every PR gets its own pr_<PR>_<branch> schema, so two concurrent CI runs cannot collide. The preview is also the analyst-facing URL; the same schema serves CI and human review.
Auto-cleanup on PR close — the cleanup workflow is idempotent and fires whether the PR merges or is rejected. Without it, preview schemas accumulate indefinitely; with it, warehouse footprint is bounded.
Cost budget per gate — publishing the cost budget makes it a contract, not a hope. When a gate exceeds its budget, the SRE-of-CI catches it in the weekly review; when a new gate is proposed, the budget question comes up in code review.
Cost — one CI orchestration layer, one preview-schema helper, one cleanup workflow, plus per-system gates that each stay under $0.30. Compared to the 45-minute full-rebuild baseline, this is one order of magnitude faster at less than 1% of the cost. Net O(delta) per PR versus O(catalog) per PR.

ETL
Topic — etl
ETL problems on pipeline-CI design

Practice →

Design Topic — design Design problems on data platform CI/CD

Practice →

2. Slim CI for dbt — state:modified, defer, and clone

`dbt build --select state:modified+ --defer --state prod-manifest/` is the four-word answer to every dbt CI/CD interview

The mental model in one line: slim ci dbt is the pattern where a CI job downloads the last successful production manifest.json, compares its own manifest against it to compute state:modified (the set of models whose SQL, config, or upstream sources actually changed), rebuilds only that set plus its descendants (state:modified+) into a per-PR schema, and defers any unbuilt parent refs to the production schema instead of re-materialising them — turning a 45-minute full rebuild into a 2-minute incremental that costs cents instead of dollars while still testing every change against real prod data. Every senior analytics engineer has adopted this pattern once; every senior interviewer probes for it in the first minute.

The four axes for slim CI dbt.

Correctness. state:modified catches every model whose SQL, config, or upstream .yml schema actually changed. The + selector suffix adds every descendant (models that ref the changed ones), so a change to a staging model rebuilds the mart that depends on it. This is the correctness contract: if the PR could affect a model, that model gets rebuilt and tested.
Cost. A slim CI run rebuilds 3-15 models on average (versus 200-1500 for a full run). Snowflake XS warehouse with AUTO_SUSPEND=60 keeps a typical run under $0.30. This is the cost contract; without slim CI, a 200-model warehouse hits $30-$500 per CI run.
Latency-to-merge. Two-minute CI turns "I'll review this after lunch" into "I'll review this in the next 5 minutes." The latency payoff compounds across a team of ten analysts each opening 3 PRs a day.
Blast radius. Slim CI writes to the per-PR preview schema (pr_<PR>_<branch>), never to analytics_prod. Deferral routes reads to the prod schema (safe — read-only) and writes to the preview schema (isolated — per-PR). The prod schema is never touched by CI.

The manifest.json — the state comparison substrate.

What it is. dbt's target/manifest.json describes the parsed project: every model, its SQL fingerprint, its config, its upstream refs, its sources. Two manifests are directly comparable; state:modified diffs them.
Where it lives. Uploaded to S3 (or GCS or Azure Blob) as an artifact at the end of every successful prod run. Path convention: s3://dbt-artifacts/prod/latest/manifest.json plus a versioned copy at s3://dbt-artifacts/prod/<run_id>/manifest.json.
How CI downloads it. The CI job pulls the "prod latest" manifest into a local prod-manifest/ directory before running slim CI. The --state prod-manifest/ flag points dbt at it.
The bootstrap problem. The first CI run has no prod manifest to compare against; the slim-CI logic falls back to a full build. Most teams solve this by seeding the S3 path from a manual initial run.

The state:modified+ selector — what it catches and what it doesn't.

What state:modified catches. Models whose SQL text changed, whose config (materialization, tags, cluster_by, etc.) changed, whose upstream sources' definitions changed, or whose upstream refs' fingerprints changed. Also new models and deleted models.
What + adds. Every descendant (children, grandchildren, etc.) of the modified set. If you change a staging model, every mart that depends on it rebuilds.
What it does not catch. Behaviour changes in seeds (use state:modified,resource_type:seed for that), macro changes (use state:modified.macros), or changes in a dbt package version bump (use state:modified.macros + state:modified.contracts).
The safety escape hatch. For a suspicious PR (a big refactor, a dbt-core upgrade), append --select "state:modified+" "state:new+" or fall back to a full build on the main branch periodically.

The --defer --state prod-manifest/ pair — the parent-routing magic.

The read. When a CI-built model runs {{ ref('unmodified_upstream') }}, dbt sees that model was not rebuilt in this run, checks the state manifest, and rewrites the ref to point at the prod schema instead of the CI schema.
The write. All modified models still write to the CI schema (pr_<PR>_<branch>); deferred parents are only read from prod. Prod is never mutated.
The permission model. The CI role needs SELECT on the prod schema (to read deferred parents) and CREATE + SELECT + INSERT on the preview schema (to build modified models). Configure once in Snowflake / BigQuery / Redshift; use forever.

Common interview probes on slim CI dbt.

"What does state:modified+ mean?" — required answer: modified models plus all descendants.
"How do you get the prod manifest into CI?" — download from S3 at the top of the CI job.
"What does --defer do?" — routes unbuilt refs to the state manifest's schema (prod) instead of the CI schema.
"How do you handle the bootstrap case where no prod manifest exists yet?" — fall back to a full build on the first run and seed the artifact.
"How do you avoid slim CI missing a macro change?" — either use state:modified.macros selector or trigger a full build on macro-file changes.

Worked example — GitHub Actions job that runs slim CI on every PR

Detailed explanation. The canonical slim CI setup: a GitHub Actions workflow that installs dbt, downloads the prod manifest from S3, computes state:modified+, defers parents to prod, and posts the results to the PR. Build the whole thing from scratch.

Trigger. pull_request open, synchronize, reopen; skip on PR close.
Runner. ubuntu-latest; Python 3.11; dbt-snowflake 1.9+.
State source. Prod manifest from s3://dbt-artifacts/prod/latest/manifest.json.
Target schema. pr_<PR>_<branch> derived from GitHub context.
Post. Slim-CI summary as a PR comment.

Question. Write the full GitHub Actions workflow that runs dbt build --select state:modified+ --defer --state prod-manifest/ on every PR.

Input.

Parameter	Value
CI runner	ubuntu-latest
dbt adapter	dbt-snowflake 1.9
State source	s3://dbt-artifacts/prod/latest/
Target	ci (pr__ schema)
Auth	OIDC to AWS + Snowflake key-pair from GitHub secret

Code.

# .github/workflows/dbt-slim.yml — reusable slim CI workflow
name: dbt-slim-ci
on:
  workflow_call:

jobs:
  slim-ci:
    runs-on: ubuntu-latest
    permissions:
      id-token: write          # OIDC to AWS
      contents: read
      pull-requests: write     # to post the comment
    env:
      DBT_PROFILES_DIR: ./ci
      DBT_TARGET: ci
      DBT_PR_NUMBER: ${{ github.event.pull_request.number }}
      DBT_BRANCH_SLUG: ${{ github.head_ref }}

    steps:
      - name: Check out PR
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dbt
        working-directory: dbt
        run: |
          pip install -r requirements.txt
          dbt --version

      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-dbt-ci
          aws-region: us-east-1

      - name: Download prod manifest
        run: |
          mkdir -p prod-manifest
          aws s3 cp s3://dbt-artifacts/prod/latest/manifest.json \
                    prod-manifest/manifest.json

      - name: Compute preview schema name
        id: schema
        run: |
          # pr_<PR>_<branch-slug> — sanitize branch name for SQL identifier
          slug=$(echo "$DBT_BRANCH_SLUG" | tr '/' '_' | tr -cd '[:alnum:]_' | tr '[:upper:]' '[:lower:]' | cut -c1-40)
          echo "schema=pr_${DBT_PR_NUMBER}_${slug}" >> "$GITHUB_OUTPUT"

      - name: dbt deps
        working-directory: dbt
        run: dbt deps

      - name: dbt build (slim CI)
        working-directory: dbt
        env:
          DBT_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
          DBT_TARGET_SCHEMA: ${{ steps.schema.outputs.schema }}
        run: |
          dbt build \
            --select state:modified+ \
            --defer \
            --state ../prod-manifest \
            --target ci \
            --fail-fast

      - name: Upload CI manifest for downstream jobs
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ci-manifest-${{ env.DBT_PR_NUMBER }}
          path: dbt/target/manifest.json

      - name: Post slim-CI summary to PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs   = require('fs');
            const path = require('path');
            const run  = fs.readFileSync('dbt/target/run_results.json', 'utf8');
            const j    = JSON.parse(run);
            const rows = j.results.map(r =>
              `| ${r.unique_id} | ${r.status} | ${r.execution_time.toFixed(2)}s |`
            ).join('\n');
            const body = `### dbt slim CI — PR #${{ env.DBT_PR_NUMBER }}\n\n` +
              `| Model | Status | Time |\n|---|---|---|\n${rows}\n\n` +
              `Preview schema: \`${{ steps.schema.outputs.schema }}\`.\n`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner:  context.repo.owner,
              repo:   context.repo.repo,
              body:   body
            });

# dbt/ci/profiles.yml — CI target profile
analytics:
  target: dev
  outputs:
    ci:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('SNOWFLAKE_USER') }}"
      private_key: "{{ env_var('DBT_SNOWFLAKE_PRIVATE_KEY') }}"
      role: DBT_CI_ROLE
      warehouse: DBT_CI_WH_XS   # AUTO_SUSPEND=60
      database: ANALYTICS
      schema: "{{ env_var('DBT_TARGET_SCHEMA') }}"  # pr_<PR>_<branch>
      threads: 8
      client_session_keep_alive: false

Step-by-step explanation.

The workflow is workflow_call-able so the top-level pipeline-ci.yml can invoke it as a reusable job. This keeps the dbt-specific logic in one file and lets the top-level orchestrator focus on gate selection.
OIDC + aws-actions/configure-aws-credentials is the modern auth pattern; there is no long-lived AWS key in a GitHub secret. The IAM role gha-dbt-ci has read access to the dbt-artifacts bucket only.
Compute preview schema name derives pr_<PR>_<slug> from the GitHub Actions context. The slug sanitiser strips non-alphanumeric characters, lowercases, and caps at 40 characters — Snowflake identifiers must fit in 255 characters and be lowercase-safe.
dbt build --select state:modified+ --defer --state ../prod-manifest --target ci is the core command. build is run + test + snapshot + seed in dependency order — one command runs everything. --fail-fast stops on the first error so CI feedback is fast.
The final step posts a table of every rebuilt model + its status + its runtime as a PR comment. Reviewers see the exact impact of the PR without opening the CI logs. The comment updates on every push (or a new comment; use a comment-updater action for one-comment-per-PR UX).

Output.

Rebuild scope	Full CI	Slim CI
Models rebuilt (median PR)	200	4
p95 CI time	45 min	2 min
p95 CI cost	$180	$0.30
Preview schema	`analytics_ci` (shared)	`pr_<PR>_<branch>` (isolated)
PR comment	none	model-by-model table

Rule of thumb. For any dbt CI setup, mount slim CI as the default gate: state:modified+ --defer --state prod-manifest/. Post the results as a PR comment. Upload the CI manifest so downstream jobs (e.g. a doc-preview or dbt-freshness job) can consume it. These three habits move a dbt project from "CI is a chore" to "CI is a review tool."

Worked example — state comparison against `manifest.json` from main

Detailed explanation. State comparison is the substrate slim CI runs on: dbt compares two manifests and computes the changed set. Walk through what "state comparison" actually means at the manifest level — which fields dbt looks at, how it computes state:modified, and how you can inspect the comparison manually to debug false positives.

Fields dbt fingerprints. SQL text (after Jinja render), config (materialization, cluster_by, partition_by, tags, meta), upstream refs, upstream sources, contracts (if enabled), and constraints.
The fingerprint algorithm. dbt computes a SHA-256 hash of the concatenation of the above fields per model. Two models with the same hash are "unmodified"; differing hashes mark state:modified.
Why false positives happen. Whitespace changes in SQL, Jinja variable resolution differences (e.g. {{ var('env') }} resolves differently in CI vs prod), or macro upgrades can flip the hash even when the effective SQL is identical.
Debugging. dbt ls --select state:modified --state prod-manifest/ lists exactly what changed; dbt-checkpoint or the dbt-project-evaluator package can help diagnose surprising diffs.

Question. Write a debug script that lists exactly which fields of a modified model changed between the CI manifest and the prod manifest.

Input.

Component	Value
Prod manifest	prod-manifest/manifest.json
CI manifest	dbt/target/manifest.json
Model to inspect	model.analytics.orders_mart
Fields to diff	raw_code, config.materialized, depends_on.nodes, checksum.checksum

Code.

# scripts/dbt_state_diff.py — diff two manifests for a single model
"""
Usage:
  python scripts/dbt_state_diff.py \
      --prod prod-manifest/manifest.json \
      --ci   dbt/target/manifest.json \
      --model model.analytics.orders_mart
"""
import argparse
import json
import difflib
from pathlib import Path


def load_node(path: Path, unique_id: str) -> dict:
    manifest = json.loads(path.read_text())
    node = manifest["nodes"].get(unique_id)
    if node is None:
        raise SystemExit(f"{unique_id!r} not in {path}")
    return node


def diff_fields(prod: dict, ci: dict) -> list[tuple[str, str, str]]:
    """Return (field, prod_value, ci_value) for each differing top-level field."""
    fields = [
        "raw_code",
        "checksum",
        "config",
        "depends_on",
        "sources",
        "refs",
        "contract",
    ]
    out: list[tuple[str, str, str]] = []
    for f in fields:
        pv = json.dumps(prod.get(f), sort_keys=True, indent=2)
        cv = json.dumps(ci.get(f),  sort_keys=True, indent=2)
        if pv != cv:
            out.append((f, pv, cv))
    return out


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prod",  required=True, type=Path)
    ap.add_argument("--ci",    required=True, type=Path)
    ap.add_argument("--model", required=True)
    args = ap.parse_args()

    prod = load_node(args.prod, args.model)
    ci   = load_node(args.ci,   args.model)
    diffs = diff_fields(prod, ci)

    if not diffs:
        print(f"{args.model}: identical (unexpected — would not be state:modified)")
        return

    for field, pv, cv in diffs:
        print(f"\n=== {field} ===")
        lines = difflib.unified_diff(
            pv.splitlines(), cv.splitlines(),
            fromfile=f"prod::{field}", tofile=f"ci::{field}",
            lineterm=""
        )
        print("\n".join(lines))


if __name__ == "__main__":
    main()

# Example output for a legit SQL change
=== raw_code ===
--- prod::raw_code
+++ ci::raw_code
@@ -12,7 +12,7 @@
 SELECT
     o.order_id,
     o.customer_id,
-    o.total_cents          AS gross_cents,
+    o.total_cents          AS gross_cents,
+    o.discount_cents       AS discount_cents,
     o.status
 FROM {{ ref('stg_orders') }} o

=== checksum ===
--- prod::checksum
+++ ci::checksum
@@ -1,4 +1,4 @@
 {
   "name": "sha256",
-  "checksum": "5a2f...9c1d"
+  "checksum": "8d3b...4e77"
 }

Step-by-step explanation.

The script loads both manifests, extracts the requested unique_id (dbt's fully-qualified node key), and diffs a curated list of top-level fields. Diffing every field is noisy; the curated list matches what dbt actually fingerprints.
raw_code is the post-Jinja-render SQL. A whitespace-only change here still flips the checksum, so if a model is unexpectedly state:modified, this is the first field to inspect. Consider adding an .editorconfig and sqlfmt pre-commit hook to eliminate whitespace-only diffs.
checksum is the SHA-256 that dbt actually compares. If checksums match, the model is not state:modified; if they differ, it is. This is the ground-truth field.
config catches materialization / cluster_by / tag changes. A model that switches from view to table is state:modified even if the SQL is identical.
depends_on.nodes catches upstream ref changes. Adding a new {{ ref('foo') }} marks the model modified; removing one also marks it modified.

Output.

Field	Change type	Marks model as state:modified?
`raw_code`	any character diff (incl whitespace)	yes
`checksum`	SHA-256 changed	yes (derived from raw_code + config)
`config.materialized`	view → table	yes
`config.tags`	added / removed	yes
`depends_on.nodes`	new / removed ref	yes
`description` (yml)	text-only change	no (documentation only)

Rule of thumb. When a PR unexpectedly rebuilds a model that "shouldn't have changed," run the state-diff script against the model — the culprit is almost always a whitespace change in raw_code, a Jinja-var resolution difference, or a macro upgrade that touched the render. Fix the root cause; do not disable slim CI.

Worked example — `dbt clone` a schema for preview

Detailed explanation. For dashboards, downstream apps, and analysts who want to use the CI schema, an empty pr_<PR>_<branch> schema is not enough — they need it seeded with all the tables that were not rebuilt by slim CI. dbt clone (added in dbt-core 1.6) does this in one command: it zero-copy clones every model in the state manifest that was not rebuilt, so the preview schema is a full mirror of prod overlaid with the rebuilt models. This is the pattern that turns slim CI into a genuine preview environment.

What dbt clone does. For every model in --state, if the CI schema doesn't already have the model, dbt executes CREATE OR REPLACE <schema>.<model> CLONE prod.<model> (Snowflake) or the equivalent zero-copy on the warehouse.
Why it's cheap. Snowflake and Databricks Unity Catalog implement zero-copy cloning as a metadata operation; no data is physically copied. A 10 TB table clones in ~1 second and costs $0.
When to use it. Whenever a human (analyst, PM, stakeholder) needs to SELECT * FROM pr_<PR>_<branch>.orders_mart and expect all the joined tables to be there.

Question. Extend the slim CI workflow to clone unmodified models into the preview schema after the slim build completes.

Input.

Component	Value
Warehouse	Snowflake (or Databricks UC)
Clone command	`dbt clone --state prod-manifest/`
Target schema	pr__
Runtime for 200-table clone	~5 s (zero-copy metadata op)
Cost for 200-table clone	~$0.00

Code.

# Extension to .github/workflows/dbt-slim.yml — add after "dbt build" step
- name: dbt clone unmodified models into preview schema
  working-directory: dbt
  env:
    DBT_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
    DBT_TARGET_SCHEMA: ${{ steps.schema.outputs.schema }}
  run: |
    dbt clone \
      --state ../prod-manifest \
      --target ci \
      --resource-type model \
      --resource-type seed \
      --resource-type snapshot

-- What `dbt clone` executes on Snowflake, per model
CREATE OR REPLACE VIEW ANALYTICS.PR_4712_ADD_ORDERS_V2.STG_CUSTOMERS
  CLONE ANALYTICS.PROD.STG_CUSTOMERS;

CREATE OR REPLACE TABLE ANALYTICS.PR_4712_ADD_ORDERS_V2.DIM_DATE
  CLONE ANALYTICS.PROD.DIM_DATE;

-- ... one CREATE OR REPLACE ... CLONE per model in the state manifest
-- that was not rebuilt by slim CI ...

-- The rebuilt models (from slim CI) are already in the preview schema
-- as freshly-materialized objects, so `CREATE OR REPLACE ... CLONE` on
-- them would overwrite the rebuild. `dbt clone` skips those.

# scripts/verify_preview_schema.py — smoke test after clone
import snowflake.connector

def verify(preview_schema: str) -> None:
    conn = snowflake.connector.connect(
        account="acme-prod",
        user="dbt_ci",
        private_key=load_private_key(),
        role="DBT_CI_ROLE",
        warehouse="DBT_CI_WH_XS",
        database="ANALYTICS",
    )
    with conn.cursor() as cur:
        cur.execute(f"""
            SELECT COUNT(*)
            FROM   ANALYTICS.INFORMATION_SCHEMA.TABLES
            WHERE  TABLE_SCHEMA = %s
        """, (preview_schema.upper(),))
        n_tables = cur.fetchone()[0]

        cur.execute(f"""
            SELECT COUNT(*)
            FROM   ANALYTICS.INFORMATION_SCHEMA.TABLES
            WHERE  TABLE_SCHEMA = 'PROD'
        """)
        n_prod = cur.fetchone()[0]

    assert n_tables == n_prod, (
        f"Preview schema has {n_tables} tables; prod has {n_prod}. "
        "Slim CI + clone should have parity."
    )
    print(f"OK — preview schema {preview_schema} has {n_tables} tables")


if __name__ == "__main__":
    import sys
    verify(sys.argv[1])

Step-by-step explanation.

The dbt clone step runs after the slim build so the rebuilt models already exist in the preview schema. Clone only fills in the gaps — the models that were not rebuilt because they were not state:modified.
Under the hood, dbt clone emits CREATE OR REPLACE ... CLONE prod.<table> for each unmodified model. Snowflake's zero-copy clone is a metadata-only operation: no data is physically duplicated, no compute is billed, and clone completes in milliseconds per table.
--resource-type model --resource-type seed --resource-type snapshot scopes what gets cloned. Skip macros and analyses (they aren't materialised). Include seeds so hard-coded reference tables are present.
The verify_preview_schema.py smoke test confirms parity: the number of tables in the preview schema equals the number in prod. If it doesn't, something in the clone step silently failed and the preview would show missing-table errors to analysts.
On PR close (see section 5), the cleanup workflow DROP SCHEMA pr_<PR>_<branch> reclaims all of it — the clones are metadata pointers, so dropping the schema releases nothing (no storage was ever allocated), but keeps the catalog tidy.

Output.

Model type	After slim build	After clone
Modified model	freshly materialized	unchanged (skipped by clone)
Unmodified upstream	missing	zero-copy clone from prod
Seed	(only if changed)	zero-copy clone from prod
Snapshot	(only if changed)	zero-copy clone from prod

Rule of thumb. For any dbt preview environment that humans will actually query, follow slim build with dbt clone to backfill unmodified tables. Zero-copy cloning is free on Snowflake and Databricks UC; on BigQuery use table snapshots (also free); on Redshift you must physically CTAS, which costs — reconsider preview UX there or use --defer alone.

Senior interview question on slim CI dbt

A senior interviewer might ask: "You inherit a dbt project with 800 models running full dbt build --target ci on every PR — 55-minute CI, $220 per run, analysts complain they can't review dashboards until merge. Design the slim CI migration: the state artifact contract, the CI workflow, the preview schema, the fallback for macro changes, and the on-call runbook when the prod manifest is missing or stale."

Solution Using slim CI with state comparison, deferral, `dbt clone`, and a manifest freshness guard

# .github/workflows/dbt-slim-full.yml — production-grade slim CI
name: dbt-slim-ci
on:
  workflow_call:

jobs:
  slim-ci:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      pull-requests: write
    env:
      DBT_PROFILES_DIR: ./ci
      DBT_PR_NUMBER:    ${{ github.event.pull_request.number }}
      DBT_BRANCH_SLUG:  ${{ github.head_ref }}
      STATE_BUCKET:     dbt-artifacts
      STATE_KEY_PREFIX: prod/latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }

      - name: Install dbt
        run: pip install -r dbt/requirements.txt

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-dbt-ci
          aws-region: us-east-1

      # 1. Download prod manifest — with freshness guard
      - name: Download prod manifest
        id: manifest
        run: |
          mkdir -p prod-manifest
          aws s3 cp s3://${STATE_BUCKET}/${STATE_KEY_PREFIX}/manifest.json prod-manifest/
          aws s3 cp s3://${STATE_BUCKET}/${STATE_KEY_PREFIX}/generated_at.txt prod-manifest/
          age_hours=$(( ( $(date +%s) - $(date -d "$(cat prod-manifest/generated_at.txt)" +%s) ) / 3600 ))
          echo "age_hours=${age_hours}" >> "$GITHUB_OUTPUT"
          if [ "$age_hours" -gt 48 ]; then
            echo "::warning::Prod manifest is ${age_hours}h old; slim CI may over-select"
          fi

      # 2. Detect macro-only or dbt-core changes — fall back to full build
      - name: Detect full-build triggers
        id: mode
        run: |
          if git diff --name-only origin/main | grep -qE '(macros/|dbt_project\.yml|packages\.yml|requirements\.txt)'; then
            echo "mode=full" >> "$GITHUB_OUTPUT"
          else
            echo "mode=slim" >> "$GITHUB_OUTPUT"
          fi

      # 3. Compute preview schema
      - name: Compute preview schema name
        id: schema
        run: |
          slug=$(echo "$DBT_BRANCH_SLUG" | tr '/' '_' | tr -cd '[:alnum:]_' | tr '[:upper:]' '[:lower:]' | cut -c1-40)
          echo "schema=pr_${DBT_PR_NUMBER}_${slug}" >> "$GITHUB_OUTPUT"

      # 4. dbt deps
      - name: dbt deps
        working-directory: dbt
        run: dbt deps

      # 5. dbt build — slim or full
      - name: dbt build
        working-directory: dbt
        env:
          DBT_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
          DBT_TARGET_SCHEMA:         ${{ steps.schema.outputs.schema }}
        run: |
          if [ "${{ steps.mode.outputs.mode }}" = "full" ]; then
            echo "Full build (macro / dbt-core change detected)"
            dbt build --target ci --fail-fast
          else
            echo "Slim build (state:modified+)"
            dbt build \
              --select state:modified+ \
              --defer \
              --state ../prod-manifest \
              --target ci \
              --fail-fast
          fi

      # 6. dbt clone — fill preview schema with unmodified prod tables
      - name: dbt clone unmodified models
        if: steps.mode.outputs.mode == 'slim'
        working-directory: dbt
        env:
          DBT_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
          DBT_TARGET_SCHEMA:         ${{ steps.schema.outputs.schema }}
        run: |
          dbt clone \
            --state ../prod-manifest \
            --target ci \
            --resource-type model \
            --resource-type seed \
            --resource-type snapshot

      # 7. Upload CI manifest for downstream jobs and post PR comment
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: ci-manifest-${{ env.DBT_PR_NUMBER }}
          path: dbt/target/manifest.json

      - name: Post summary to PR
        if: always()
        uses: actions/github-script@v7
        env:
          DBT_MODE:            ${{ steps.mode.outputs.mode }}
          DBT_PREVIEW_SCHEMA:  ${{ steps.schema.outputs.schema }}
          DBT_MANIFEST_AGE_H:  ${{ steps.manifest.outputs.age_hours }}
        with:
          script: |
            const fs = require('fs');
            const j  = JSON.parse(fs.readFileSync('dbt/target/run_results.json', 'utf8'));
            const rows = j.results.slice(0, 40).map(r =>
              `| ${r.unique_id} | ${r.status} | ${r.execution_time.toFixed(2)}s |`
            ).join('\n');
            const body =
              `### dbt CI — PR #${process.env.DBT_PR_NUMBER} (${process.env.DBT_MODE})\n\n` +
              `Preview schema: \`${process.env.DBT_PREVIEW_SCHEMA}\`\n` +
              `Manifest age: ${process.env.DBT_MANIFEST_AGE_H}h\n\n` +
              `| Node | Status | Time |\n|---|---|---|\n${rows}\n`;
            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner:  context.repo.owner,
              repo:   context.repo.repo,
              body:   body
            });

# prod dbt job — uploads manifest at end of every successful run
# (runs in Airflow; upload snippet at task boundary)
import boto3
from pathlib import Path
from datetime import datetime, timezone

def upload_prod_manifest():
    s3 = boto3.client("s3")
    bucket = "dbt-artifacts"
    now    = datetime.now(timezone.utc).isoformat()

    s3.upload_file("dbt/target/manifest.json",   bucket, "prod/latest/manifest.json")
    s3.upload_file("dbt/target/run_results.json", bucket, "prod/latest/run_results.json")
    s3.put_object(Bucket=bucket, Key="prod/latest/generated_at.txt", Body=now)

    # Also keep a versioned copy for backfills
    run_id = f"{datetime.now(timezone.utc):%Y%m%dT%H%M%SZ}"
    s3.upload_file("dbt/target/manifest.json", bucket, f"prod/{run_id}/manifest.json")

Step-by-step trace.

Step	Behaviour	Reasoning
Manifest download	pull prod-manifest/manifest.json + generated_at.txt	freshness guard prevents stale state
Manifest age check	warn if >48h old	stale manifest over-selects models
Full-build fallback	trigger on macros/, dbt_project.yml, packages.yml, requirements.txt	slim CI can miss macro impact
Preview schema name	pr__	deterministic; sanitized for Snowflake
dbt build	slim (state:modified+ --defer) or full	mode-aware
dbt clone	zero-copy clone unmodified models into preview	preview UX = full mirror
PR comment	model-by-model results with schema name	reviewers see impact + preview URL
Manifest upload	store dbt/target/manifest.json for downstream	doc-preview / freshness jobs consume it

After deployment, a typical PR rebuilds 4-15 models in ~90 seconds against the preview schema, clones the remaining 785 tables in ~5 seconds (zero-copy metadata operations), and posts a PR comment with the model table + preview schema name; analysts open pr_4712_add_orders_v2.orders_mart in Snowsight and see the new column immediately. Macro-touching PRs fall back to a full build (still under 15 minutes on the XS warehouse with parallel threads). The November invoice drops from ~$9k to ~$200 for CI compute; the average PR is reviewed within 10 minutes of open instead of the next day.

Output:

Metric	Before	After
Median CI time	55 min	2 min
p95 CI cost per PR	$220	$0.35
Preview UX	none	full mirror on PR open
Macro-change safety	none	auto-fallback to full build
Manifest staleness alert	none	warns at 48h
Reviewer feedback loop	next day	~10 min

Why this works — concept by concept:

state:modified+ --defer --state — the slim CI trinity. state:modified+ scopes the build to only what changed and its descendants; --defer routes unbuilt refs to prod; --state points at the downloaded prod manifest. Together they make the CI job proportionate to the PR, not proportionate to the catalog.
Prod manifest as artifact contract — the production run uploads manifest.json to a well-known S3 key at the end of every successful run; the CI job downloads it at the top. This is the durable state contract that makes state comparison possible.
Manifest freshness guard — if the prod manifest is more than 48 hours old, slim CI may over-select (mark models as modified because prod is stale) or under-select (miss real changes because prod-manifest fingerprints don't match current prod). The warning surfaces the problem before it silently distorts the CI output.
Macro-change full-build fallback — a change to macros/ can affect every model that references the macro without changing the model's SQL text. Slim CI would miss it. Detecting the touched paths and falling back to a full build closes the correctness gap.
dbt clone for preview parity — slim CI only builds the changed subset; without cloning, the preview schema has 4 tables and analysts see missing-table errors. dbt clone fills the gap in seconds at zero cost via zero-copy metadata.
Cost — one prod-side upload step (~$0.00), one CI job with slim build + clone (~$0.30 typical), one XS warehouse with AUTO_SUSPEND=60. The eliminated cost is the 55-minute full rebuild ($220 per PR × 30 PRs/day = ~$200k/year). Net O(changed) per PR versus O(catalog) per PR.

SQL
Topic — sql
SQL modeling and warehouse-CI problems

Practice →

ETL Topic — etl ETL problems on incremental modeling

Practice →

3. Airflow CI — DAG parse, integration tests, deployable artifacts

`DagBag(import_errors={})` is the first gate every Airflow monorepo needs — parse before you promote

The mental model in one line: airflow ci cd is the pattern where every PR that touches airflow/ runs three gates in sequence — a DAG-parse gate that constructs a DagBag in under 30 seconds and asserts zero import_errors, a unit-test matrix that exercises custom operators and hooks with mocked backends, and (optionally) an integration test that runs the changed DAG end-to-end against a LocalExecutor — before the versioned deployable artifact (an OCI image, a dags.zip for MWAA, or an Astronomer deployment bundle) is even built — because a DAG that fails at import time takes down every other DAG in the DagBag, not just its own scheduling. Every senior Airflow deployment has been rescued by the parse gate at least once; skipping it turns "one broken DAG" into "the whole scheduler is red."

The four axes for Airflow CI.

Correctness. Three levels: parse (does every DAG file import without error?), unit (do custom operators return the right values for mocked inputs?), integration (does a whole DAG execute end-to-end?). Each layer catches a different class of bug. The parse gate is non-negotiable; the unit tests are the coverage layer; the integration test is the "does this DAG actually work?" layer.
Cost. Airflow CI is essentially free — parse and unit tests run in seconds on ubuntu-latest. The optional integration test is more expensive because it spins a Postgres + LocalExecutor, but still under $0.10 per run. Cost is not the axis that bites for Airflow CI; latency is.
Latency-to-merge. Parse gate in ~15 seconds is the fastest fail loop of any data-pipeline CI gate. Unit tests in 2-3 minutes. Integration tests in 4-6 minutes. A DAG-only PR should complete CI in under 5 minutes p95.
Blast radius. A DAG that fails at import time fails every DAG in the DagBag — the scheduler pauses because dagbag_import_error_traceback_depth triggers. The parse gate stops that class of change from reaching the scheduler at all.

The DAG parse gate — 15 seconds, one assertion.

The mechanism. Instantiate DagBag(dag_folder="airflow/dags", include_examples=False) in a subprocess. If any DAG file raises during import, dagbag.import_errors is a non-empty dict.
The assertion. assert not dagbag.import_errors, dagbag.import_errors. The assertion message dumps the file paths and tracebacks.
The trap. DAG files often do work at module import time (query metadata DB, hit Airflow variables, call Variable.get). CI runs without those variables set. Solution: use Variable.get("x", default_var="ci_placeholder") or gate DAG-time work behind if not os.getenv("SKIP_DAG_INIT_WORK").
The timing. A modern DagBag with 200 DAGs parses in 5-15 seconds. The 30-second Airflow scheduler timeout is your reference point — if CI takes longer, the scheduler will too.

The operator + hook unit-test matrix.

What it covers. Custom operators (class MyOperator(BaseOperator)), custom hooks (class MyHook(BaseHook)), custom sensors, and utilities. Everything you wrote rather than imported from airflow or a provider package.
How to structure tests. tests/operators/test_my_operator.py; instantiate the operator with mock upstream context; assert on the return value or the mock's call args.
What to mock. External services (S3, Snowflake, Kafka) with unittest.mock or moto for AWS. XCom pushes; assert via mock_ti.xcom_push.assert_called_with(...).
What not to mock. Airflow's own execution machinery — use airflow.models.taskinstance.TaskInstance in "test mode" where possible. Excessive mocking of Airflow itself creates tests that pass but tell you nothing.

The DAG integration test — LocalExecutor + ephemeral Postgres.

When to run. For DAGs that touch shared resources (a common warehouse table, a critical Kafka topic) or for DAGs that use complex dynamic task mapping. Not every DAG needs one.
The setup. airflow db init against a Postgres in docker-compose; airflow dags test <dag_id> <execution_date> runs the DAG end-to-end synchronously without a scheduler.
The runtime. 30 seconds to 6 minutes depending on DAG complexity. Cap the integration-test matrix at the DAGs whose failure would cost more than 6 minutes of CI time to catch.

The deployable artifact — MWAA vs Astronomer vs self-hosted.

MWAA. dags.zip uploaded to S3; requirements.txt in the same bucket; plugins.zip for custom operators. Versioned by S3 object version + airflow_version field.
Astronomer. astro deploy builds an OCI image, tags it, and rolls it out to the deployment. Versioned by image digest.
Self-hosted on Kubernetes. Custom OCI image built from apache/airflow:2.10.x with your DAGs baked in; Helm chart with the image tag.
The versioning contract. Every deployable artifact carries the git SHA, the branch name, the CI run ID, and the Airflow version. Any deployed artifact must be traceable to a specific commit and CI run.

Common interview probes on Airflow CI.

"What's the first CI check for Airflow?" — required answer: DAG parse gate (DagBag(import_errors={})).
"Why does DAG parse fail differently from operator execution?" — required answer: parse errors take down every DAG in the DagBag, not just the broken one.
"How do you unit-test a custom operator?" — instantiate, provide mock context, assert on return / mock calls.
"How do you handle Airflow Variables in CI?" — default-value pattern; do not connect to prod metadata DB from CI.
"How do you version a deployable Airflow artifact?" — git SHA + branch + CI run ID + Airflow version, baked into the image label or zip metadata.

Worked example — GitHub Actions matrix for DAG parse + operator unit tests

Detailed explanation. The canonical Airflow CI matrix: one job for DAG parse (fastest), one job for the operator unit tests (parallel), one job (conditional) for the DAG integration test. Build the whole thing from scratch.

Matrix. parse (always), unit (always), integration (only if DAG files changed).
Runners. ubuntu-latest for parse + unit; ubuntu-latest with docker-compose for integration.
Python. 3.11 (match the Airflow deployment).
Airflow. 2.10.x pinned to match prod.

Question. Write the GitHub Actions workflow that runs parse + unit tests + optional integration on every airflow-touching PR.

Input.

Job	Runtime	When it runs
dag-parse	15-30 s	every airflow-touching PR
operator-unit	2-3 min	every airflow-touching PR
dag-integration	4-6 min	only if `airflow/dags/**` changed

Code.

# .github/workflows/airflow.yml — parse, unit, integration
name: airflow-ci
on:
  workflow_call:

jobs:
  # 1. DAG parse gate — fastest fail, 15-30 s
  dag-parse:
    runs-on: ubuntu-latest
    env:
      AIRFLOW_HOME:                       /tmp/airflow
      AIRFLOW__CORE__LOAD_EXAMPLES:       'False'
      AIRFLOW__CORE__UNIT_TEST_MODE:      'True'
      AIRFLOW__CORE__EXECUTOR:            SequentialExecutor
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install Airflow + providers
        run: |
          pip install -r airflow/requirements-ci.txt
      - name: Initialize Airflow metadata DB
        run: airflow db migrate
      - name: Parse every DAG file
        run: python -m airflow.utils.cli_action_loggers ; python scripts/dag_parse_gate.py

  # 2. Operator + hook unit tests — 2-3 min
  operator-unit:
    runs-on: ubuntu-latest
    env:
      AIRFLOW_HOME:                       /tmp/airflow
      AIRFLOW__CORE__LOAD_EXAMPLES:       'False'
      AIRFLOW__CORE__UNIT_TEST_MODE:      'True'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install
        run: pip install -r airflow/requirements-ci.txt
      - name: Run unit tests
        run: pytest airflow/tests/operators airflow/tests/hooks -v --tb=short

  # 3. DAG integration — only when a DAG file changed
  dag-integration:
    needs: dag-parse
    runs-on: ubuntu-latest
    if: contains(join(github.event.pull_request.changed_files, ','), 'airflow/dags/')
    env:
      AIRFLOW_HOME:                       /tmp/airflow
      AIRFLOW__CORE__LOAD_EXAMPLES:       'False'
      AIRFLOW__CORE__EXECUTOR:            LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql://airflow:airflow@localhost/airflow
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER:     airflow
          POSTGRES_PASSWORD: airflow
          POSTGRES_DB:       airflow
        ports: [ '5432:5432' ]
        options: --health-cmd "pg_isready -U airflow"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install
        run: pip install -r airflow/requirements-ci.txt
      - name: Init metadata DB
        run: |
          airflow db migrate
          airflow connections add fs_default --conn-type fs
      - name: List changed DAG ids
        id: dags
        run: |
          changed=$(git diff --name-only origin/main -- 'airflow/dags/*.py' | \
                     xargs -I{} python -c 'import ast, sys, pathlib
p = pathlib.Path(sys.argv[1])
tree = ast.parse(p.read_text())
for n in ast.walk(tree):
    if isinstance(n, ast.Assign):
        for t in n.targets:
            if getattr(t, "id", None) == "DAG_ID":
                if isinstance(n.value, ast.Constant):
                    print(n.value.value)' {})
          echo "ids=${changed}" >> "$GITHUB_OUTPUT"
      - name: airflow dags test each changed DAG
        run: |
          for dag_id in ${{ steps.dags.outputs.ids }}; do
            echo "=== testing $dag_id ==="
            airflow dags test "$dag_id" 2026-07-30
          done

# scripts/dag_parse_gate.py — the parse gate assertion
"""Assert every DAG in airflow/dags/ imports without error.

The gate mirrors the check the Airflow scheduler performs — if any DAG
raises during import, the scheduler flags an `import_errors` entry and
the DAG is invisible until fixed. Catching this in CI stops broken DAGs
from ever reaching the scheduler.
"""
import sys
from pathlib import Path
from airflow.models.dagbag import DagBag


def main() -> int:
    dag_folder = Path("airflow/dags").resolve()
    if not dag_folder.exists():
        print(f"ERROR: {dag_folder} does not exist", file=sys.stderr)
        return 2

    dagbag = DagBag(dag_folder=str(dag_folder), include_examples=False)
    if dagbag.import_errors:
        print("DAG parse errors detected:", file=sys.stderr)
        for filepath, err in dagbag.import_errors.items():
            print(f"\n--- {filepath} ---", file=sys.stderr)
            print(err, file=sys.stderr)
        return 1

    print(f"OK — {len(dagbag.dags)} DAGs parsed with 0 import errors")
    for dag_id in sorted(dagbag.dags):
        print(f"  • {dag_id}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

# airflow/tests/operators/test_reverse_etl_operator.py — unit test example
from unittest.mock import MagicMock, patch
import pytest
from airflow.utils.context import Context
from airflow_plugins.operators.reverse_etl import ReverseEtlOperator


@pytest.fixture
def op():
    return ReverseEtlOperator(
        task_id="test_reverse_etl",
        source_table="analytics.customers",
        target_system="hubspot",
        target_object="contacts",
    )


def test_operator_builds_correct_payload(op):
    """Given a source row, the operator should build the expected API payload."""
    row = {"customer_id": 42, "email": "a@example.com", "first_name": "Ada"}
    payload = op._build_payload(row)
    assert payload == {
        "properties": {
            "email":      "a@example.com",
            "firstname":  "Ada",
        },
        "id": "42",
    }


@patch("airflow_plugins.operators.reverse_etl.SnowflakeHook")
@patch("airflow_plugins.operators.reverse_etl.HubspotHook")
def test_operator_execute_calls_hubspot(mock_hs, mock_sf, op):
    """Execute should read from Snowflake and POST to Hubspot."""
    mock_sf.return_value.get_records.return_value = [
        {"customer_id": 42, "email": "a@example.com", "first_name": "Ada"},
    ]
    ctx = MagicMock(spec=Context)
    op.execute(ctx)
    assert mock_hs.return_value.upsert_contact.call_count == 1
    call = mock_hs.return_value.upsert_contact.call_args
    assert call.kwargs["payload"]["properties"]["email"] == "a@example.com"

Step-by-step explanation.

The dag-parse job is the fastest possible fail. It installs Airflow, migrates a SQLite metadata DB (fast), and runs dag_parse_gate.py which instantiates a DagBag and asserts import_errors is empty. Under 30 seconds; catches the highest-impact class of DAG bug.
AIRFLOW__CORE__UNIT_TEST_MODE=True tells Airflow to use in-memory connections and skip heavy initialisation; AIRFLOW__CORE__EXECUTOR=SequentialExecutor avoids requiring a real executor. Both are CI-only environment tricks.
The operator-unit job runs pytest against airflow/tests/operators and airflow/tests/hooks. Custom operators get mocked hooks (@patch("...SnowflakeHook")); the test asserts on mock_hs.return_value.upsert_contact.call_args. Coverage bar: every custom operator has at least one happy-path and one error-path test.
The dag-integration job runs only if a DAG file changed. It spins up Postgres as a GitHub Actions service, migrates the Airflow metadata DB against Postgres, extracts the changed DAG IDs, and runs airflow dags test <dag_id> <date> on each. This runs the DAG end-to-end without the scheduler.
The AST-based extraction of DAG_ID from changed DAG files is a pragmatic way to run only the changed DAGs. Alternatives: parse the whole DagBag and match by file path; require every DAG to be tagged with the file it's in. Both work.

Output.

Gate	Runtime	Catches
dag-parse	15-30 s	import errors, syntax errors, missing imports
operator-unit	2-3 min	logic errors in custom operators / hooks
dag-integration	4-6 min	wiring errors, DAG-level task interactions

Rule of thumb. For every Airflow CI setup, run parse first (fastest fail), unit second (broad coverage), integration third (deep coverage on high-value DAGs). Never skip the parse gate; it is 30 seconds and catches the single most common Airflow bug.

Worked example — full-DAG integration test with the LocalExecutor

Detailed explanation. The DAG integration test catches wiring bugs the unit tests cannot: task A returns X but task B expects Y, dynamic task mapping produces the wrong fan-out, or a TaskGroup fails to render under a specific execution date. Walk through a concrete integration test for a DAG that dedupes an S3 landing zone into a Snowflake table.

DAG under test. s3_to_snowflake_dedupe — 4 tasks: sensor (waits for S3 file) → download → dedupe → merge.
Fixtures needed. LocalStack S3 (via moto), Snowflake mock (via snowflake.connector mock).
Execution date. A fixed date so the test is deterministic.

Question. Write an integration test that runs the whole DAG end-to-end and asserts the target Snowflake table receives the expected rows.

Input.

Component	Value
DAG under test	s3_to_snowflake_dedupe
S3 backend	moto (in-process)
Snowflake backend	monkey-patched connector
Execution date	2026-07-30
Fixture rows	3 rows, one duplicate

Code.

# airflow/tests/dags/test_s3_to_snowflake_dedupe.py
import os
import json
from datetime import datetime
from unittest.mock import patch, MagicMock

import pytest
import boto3
from moto import mock_aws

from airflow.models.dagbag import DagBag
from airflow.utils.state import State


@pytest.fixture(scope="module")
def dagbag():
    return DagBag(dag_folder="airflow/dags", include_examples=False)


@pytest.fixture
def s3_landing():
    with mock_aws():
        s3 = boto3.client("s3", region_name="us-east-1")
        s3.create_bucket(Bucket="landing")
        s3.put_object(
            Bucket="landing",
            Key="orders/2026/07/30/orders.jsonl",
            Body=b'{"order_id":1,"total":100}\n'
                 b'{"order_id":2,"total":200}\n'
                 b'{"order_id":1,"total":100}\n',   # duplicate
        )
        yield s3


@pytest.fixture
def snowflake_stub():
    """Replace snowflake.connector.connect with a MagicMock that captures MERGE calls."""
    with patch("snowflake.connector.connect") as mock_connect:
        mock_conn = MagicMock()
        mock_connect.return_value = mock_conn
        mock_cur = MagicMock()
        mock_conn.cursor.return_value.__enter__.return_value = mock_cur
        yield mock_cur


def test_s3_to_snowflake_dedupe_end_to_end(dagbag, s3_landing, snowflake_stub):
    """Run the DAG end-to-end and assert the MERGE receives 2 deduped rows."""
    dag = dagbag.get_dag("s3_to_snowflake_dedupe")
    assert dag is not None, "DAG not found"

    execution_date = datetime(2026, 7, 30)

    # airflow dags test — synchronous, no scheduler needed
    dag.test(execution_date=execution_date)

    # Verify the merge was executed with the deduped payload
    merge_calls = [
        c for c in snowflake_stub.execute.call_args_list
        if "MERGE" in (c.args[0] if c.args else "")
    ]
    assert len(merge_calls) == 1, f"expected 1 MERGE call, got {len(merge_calls)}"

    merge_sql = merge_calls[0].args[0]
    assert "orders_dedupe_stage" in merge_sql
    # Row count sanity
    for ti in dag.get_task_instances():
        assert ti.state in {State.SUCCESS, State.SKIPPED}, (
            f"task {ti.task_id} ended in state {ti.state}"
        )

Step-by-step explanation.

The dagbag fixture is module-scoped so parsing runs once for the whole test file. Repeated DagBag() calls per test would inflate CI time by 10-30 seconds per test.
moto provides an in-process S3 mock that behaves like real S3 (bucket ops, put_object, list_objects_v2) without any network calls. mock_aws() is the modern moto 5.x context manager.
The Snowflake stub replaces snowflake.connector.connect at import time, so any operator code that calls snowflake.connector.connect() receives a MagicMock instead of a real connection. All execute calls are captured in mock_cur.execute.call_args_list.
dag.test(execution_date=...) (added in Airflow 2.5) runs the entire DAG synchronously in-process. No scheduler, no executor, no workers — just a straight-through execution. This is the CI-friendly way to run a DAG; airflow dags test from the CLI is the equivalent.
The assertions walk the captured Snowflake calls, find the MERGE statement, and verify the deduped row count. Alternative assertion styles: parse the MERGE SQL and check the row count; assert on the task's XCom returns; check the mock's total call count.

Output.

Task	State	Notes
wait_for_s3	SUCCESS	poked once; file present
download_landing	SUCCESS	3 rows from S3
dedupe_orders	SUCCESS	2 unique order_ids
merge_to_snowflake	SUCCESS	MERGE call captured with 2 rows

Rule of thumb. For DAGs that touch high-value shared resources (a warehouse table, a critical topic), invest in a dag.test()-based integration test with moto/monkey-patch stubs. For simpler DAGs, the parse gate + operator unit tests are sufficient. Don't try to integration-test every DAG — the maintenance cost outstrips the value.

Worked example — build and version a deployable Airflow image

Detailed explanation. After CI passes, the last CI job builds the deployable artifact. For self-hosted Kubernetes Airflow, this is an OCI image; for MWAA, it's a dags.zip + requirements.txt uploaded to S3; for Astronomer, it's astro deploy. Walk through the OCI image path — the most versatile.

Base. apache/airflow:2.10.3-python3.11.
Overlay. Copy airflow/dags, airflow/plugins, and install airflow/requirements-prod.txt.
Version labels. git SHA, branch, CI run ID, Airflow version.
Registry. ghcr.io/acme/airflow tagged with sha-<short> and pr-<PR_NUMBER> (for PRs) or main (for merges).

Question. Write the Dockerfile and the GitHub Actions job that builds, tags, and pushes the versioned image.

Input.

Component	Value
Base image	apache/airflow:2.10.3-python3.11
Registry	ghcr.io/acme/airflow
Tag scheme	`sha-<short>`, `pr-<PR>`, `main`
Labels	git SHA, branch, ci run id, airflow version

Code.

# airflow/Dockerfile
FROM apache/airflow:2.10.3-python3.11

ARG GIT_SHA=unknown
ARG GIT_BRANCH=unknown
ARG CI_RUN_ID=unknown

LABEL org.opencontainers.image.source="https://github.com/acme/data-monorepo"
LABEL org.opencontainers.image.revision="${GIT_SHA}"
LABEL org.opencontainers.image.ref.name="${GIT_BRANCH}"
LABEL ai.pipecode.ci.run.id="${CI_RUN_ID}"
LABEL ai.pipecode.airflow.version="2.10.3"

USER root
RUN apt-get update && apt-get install -y --no-install-recommends git \
 && rm -rf /var/lib/apt/lists/*
USER airflow

COPY --chown=airflow:root requirements-prod.txt /requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt

# DAGs and plugins baked into the image
COPY --chown=airflow:root dags     /opt/airflow/dags
COPY --chown=airflow:root plugins  /opt/airflow/plugins

# Sanity: parse gate at build time
RUN python -c "from airflow.models.dagbag import DagBag; \
               dagbag = DagBag('/opt/airflow/dags', include_examples=False); \
               assert not dagbag.import_errors, dagbag.import_errors; \
               print(f'{len(dagbag.dags)} DAGs, 0 errors')"

# .github/workflows/airflow-build.yml — build + push deployable image
name: airflow-build
on:
  workflow_call:

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents:  read
      packages:  write
      id-token:  write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Compute tags
        id: tags
        run: |
          short_sha=$(git rev-parse --short HEAD)
          echo "sha_tag=ghcr.io/acme/airflow:sha-${short_sha}" >> "$GITHUB_OUTPUT"
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            echo "extra_tag=ghcr.io/acme/airflow:pr-${{ github.event.pull_request.number }}" >> "$GITHUB_OUTPUT"
          else
            echo "extra_tag=ghcr.io/acme/airflow:main" >> "$GITHUB_OUTPUT"
          fi

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: airflow
          push:    true
          tags: |
            ${{ steps.tags.outputs.sha_tag }}
            ${{ steps.tags.outputs.extra_tag }}
          build-args: |
            GIT_SHA=${{ github.sha }}
            GIT_BRANCH=${{ github.ref_name }}
            CI_RUN_ID=${{ github.run_id }}
          cache-from: type=gha
          cache-to:   type=gha,mode=max

Step-by-step explanation.

The Dockerfile takes apache/airflow:2.10.3-python3.11 as its base — pinning the patch version prevents accidental upgrades. Use the same tag as prod for tightest parity.
Build args flow into LABELs so the running image self-describes: docker inspect ghcr.io/acme/airflow:sha-abc123 | jq '.[0].Config.Labels' reveals git SHA, branch, CI run ID, and Airflow version. Any deployed image can be traced back to a specific commit and CI run.
requirements-prod.txt is installed before the DAGs are copied so Docker's layer cache re-uses the pip layer across DAG-only changes — typical CI build drops from 90 s to 15 s on a DAG-only PR.
The RUN parse-gate at build time is a belt-and-braces check: even if the CI job's parse gate somehow passed a broken DAG, the image build will fail. This is the last chance to catch an import error before the image goes to the registry.
Tags: sha-<short> for every commit (immutable, provenance-preserving) plus pr-<PR_NUMBER> (mutable, points at latest CI build for the PR) or main (mutable, points at latest merge). The immutable sha tag is what deployments actually reference.

Output.

Trigger	Tags pushed	Traceable to
PR open / update	`sha-<short>`, `pr-<PR>`	commit SHA + CI run ID
Merge to main	`sha-<short>`, `main`	commit SHA + CI run ID
Release tag	`sha-<short>`, `v<VERSION>`	commit SHA + git tag

Rule of thumb. For every Airflow deployment, bake DAGs into the image, parse-gate at build time as a safety net, and use immutable sha-<short> tags for actual deployments. The mutable main / pr-<PR> tags are for humans; deployments reference the immutable ones.

Senior interview question on Airflow CI

A senior interviewer might ask: "Your Airflow deployment (MWAA on Airflow 2.10) has 180 DAGs. Once a month, a DAG merges that breaks at import time and the scheduler pauses for the whole DagBag. Design the CI workflow that catches this before merge — the parse gate, the operator unit-test coverage bar, the DAG integration test policy for high-value DAGs, and the deployable-artifact contract including version labels."

Solution Using a parse gate + unit matrix + integration matrix + labelled OCI image

# .github/workflows/airflow-full.yml — production-grade Airflow CI
name: airflow-ci
on:
  workflow_call:

jobs:
  # 1. Parse gate — fastest fail
  parse:
    runs-on: ubuntu-latest
    env:
      AIRFLOW_HOME:                       /tmp/airflow
      AIRFLOW__CORE__LOAD_EXAMPLES:       'False'
      AIRFLOW__CORE__UNIT_TEST_MODE:      'True'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r airflow/requirements-ci.txt
      - run: airflow db migrate
      - run: python scripts/dag_parse_gate.py

  # 2. Operator/hook unit tests — parallel
  unit:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        suite: [ operators, hooks, sensors, utils ]
    env:
      AIRFLOW_HOME:                       /tmp/airflow
      AIRFLOW__CORE__LOAD_EXAMPLES:       'False'
      AIRFLOW__CORE__UNIT_TEST_MODE:      'True'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r airflow/requirements-ci.txt
      - name: pytest ${{ matrix.suite }}
        run: pytest airflow/tests/${{ matrix.suite }} -v --tb=short

  # 3. Integration tests for changed DAGs
  integration:
    needs: parse
    runs-on: ubuntu-latest
    if: contains(join(github.event.pull_request.changed_files, ','), 'airflow/dags/')
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER:     airflow
          POSTGRES_PASSWORD: airflow
          POSTGRES_DB:       airflow
        ports: [ '5432:5432' ]
    env:
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql://airflow:airflow@localhost/airflow
      AIRFLOW__CORE__EXECUTOR:             LocalExecutor
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r airflow/requirements-ci.txt
      - run: airflow db migrate
      - run: pytest airflow/tests/integration -v --tb=short

  # 4. Build labelled deployable OCI image
  build-image:
    needs: [ parse, unit, integration ]
    if: always() && needs.parse.result == 'success' && needs.unit.result == 'success' && (needs.integration.result == 'success' || needs.integration.result == 'skipped')
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build + push
        uses: docker/build-push-action@v6
        with:
          context: airflow
          push:    true
          tags: |
            ghcr.io/acme/airflow:sha-${{ github.sha }}
            ghcr.io/acme/airflow:pr-${{ github.event.pull_request.number }}
          build-args: |
            GIT_SHA=${{ github.sha }}
            GIT_BRANCH=${{ github.ref_name }}
            CI_RUN_ID=${{ github.run_id }}

# Airflow CI budget contract (per gate)
Gate           p95 latency   Coverage target
─────────────────────────────────────────────
parse          30 s          100% of DAG files
unit-operators 3 min         95% line coverage on custom operators
unit-hooks     2 min         95% line coverage on custom hooks
integration    6 min         100% of DAGs touching shared warehouse tables
build-image    2 min         every merge; every PR

Step-by-step trace.

Gate	Config	Result
parse	DagBag(include_examples=False) + assert not import_errors	catches syntax + import errors
unit (matrix)	pytest airflow/tests/{operators,hooks,sensors,utils}	catches operator logic errors
integration	LocalExecutor + Postgres service + pytest airflow/tests/integration	catches DAG wiring errors
build-image	apache/airflow:2.10.3 + copy DAGs + build-time parse check + labels	produces immutable sha-tagged artifact
Version labels	GIT_SHA / GIT_BRANCH / CI_RUN_ID as OCI LABELs	every deployment traceable

After deployment, the once-a-month "broken DAG paused the scheduler" incident drops to zero — the parse gate catches every broken import before merge. The p95 CI time for an Airflow-only PR is about 5 minutes; the p95 cost is about $0.15. The operator unit-test coverage bar (95%) is enforced by pytest-cov + --cov-fail-under=95. Every deployed image has git SHA + branch + CI run ID + Airflow version in its labels, so any prod incident can be traced back to a specific commit in seconds.

Output:

Metric	Before	After
Broken-DAG incidents	~1/month	~0
p95 CI time (Airflow-only PR)	12 min	5 min
p95 CI cost (Airflow-only PR)	$0.60	$0.15
Operator unit-test coverage	~40%	95% (enforced)
Image provenance	none	git SHA + branch + run ID
DAG integration coverage	0 DAGs	~30 high-value DAGs

Why this works — concept by concept:

Parse gate first — the single highest-ROI CI check. 30 seconds, one assertion, catches the highest-impact class of Airflow bug (import-time errors that pause the whole DagBag). Every Airflow project should have this on day one.
Unit-test matrix — pytest matrix parallelises the operator / hook / sensor / util suites. Coverage bar (95%) enforced in CI via --cov-fail-under. Regressions in custom operator logic caught before merge.
Integration test for shared-resource DAGs — LocalExecutor + Postgres service + dag.test() runs the whole DAG end-to-end in ~1 minute per DAG. Scoped to DAGs that touch shared warehouse tables so the CI time doesn't explode.
Labelled OCI image — every image carries git SHA, branch, CI run ID, and Airflow version as OCI labels. docker inspect on any deployed image reveals its provenance in one command. Immutable sha-<short> tag is the actual deployment target.
Build-time parse gate — a belt-and-braces RUN python -c "DagBag(...)" inside the Dockerfile. If the CI parse-gate somehow passed a broken DAG (e.g. Airflow version mismatch), the image build fails before push. Defence in depth.
Cost — one parse gate ($0.02), one unit matrix ($0.05), one conditional integration ($0.08), one image build ($0.02). Total ~$0.15 per PR. Compared to the "broken DAG paused scheduler for 4 hours" incident cost (~$800 in delayed data), this is essentially free. Net O(changed) per PR versus O(catalog) impact of every prod incident.

Python
Topic — python
Python operator and DAG testing problems

Practice →

API Topic — api-integration API integration problems on Airflow hooks

Practice →

4. Spark CI — packaged jobs, unit + integration tests, dry-run submits

`spark-submit --deploy-mode client --num-executors 0` is the dry-run gate every Spark CI needs before the real cluster is billed

The mental model in one line: spark ci cd is the pattern where every PR that touches spark/ runs three gates — packaged JAR / wheel that carries the job entry point + its Python / Scala dependencies, local unit tests against a SparkSession.builder.master("local[2]") fixture that catches logic and schema bugs, and a dry-run spark-submit against a small dev cluster (or a docker-compose Spark mini-cluster) that validates cluster reachability, JAR resolution, and config before executors are ever provisioned — because a Spark job that fails after 40 minutes of shuffle costs orders of magnitude more than a job that fails in 30 seconds of local pytest. Every senior Spark deployment has migrated to this pattern once; skipping the local-unit + dry-run gates turns "one bad UDF" into "40-minute cluster time wasted plus a 3 AM page."

The four axes for Spark CI.

Correctness. Three levels: unit (does the transformation logic produce the right output for known input?), schema (does the job's output schema match downstream expectations?), integration (does spark-submit actually accept the config in the target cluster?). Each layer catches a different class of bug.
Cost. Local SparkSession is free (runs on the GHA runner). Docker-compose mini-cluster is ~$0.05 per run. Dry-run submit against a dev cluster is ~$0.02. The cost trap is running the real job on the real cluster from CI — that's how a $500 CI run happens.
Latency-to-merge. Local pytest in 3-4 minutes. Dry-run in 30-60 seconds. A Spark-only PR should complete CI in under 6 minutes p95.
Blast radius. Local Spark writes to a tempdir. Docker-compose Spark writes to a container-scoped volume. Dry-run submit does not write at all. The real cluster is only touched at deploy time, never at CI time.

The packaged Spark job — one artifact, one entry point.

PySpark. pip wheel producing job-1.2.3-py3-none-any.whl with entry_points = {"console_scripts": ["run-job = mypkg.entry:main"]}. spark-submit --py-files job-1.2.3.whl --archives venv.tar.gz#env script.py.
Scala/Java. sbt assembly producing a fat JAR job-1.2.3-assembly.jar. spark-submit --class com.acme.Job job.jar.
Config. Job-specific YAML (config/prod.yaml, config/dev.yaml) selected by env var; never baked into the artifact.
Version. Semver tag + git SHA baked into a --version command that prints the SHA. Enables prod-side triage — "which commit is this?" is one flag away.

The local SparkSession fixture — the unit-test workhorse.

The fixture. pytest session-scoped fixture creating SparkSession.builder.master("local[2]").appName("ci").getOrCreate(). Two cores is enough for logic tests; more is wasted CI time.
What to test. Pure transformations (def dedupe(df: DataFrame) -> DataFrame:) against known-input DataFrames; UDFs against small Python lists; schema-enforcement (assertSchemaEqual).
What not to test. Anything requiring shuffle at scale — leave that to the integration test. Anything requiring cluster-specific config (Kryo serializer settings, dynamic allocation) — same.
Test data. Small in-memory DataFrames (spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"])) or small JSON fixtures. Never load prod data.

The docker-compose mini-cluster — one CI gate for shuffle behaviour.

The setup. bitnami/spark:3.5 for master + 2 workers in docker-compose. About 90 seconds to spin up on GHA.
When to use. Jobs that exercise shuffle, partitioning, or broadcast joins. Local mode does not test these accurately at CI scale.
When to skip. Pure UDF / column-arithmetic jobs — local mode is enough.

The dry-run spark-submit — the cluster-config gate.

What it does. Submits the job with --num-executors 0 (or --conf spark.dynamicAllocation.enabled=false --conf spark.executor.instances=0), so the driver starts, resolves the JAR, checks Kerberos / IAM, but no executors spin up.
What it catches. Missing JAR dependencies, mistyped config keys, cluster-reachability, auth failures, --files resolution errors. Everything that would fail in the first 30 seconds of a real submit.
What it does not catch. Actual data processing bugs — leave those to the unit tests.
The Databricks equivalent. POST to /api/2.1/jobs/runs/submit-and-wait with --dry-run=true (or preview via the CLI); validates the job config against the workspace.

Common interview probes on Spark CI.

"How do you unit-test a PySpark job?" — required answer: pytest fixture with local SparkSession.
"How do you validate cluster config in CI without spinning up a cluster?" — dry-run submit with --num-executors 0.
"What's the packaging story for PySpark vs Scala?" — wheel vs fat JAR.
"How do you test a UDF?" — pytest with the UDF's Python function directly, plus a Spark-side sanity test.
"How do you avoid running the real job on the real cluster in CI?" — packaging + local mode + dry-run. The real cluster is never touched by CI.

Worked example — pytest against a local SparkSession

Detailed explanation. The canonical PySpark unit test: a session-scoped SparkSession fixture that constructs a local[2] session once per pytest run, plus per-test DataFrames built inline. Walk through a full test module for a dedupe transformation.

Fixture. spark_session fixture, session scope, local[2].
Transformation under test. dedupe_orders(df: DataFrame) -> DataFrame.
Assertions. Row count, schema, specific column values via .collect().

Question. Write the pytest module for a dedupe function with three test cases.

Input.

Test	Input rows	Expected rows
test_basic_dedupe	3 rows (1 duplicate)	2
test_all_duplicate	5 identical rows	1
test_no_duplicates	5 distinct rows	5

Code.

# spark/tests/conftest.py — the shared SparkSession fixture
import pytest
from pyspark.sql import SparkSession


@pytest.fixture(scope="session")
def spark():
    """A local Spark session shared across the pytest session."""
    spark = (
        SparkSession.builder
        .master("local[2]")
        .appName("ci")
        .config("spark.sql.shuffle.partitions", "2")
        .config("spark.ui.enabled", "false")
        .config("spark.sql.session.timeZone", "UTC")
        .getOrCreate()
    )
    yield spark
    spark.stop()

# spark/tests/test_dedupe_orders.py
from pyspark.sql import Row
from mypkg.transforms import dedupe_orders


def test_basic_dedupe(spark):
    df_in = spark.createDataFrame([
        Row(order_id=1, total=100),
        Row(order_id=2, total=200),
        Row(order_id=1, total=100),
    ])
    df_out = dedupe_orders(df_in)
    assert df_out.count() == 2
    ids = sorted(r.order_id for r in df_out.collect())
    assert ids == [1, 2]


def test_all_duplicate(spark):
    df_in = spark.createDataFrame([Row(order_id=42, total=1)] * 5)
    df_out = dedupe_orders(df_in)
    assert df_out.count() == 1


def test_no_duplicates(spark):
    df_in = spark.createDataFrame([
        Row(order_id=i, total=i * 10) for i in range(1, 6)
    ])
    df_out = dedupe_orders(df_in)
    assert df_out.count() == 5


def test_schema_preserved(spark):
    df_in = spark.createDataFrame([
        Row(order_id=1, total=100, extra="x"),
        Row(order_id=1, total=100, extra="x"),
    ])
    df_out = dedupe_orders(df_in)
    assert df_out.schema == df_in.schema

# mypkg/transforms.py — the transformation under test
from pyspark.sql import DataFrame


def dedupe_orders(df: DataFrame) -> DataFrame:
    """Return the DataFrame with duplicate rows removed (all-column key)."""
    return df.dropDuplicates()

Step-by-step explanation.

The spark fixture is session-scoped, so the JVM starts up once for the whole test run — not once per test. This is critical: SparkSession.builder.getOrCreate() is ~2-3 seconds of JVM startup, and per-test creation would inflate a 50-test suite by 2-3 minutes.
spark.sql.shuffle.partitions=2 overrides the default 200 for the local session; with 2 cores, 200 partitions creates enormous scheduling overhead that dominates the test time. spark.ui.enabled=false skips the Spark UI web server (unneeded in CI).
spark.createDataFrame(rows, schema) builds an in-memory DataFrame directly. No file IO, no serialization overhead — the fastest way to construct test data. Row gives typed named-tuple-like construction.
.count() and .collect() are the assertion primitives. For larger expected outputs, use .toPandas() + pandas assertions; for schema, use .schema equality or assertSchemaEqual from pyspark.testing.
The test_schema_preserved case is a common trap: transformations often silently drop columns or change types. Asserting schema equality catches this regression class.

Output.

Test	Runtime	Fixture cost
First test (JVM startup)	~3 s	one-time
Subsequent tests	~50-200 ms each	none
Full 50-test suite	~30-40 s total	amortised

Rule of thumb. For every PySpark project, ship a session-scoped SparkSession fixture with local[2] + shuffle=2 + ui=false + UTC timezone. Then every unit test is one createDataFrame + one transformation call + one assertion. Under 200 ms per test after the JVM warms.

Worked example — docker-compose Spark mini-cluster

Detailed explanation. For jobs that exercise shuffle, broadcast joins, or specific spark-defaults.conf behaviour, local mode is insufficient — you need a real cluster with executor JVMs. Docker-compose is the CI-friendly way to spin one up in about 90 seconds. Walk through the compose file and a pytest that submits a job against it.

Compose services. spark-master + 2 × spark-worker + optional spark-history.
Base image. bitnami/spark:3.5.1 (matches prod Spark version).
Job submission. docker exec spark-master spark-submit --master spark://spark-master:7077 ….

Question. Write the docker-compose file and a pytest that spins the cluster, submits a job, and asserts the output.

Input.

Component	Value
Spark version	3.5.1
Master + workers	1 + 2
Master port	7077
Job type	JAR
Assertion	output S3 prefix has expected file count

Code.

# spark/tests/integration/docker-compose.yml
services:
  spark-master:
    image: bitnami/spark:3.5.1
    environment:
      - SPARK_MODE=master
      - SPARK_LOCAL_IP=spark-master
    ports:
      - "7077:7077"
      - "8080:8080"
    volumes:
      - ./workdir:/workdir
      - ../../dist:/opt/jobs

  spark-worker-1:
    image: bitnami/spark:3.5.1
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=2g
      - SPARK_WORKER_CORES=2
    depends_on: [ spark-master ]
    volumes: [ '../../dist:/opt/jobs' ]

  spark-worker-2:
    image: bitnami/spark:3.5.1
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=2g
      - SPARK_WORKER_CORES=2
    depends_on: [ spark-master ]
    volumes: [ '../../dist:/opt/jobs' ]

# spark/tests/integration/test_dedupe_cluster.py
import subprocess
import time
from pathlib import Path
import pytest


COMPOSE_FILE = Path(__file__).parent / "docker-compose.yml"


@pytest.fixture(scope="module", autouse=True)
def cluster():
    subprocess.check_call(["docker", "compose", "-f", str(COMPOSE_FILE), "up", "-d"])
    # Wait for master to be ready
    for _ in range(30):
        try:
            r = subprocess.run(
                ["docker", "compose", "-f", str(COMPOSE_FILE), "exec", "-T",
                 "spark-master", "curl", "-fsS", "http://localhost:8080"],
                check=True, capture_output=True,
            )
            break
        except subprocess.CalledProcessError:
            time.sleep(2)
    else:
        raise RuntimeError("Spark master did not come up in 60 s")
    yield
    subprocess.check_call(["docker", "compose", "-f", str(COMPOSE_FILE), "down", "-v"])


def test_dedupe_job_end_to_end(cluster, tmp_path):
    """Submit the dedupe job against the mini-cluster and assert output."""
    input_path  = tmp_path / "in"
    output_path = tmp_path / "out"
    input_path.mkdir()
    (input_path / "orders.jsonl").write_text(
        '{"order_id":1,"total":100}\n'
        '{"order_id":2,"total":200}\n'
        '{"order_id":1,"total":100}\n'
    )

    subprocess.check_call([
        "docker", "compose", "-f", str(COMPOSE_FILE), "exec", "-T",
        "spark-master", "spark-submit",
        "--master", "spark://spark-master:7077",
        "--class", "com.acme.DedupeJob",
        "/opt/jobs/dedupe-job-1.0.0.jar",
        "--input",  f"/workdir/in",
        "--output", f"/workdir/out",
    ])

    # Assert output exists and has 2 rows
    result = (tmp_path / "out").glob("*.json")
    files  = list(result)
    assert files, "no output produced"
    lines = sum(1 for f in files for _ in f.read_text().splitlines())
    assert lines == 2, f"expected 2 output rows, got {lines}"

Step-by-step explanation.

The compose file defines master + 2 workers using bitnami/spark:3.5.1. Ports 7077 (master) and 8080 (UI) are published so the pytest can healthcheck via HTTP. Volume mounts share the JAR and the working directory between host and containers.
The cluster fixture is module-scoped so the cluster spins up once per test module. docker compose up -d runs detached; the polling loop hits the master UI on 8080 until it responds — typically 60-90 seconds.
Job submission runs spark-submit inside the master container via docker compose exec. This uses the master's classpath and network — no host-side Spark install needed on the GHA runner.
The mounted /workdir shares the input JSON and receives the output — no S3 dependency; everything stays local to the compose stack. This keeps the integration test hermetic and fast.
The test asserts on output-file existence and row count. For richer assertions, load the output with pandas or run a validation Spark job. The teardown (down -v) removes volumes so nothing persists.

Output.

Phase	Runtime
docker compose up	45-60 s (first run); 20-30 s (cached image)
Master healthcheck	15-30 s
Job submit + run	30-60 s
docker compose down	5 s
Total (first run)	~2 min

Rule of thumb. Reserve the docker-compose Spark cluster for jobs that need real shuffle / broadcast / partitioning behaviour. For pure-transformation jobs, local mode is enough and finishes in 3-5 minutes total. The compose cluster adds 1-2 minutes for the environment; only pay that cost when the test needs it.

Worked example — dry-run submit that validates cluster config

Detailed explanation. The dry-run submit is the cheapest cluster-level gate: submit the job to a real dev cluster with --num-executors 0, so the driver starts, JAR resolves, config validates, but no executors spin up. If the submit is accepted, the cluster will accept the real submit; if it fails, you catch the config bug in 30 seconds instead of after a 40-minute allocation wait.

The invocation. spark-submit --master yarn --deploy-mode client --num-executors 0 --class com.acme.Job job.jar --dry-run.
What passes. JAR resolution, --files resolution, Kerberos ticket, HDFS access.
What fails. Missing JAR, mistyped --conf, wrong --class, invalid YARN queue.

Question. Write the GHA step and a wrapper script that runs the dry-run submit against a dev EMR cluster.

Input.

Component	Value
Target cluster	dev EMR (via IAM role)
Executors	0 (dry run)
Timeout	60 s
Exit code contract	0 = accepted; non-zero = config bug

Code.

#!/usr/bin/env bash
# scripts/spark_dry_run.sh
set -euo pipefail

JAR="$1"                # path to fat JAR
CLASS="$2"              # main class
CLUSTER_ID="${DEV_EMR_CLUSTER_ID}"

# Submit via EMR CLI with --dry-run-esque args
step_config=$(cat <<EOF
[
  {
    "Type": "Spark",
    "Name": "ci-dry-run-$(git rev-parse --short HEAD)",
    "ActionOnFailure": "CONTINUE",
    "Args": [
      "spark-submit",
      "--deploy-mode", "cluster",
      "--num-executors", "0",
      "--conf", "spark.dynamicAllocation.enabled=false",
      "--conf", "spark.executor.instances=0",
      "--conf", "spark.driver.memory=512m",
      "--conf", "spark.app.name=ci-dry-run",
      "--class", "${CLASS}",
      "${JAR}",
      "--dry-run"
    ]
  }
]
EOF
)

step_id=$(aws emr add-steps \
  --cluster-id "${CLUSTER_ID}" \
  --steps "${step_config}" \
  --query 'StepIds[0]' --output text)

echo "Submitted step ${step_id}; polling…"

for _ in $(seq 1 20); do
  state=$(aws emr describe-step --cluster-id "${CLUSTER_ID}" --step-id "${step_id}" \
          --query 'Step.Status.State' --output text)
  echo "state=${state}"
  case "${state}" in
    COMPLETED)      exit 0 ;;
    FAILED|CANCELLED) exit 1 ;;
    *)              sleep 3 ;;
  esac
done

echo "timeout waiting for dry-run to complete" >&2
exit 2

# GHA step
- name: Spark dry-run against dev cluster
  env:
    AWS_REGION: us-east-1
    DEV_EMR_CLUSTER_ID: ${{ secrets.DEV_EMR_CLUSTER_ID }}
  run: |
    bash scripts/spark_dry_run.sh \
      s3://acme-artifacts/spark/dedupe-job-${{ github.sha }}.jar \
      com.acme.DedupeJob

// spark/src/main/scala/com/acme/DedupeJob.scala — dry-run branch
object DedupeJob {
  def main(args: Array[String]): Unit = {
    val cliArgs = parseArgs(args)
    if (cliArgs.dryRun) {
      println("DRY RUN — validating config only")
      val spark = SparkSession.builder.appName("dedupe-dry-run").getOrCreate()
      // Touch config to force resolution
      println(s"master=${spark.conf.get("spark.master")}")
      println(s"appName=${spark.conf.get("spark.app.name")}")
      spark.stop()
      System.exit(0)
    }
    // real job body …
  }
}

Step-by-step explanation.

--num-executors 0 + spark.dynamicAllocation.enabled=false tells the cluster manager "start the driver, but don't allocate any executors." The driver comes up, resolves the JAR, checks config, then exits immediately — 30-60 seconds end to end.
Submitting via aws emr add-steps uses the EMR steps API rather than SSHing to the master node. The step ID is returned; the polling loop calls describe-step until the state is terminal.
The job's --dry-run flag is recognised in the main class; if set, the job constructs a SparkSession (which forces config resolution) and exits 0. This catches bad --conf values that would otherwise only surface once the real job started reading data.
The exit-code contract: 0 = accepted, non-zero = config bug. The GHA step fails fast on non-zero, blocking the PR merge until the config is fixed.
The dev cluster is deliberately small (single m5.xlarge master) and shared across CI jobs — it's OK because dry-runs are 30 seconds each. Concurrent CI runs queue up, but no one waits long.

Output.

Failure mode	Caught by dry-run?
Missing JAR at S3	yes (JAR resolution fails)
Mistyped `--conf` key	partially (some keys validated at driver start)
Wrong `--class`	yes (ClassNotFoundException at driver init)
Invalid YARN queue	yes (rejected before executors requested)
Data-processing bug	no (need unit tests)
Shuffle-size bug	no (need integration test)

Rule of thumb. For every Spark job, ship a --dry-run code path in the main class that constructs a SparkSession and exits 0. Wire the dry-run into CI as the last gate before merge. This is the cheapest possible "will this run in prod?" check.

Senior interview question on Spark CI

A senior interviewer might ask: "You inherit a Spark job pipeline where CI submits every PR to the prod cluster to test it, burning ~$400/day in wasted compute. Design the CI redesign: packaging, local unit tests, docker-compose integration for high-value jobs, and a dry-run gate. Show me the CI budget, the coverage targets, and the on-call runbook when the dry-run fails on a working PR."

Solution Using packaged JAR + local pytest + docker-compose mini-cluster + dry-run submit

# .github/workflows/spark-full.yml — production-grade Spark CI
name: spark-ci
on:
  workflow_call:

jobs:
  # 1. Package the job — one artifact, reused downstream
  package:
    runs-on: ubuntu-latest
    outputs:
      jar_key: ${{ steps.upload.outputs.key }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: '17' }
      - name: sbt assembly
        working-directory: spark
        run: sbt assembly
      - name: Upload JAR
        id: upload
        env:
          KEY: spark/dedupe-job-${{ github.sha }}.jar
        run: |
          aws s3 cp spark/target/scala-2.13/dedupe-job-assembly-*.jar \
                    s3://acme-artifacts/${KEY}
          echo "key=${KEY}" >> "$GITHUB_OUTPUT"

  # 2. Local unit tests — fast, cheap
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: '17' }
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install PySpark for tests
        run: pip install -r spark/requirements-ci.txt
      - name: PySpark unit tests
        run: pytest spark/tests/unit --cov=mypkg --cov-fail-under=90
      - name: Scala unit tests
        working-directory: spark
        run: sbt test

  # 3. Integration — only for shuffle-heavy jobs
  integration:
    if: contains(join(github.event.pull_request.changed_files, ','), 'spark/jobs/')
    runs-on: ubuntu-latest
    needs: package
    steps:
      - uses: actions/checkout@v4
      - name: docker-compose spark cluster + integration test
        run: |
          cd spark/tests/integration
          docker compose up -d
          for i in $(seq 1 30); do
            docker compose exec -T spark-master curl -fsS http://localhost:8080 && break
            sleep 2
          done
          pip install pytest pyspark
          pytest .
          docker compose down -v

  # 4. Dry-run submit — cluster gate
  dry-run:
    if: needs.unit.result == 'success' && needs.package.result == 'success'
    needs: [ package, unit ]
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-spark-dry-run
          aws-region: us-east-1
      - env:
          DEV_EMR_CLUSTER_ID: ${{ secrets.DEV_EMR_CLUSTER_ID }}
        run: |
          bash scripts/spark_dry_run.sh \
            s3://acme-artifacts/${{ needs.package.outputs.jar_key }} \
            com.acme.DedupeJob

# Spark CI budget contract
Gate           p95 latency   Cost budget   Coverage
─────────────────────────────────────────────────────
package         60 s         $0.02         100% jobs
unit            4 min        $0.03         90% line coverage
integration     6 min        $0.05         high-value jobs only
dry-run         60 s         $0.02         100% jobs before merge
Total (typical) 6 min        $0.12

Step-by-step trace.

Gate	Config	Result
package	sbt assembly + upload to s3	one immutable JAR per commit SHA
unit	pytest + sbt test, 90% coverage floor	catches logic + schema bugs
integration	docker-compose spark-3.5.1 mini-cluster	catches shuffle + join bugs
dry-run	EMR add-steps with --num-executors 0	catches cluster-config bugs
Prod cluster touched?	never in CI	cost isolated to deploy

After deployment, CI submits zero real jobs to the prod cluster — the daily $400 wasted-compute bill goes to zero. The p95 CI time for a Spark-only PR is about 6 minutes; the p95 cost is about $0.12. Unit test coverage on the transformation package is enforced at 90% via --cov-fail-under=90. Every merged commit has a JAR at s3://acme-artifacts/spark/dedupe-job-<sha>.jar, so prod rollouts and rollbacks are one S3 pointer change.

Output:

Metric	Before	After
Wasted-compute bill (CI)	~$400/day	$0
p95 CI time (Spark-only PR)	42 min	6 min
p95 CI cost per PR	$8	$0.12
Unit-test coverage	~30%	90% (enforced)
Job-versioning story	none	immutable JAR per SHA
Broken-config incidents	~2/week	~0

Why this works — concept by concept:

Packaged JAR / wheel per commit — one immutable artifact per SHA, keyed by git SHA. Every downstream test uses the same artifact; prod deploys reference it by SHA. This kills the "which version is in prod?" question at its root.
Local SparkSession fixture — fastest possible unit-test loop. local[2] + shuffle=2 + ui=false keeps the fixture 3 seconds to warm and 100-200 ms per test after. The 90% coverage floor is a contract, not a hope; --cov-fail-under=90 enforces it.
Docker-compose mini-cluster — scoped to jobs that actually need real shuffle. Adds ~2 minutes to CI when it runs, but stays gated on spark/jobs/** path filter so most PRs skip it.
Dry-run submit — the cheapest cluster-level gate. --num-executors 0 starts the driver, resolves everything, exits — 30-60 seconds. Catches config, JAR, class, queue bugs before the real submit would.
Never touch the prod cluster from CI — the invariant. All gates run on the GHA runner, a local docker-compose stack, or a small dev EMR cluster. Prod compute is only billed for prod work.
Cost — $0.02 package + $0.03 unit + optional $0.05 integration + $0.02 dry-run = $0.12 per PR typical. Compared to the $8/PR baseline (mostly wasted prod-cluster time), that's a 60× cost reduction. Net O(local) per PR versus O(cluster) per PR.

Python
Topic — python
Python data-transformation testing problems

Practice →

Streaming Topic — streaming Streaming and Spark job problems

Practice →

5. Preview environments — one branch, one warehouse schema

`pr_<PR>_<branch>` schema on Snowflake / Databricks UC — created on PR open, dropped on PR close

The mental model in one line: dbt preview environment is the pattern where every open pull request maps to its own throwaway warehouse schema named pr_<PR>_<branch> — created on PR open via CREATE SCHEMA ... CLONE prod (Snowflake / Databricks UC zero-copy), targeted by the slim-CI dbt build, exposed to analysts as a preview URL, and auto-dropped by a cleanup workflow when the PR closes — so every PR has an isolated, near-prod dataset for review without ever touching the prod schema and without paying for full-copy storage.

The four axes for preview environments.

Correctness. The preview schema is a near-prod dataset — clones of every unmodified table, freshly-built copies of changed models. Analysts querying the preview see exactly what production will look like after merge.
Cost. Snowflake and Databricks UC zero-copy clones are metadata operations — no data is physically duplicated, no storage is billed. XS warehouse with AUTO_SUSPEND=60 keeps compute cost under $0.10 per preview session.
Latency-to-preview. Slim build (~90 s) + clone (~5 s) + link posted to PR (~2 s) = analyst has the preview URL within 100 seconds of PR open.
Blast radius. Each preview lives in its own schema; concurrent PRs never collide. Cleanup on PR close reclaims the catalog entry.

Schema naming — deterministic and cleanable.

Format. pr_<PR_NUMBER>_<BRANCH_SLUG> — e.g. pr_4712_add_orders_v2.
Branch slug rules. Lowercase, strip non-alphanumeric, cap at 40 characters. Snowflake identifier limit is 255; capping at 40 leaves room for the pr_<num>_ prefix.
Case. Lowercase everywhere — Snowflake folds unquoted identifiers to uppercase; dbt writes lowercase by default. Consistency across CI + prod avoids "table not found" surprises.
Cleanup. A cleanup workflow triggers on pull_request closed (both merge and reject) and runs DROP SCHEMA IF EXISTS pr_<PR>_<branch> CASCADE.

Snowflake zero-copy CLONE.

Command. CREATE OR REPLACE SCHEMA analytics.pr_4712_add_orders_v2 CLONE analytics.prod;.
Semantics. All tables, views, and pipes in the source schema are cloned as metadata pointers. Reads work immediately; writes copy-on-write.
Cost. $0 storage until a write happens. A read-only preview costs storage $0.

Databricks Unity Catalog SCHEMA CLONE.

Command. CREATE SCHEMA <catalog>.pr_4712_add_orders_v2 DEEP CLONE <catalog>.prod; for deep, or SHALLOW CLONE for zero-copy.
Semantics. Shallow clone is metadata-only (like Snowflake); deep clone physically copies. Prefer shallow for previews.
Cost. Shallow clone = $0. Deep clone = full storage cost.

Ephemeral compute — AUTO_SUSPEND=60 is the invariant.

Snowflake. WAREHOUSE_SIZE=XSMALL, AUTO_SUSPEND=60, AUTO_RESUME=TRUE. XS warehouse is $1/hour; suspending after 60 s of idle keeps a typical CI session under 5 minutes of billed compute.
Databricks. Interactive clusters with autotermination_minutes: 10; SQL warehouses with auto_stop_mins: 5.
BigQuery. On-demand billing means no explicit suspend — every query is priced independently.

Cleanup on PR close — the non-negotiable half of the pattern.

The trigger. on: pull_request: types: [closed] in GitHub Actions.
The action. DROP SCHEMA IF EXISTS pr_<PR>_<branch> CASCADE — idempotent.
The safety net. Nightly cron that drops any pr_% schema older than 14 days — catches PRs abandoned without close.

Common interview probes on preview environments.

"How do you avoid full-copy storage cost for previews?" — zero-copy clone (Snowflake CREATE SCHEMA CLONE / Databricks SHALLOW CLONE).
"How do you name preview schemas?" — pr_<PR>_<branch_slug>, deterministic, cleanable.
"What if a PR is abandoned?" — cleanup workflow on closed + nightly cron for pr_% older than 14 days.
"How do analysts access the preview?" — preview URL posted to PR comment; Snowsight / Databricks SQL link with the schema pre-selected.
"What's the cost per preview?" — $0 storage (zero-copy), $0.05-$0.30 compute for a typical review session.

Worked example — create-preview-schema GitHub Action

Detailed explanation. The create-preview workflow runs on PR open (and on every push to the PR) and produces the pr_<PR>_<branch> schema. It's idempotent — CREATE OR REPLACE — so a re-push refreshes the preview against the latest changes.

Trigger. pull_request types: [opened, synchronize, reopened].
Action. Snowflake CREATE OR REPLACE SCHEMA ... CLONE prod.
Grant. Read-only access to the schema for the analyst role.

Question. Write the workflow and the SQL that creates a preview schema for a PR.

Input.

Component	Value
Trigger	pull_request opened / synchronize / reopened
Warehouse	DBT_CI_WH_XS (AUTO_SUSPEND=60)
Source schema	ANALYTICS.PROD
Target schema	ANALYTICS.pr__
Analyst grant	SELECT to ANALYST_ROLE

Code.

# .github/workflows/preview-schema-create.yml
name: preview-schema-create
on:
  workflow_call:

jobs:
  create-preview:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      pull-requests: write
    env:
      PR_NUMBER:    ${{ github.event.pull_request.number }}
      BRANCH_SLUG:  ${{ github.head_ref }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install snowflake-connector-python
      - name: Compute schema name
        id: schema
        run: |
          slug=$(echo "$BRANCH_SLUG" | tr '/' '_' | tr -cd '[:alnum:]_' | tr '[:upper:]' '[:lower:]' | cut -c1-40)
          echo "name=pr_${PR_NUMBER}_${slug}" >> "$GITHUB_OUTPUT"
      - name: Create preview schema
        env:
          SNOWFLAKE_ACCOUNT:     ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_USER:        dbt_ci
          SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
          SCHEMA_NAME:           ${{ steps.schema.outputs.name }}
        run: python scripts/create_preview_schema.py
      - name: Post preview URL to PR
        uses: actions/github-script@v7
        env:
          SCHEMA_NAME: ${{ steps.schema.outputs.name }}
        with:
          script: |
            const url = `https://acme.snowflakecomputing.com/console#/data/databases/ANALYTICS/schemas/${process.env.SCHEMA_NAME.toUpperCase()}`;
            const body = `Preview schema created: \`ANALYTICS.${process.env.SCHEMA_NAME}\`\n\n[Open in Snowsight](${url})`;
            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo:  context.repo.repo,
              body:  body,
            });

# scripts/create_preview_schema.py
import os
import snowflake.connector

def main() -> None:
    schema = os.environ["SCHEMA_NAME"].upper()

    conn = snowflake.connector.connect(
        account=os.environ["SNOWFLAKE_ACCOUNT"],
        user=os.environ["SNOWFLAKE_USER"],
        private_key=_load_pk(os.environ["SNOWFLAKE_PRIVATE_KEY"]),
        role="DBT_CI_ROLE",
        warehouse="DBT_CI_WH_XS",
        database="ANALYTICS",
    )
    with conn.cursor() as cur:
        cur.execute(f"CREATE OR REPLACE SCHEMA ANALYTICS.{schema} CLONE ANALYTICS.PROD")
        cur.execute(f"GRANT USAGE  ON SCHEMA ANALYTICS.{schema} TO ROLE ANALYST_ROLE")
        cur.execute(f"GRANT SELECT ON ALL TABLES IN SCHEMA ANALYTICS.{schema} TO ROLE ANALYST_ROLE")
        cur.execute(f"GRANT SELECT ON ALL VIEWS  IN SCHEMA ANALYTICS.{schema} TO ROLE ANALYST_ROLE")
    print(f"OK — created {schema}")


def _load_pk(pem: str):
    from cryptography.hazmat.primitives.serialization import load_pem_private_key
    return load_pem_private_key(pem.encode(), password=None).private_bytes(
        encoding=__import__("cryptography").hazmat.primitives.serialization.Encoding.DER,
        format=__import__("cryptography").hazmat.primitives.serialization.PrivateFormat.PKCS8,
        encryption_algorithm=__import__("cryptography").hazmat.primitives.serialization.NoEncryption(),
    )


if __name__ == "__main__":
    main()

Step-by-step explanation.

The workflow triggers on PR open + every synchronize (push). CREATE OR REPLACE is idempotent — re-running refreshes the clone against the current prod state.
Schema name construction sanitises the branch (lowercase, alphanumeric + underscore only, capped at 40 chars). Snowflake identifiers are case-folded to uppercase unless quoted; sticking to lowercase in code + uppercase at the SQL layer keeps everything predictable.
CREATE OR REPLACE SCHEMA ... CLONE ANALYTICS.PROD is the zero-copy clone. All tables, views, sequences, pipes in PROD become pointers in the new schema. No data copied; no storage billed until write.
The GRANT statements give the analyst role read access to the preview schema — Snowsight browses will work, dashboards can be repointed, and stakeholders can SELECT. No write access, so an analyst cannot accidentally mutate the preview.
The PR comment includes a Snowsight deep-link with the schema pre-selected, so reviewers click through to the preview in one hop.

Output.

Step	Runtime	Cost
Snowflake connect	~2 s	$0
CREATE OR REPLACE SCHEMA ... CLONE	~3 s	$0 (metadata op)
GRANT statements	~1 s	$0
Post PR comment	~1 s	$0
Total	~10 s	$0

Rule of thumb. Create the preview schema via zero-copy clone on PR open; make it idempotent (CREATE OR REPLACE); post the schema name and a Snowsight link to the PR. Analysts get the preview URL as fast as CI can post a comment.

Worked example — destroy-preview-schema on PR close

Detailed explanation. The cleanup workflow is the non-negotiable other half of the pattern. Without it, preview schemas accumulate indefinitely and eventually clutter the catalog. With it, catalog footprint is bounded to the count of open PRs.

Trigger. pull_request type: closed (fires on both merge and reject).
Action. DROP SCHEMA IF EXISTS ... CASCADE — idempotent; safe even if the schema was already dropped.

Question. Write the cleanup workflow.

Input.

Component	Value
Trigger	pull_request closed
Action	DROP SCHEMA IF EXISTS pr__ CASCADE
Cleanup guarantee	fires whether PR merged or rejected

Code.

# .github/workflows/preview-schema-drop.yml
name: preview-schema-drop
on:
  pull_request:
    types: [ closed ]

jobs:
  drop-preview:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    env:
      PR_NUMBER:   ${{ github.event.pull_request.number }}
      BRANCH_SLUG: ${{ github.head_ref }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install snowflake-connector-python
      - name: Compute schema name
        id: schema
        run: |
          slug=$(echo "$BRANCH_SLUG" | tr '/' '_' | tr -cd '[:alnum:]_' | tr '[:upper:]' '[:lower:]' | cut -c1-40)
          echo "name=pr_${PR_NUMBER}_${slug}" >> "$GITHUB_OUTPUT"
      - name: Drop preview schema
        env:
          SNOWFLAKE_ACCOUNT:     ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_USER:        dbt_ci
          SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
          SCHEMA_NAME:           ${{ steps.schema.outputs.name }}
        run: python scripts/drop_preview_schema.py

# scripts/drop_preview_schema.py
import os
import snowflake.connector

def main() -> None:
    schema = os.environ["SCHEMA_NAME"].upper()
    conn = snowflake.connector.connect(
        account=os.environ["SNOWFLAKE_ACCOUNT"],
        user=os.environ["SNOWFLAKE_USER"],
        private_key=_load_pk(os.environ["SNOWFLAKE_PRIVATE_KEY"]),
        role="DBT_CI_ROLE",
        warehouse="DBT_CI_WH_XS",
        database="ANALYTICS",
    )
    with conn.cursor() as cur:
        cur.execute(f"DROP SCHEMA IF EXISTS ANALYTICS.{schema} CASCADE")
    print(f"OK — dropped {schema} (or it did not exist)")


def _load_pk(pem: str):
    from cryptography.hazmat.primitives.serialization import (
        load_pem_private_key, Encoding, PrivateFormat, NoEncryption,
    )
    return load_pem_private_key(pem.encode(), password=None).private_bytes(
        encoding=Encoding.DER, format=PrivateFormat.PKCS8,
        encryption_algorithm=NoEncryption(),
    )


if __name__ == "__main__":
    main()

-- Nightly safety-net cron — drop pr_% schemas older than 14 days
DECLARE
  schemas_to_drop VARRAY(2000);
BEGIN
  SELECT ARRAY_AGG(SCHEMA_NAME) INTO :schemas_to_drop
  FROM   ANALYTICS.INFORMATION_SCHEMA.SCHEMATA
  WHERE  SCHEMA_NAME LIKE 'PR_%'
    AND  CREATED < DATEADD(day, -14, CURRENT_TIMESTAMP());

  FOR i IN 0 TO ARRAY_SIZE(:schemas_to_drop) - 1 DO
    EXECUTE IMMEDIATE 'DROP SCHEMA IF EXISTS ANALYTICS.' || GET(:schemas_to_drop, i) || ' CASCADE';
  END FOR;
END;

Step-by-step explanation.

The pull_request: closed trigger fires whether the PR is merged or rejected. GitHub does not distinguish between merged and abandoned closes in the trigger itself — both should trigger cleanup because both leave the preview schema orphaned.
DROP SCHEMA IF EXISTS ... CASCADE is idempotent: IF EXISTS prevents an error if the schema was already dropped (concurrent runs, manual cleanup, etc.). CASCADE drops all objects in the schema; for zero-copy clones there's no storage impact.
The Snowflake schema name is uppercased at the SQL layer because Snowflake case-folds unquoted identifiers. Consistency: lowercase in Python + workflow logic, uppercase at SQL.
The nightly safety-net cron catches schemas that escaped the pull_request trigger — e.g. PRs closed while GitHub Actions was down, PRs deleted rather than closed, PRs from forks that bypass the trigger. Bounded staleness = 14 days.
The DROP is essentially free — the source schema (PROD) is untouched; only the metadata pointer schema is removed. Runtime is under 5 seconds even for schemas with thousands of tables.

Output.

Event	Cleanup path
PR merged	pull_request-closed trigger → DROP within 30 s
PR rejected	pull_request-closed trigger → DROP within 30 s
PR abandoned (never closed)	nightly cron catches at day 14
Fork PR (limited trigger)	nightly cron catches at day 14
GitHub Actions outage	nightly cron catches at day 14

Rule of thumb. Ship the cleanup workflow the same day as the create workflow. Add the nightly cron the same week. A preview-environment system without cleanup accumulates schemas until someone screams; a system with both cleanup layers stays bounded forever.

Worked example — seeding a preview from a sampled production dataset

Detailed explanation. For sensitive workloads (PII, revenue data) where the preview should not contain full prod data, seed the preview from a sampled subset instead of a zero-copy clone. This trades preview fidelity for privacy.

The sampling. CREATE TABLE ... AS SELECT * FROM prod.orders SAMPLE (5) — Snowflake's 5% random sample.
The PII handling. Redact or hash sensitive columns during the sample.

Question. Write the sampling script that seeds a preview schema with 5% of prod data, redacting emails.

Input.

Component	Value
Source	ANALYTICS.PROD
Sample rate	5%
Redacted columns	orders.customer_email → sha256
Target	ANALYTICS.pr__

Code.

# scripts/seed_sampled_preview.py
import os
import snowflake.connector

TABLES_TO_SAMPLE = {
    "ORDERS":     ["customer_email"],  # columns to redact
    "CUSTOMERS":  ["email", "phone"],
    "SHIPMENTS":  [],
}


def main() -> None:
    schema = os.environ["SCHEMA_NAME"].upper()
    conn = snowflake.connector.connect(
        account=os.environ["SNOWFLAKE_ACCOUNT"],
        user=os.environ["SNOWFLAKE_USER"],
        private_key=_load_pk(os.environ["SNOWFLAKE_PRIVATE_KEY"]),
        role="DBT_CI_ROLE",
        warehouse="DBT_CI_WH_XS",
        database="ANALYTICS",
    )
    with conn.cursor() as cur:
        cur.execute(f"CREATE SCHEMA IF NOT EXISTS ANALYTICS.{schema}")
        for table, redact_cols in TABLES_TO_SAMPLE.items():
            col_expr = _column_expressions(cur, table, redact_cols)
            sql = (
                f"CREATE OR REPLACE TABLE ANALYTICS.{schema}.{table} AS "
                f"SELECT {col_expr} FROM ANALYTICS.PROD.{table} SAMPLE (5)"
            )
            cur.execute(sql)
            print(f"OK — seeded {schema}.{table}")


def _column_expressions(cur, table: str, redact_cols: list[str]) -> str:
    cur.execute(f"""
        SELECT COLUMN_NAME FROM ANALYTICS.INFORMATION_SCHEMA.COLUMNS
        WHERE  TABLE_SCHEMA = 'PROD' AND TABLE_NAME = '{table}'
        ORDER  BY ORDINAL_POSITION
    """)
    cols = [r[0] for r in cur.fetchall()]
    parts = []
    for c in cols:
        if c.lower() in {r.lower() for r in redact_cols}:
            parts.append(f"SHA2({c}, 256) AS {c}")
        else:
            parts.append(c)
    return ", ".join(parts)


def _load_pk(pem: str):
    from cryptography.hazmat.primitives.serialization import (
        load_pem_private_key, Encoding, PrivateFormat, NoEncryption,
    )
    return load_pem_private_key(pem.encode(), password=None).private_bytes(
        encoding=Encoding.DER, format=PrivateFormat.PKCS8,
        encryption_algorithm=NoEncryption(),
    )


if __name__ == "__main__":
    main()

Step-by-step explanation.

The sampling script iterates a table → redact-cols dict, so adding a new table is a one-line change. Column names are looked up dynamically from INFORMATION_SCHEMA.COLUMNS to avoid schema drift.
SAMPLE (5) is Snowflake's Bernoulli row sample: each row has a 5% independent chance of inclusion. For a 100M-row table this yields ~5M rows; runtime is roughly proportional to the sample size, not the source size.
Redacted columns are SHA-256 hashed rather than dropped, so downstream JOIN semantics survive. SHA2(email, 256) keeps the column at its original width and preserves uniqueness for de-dupe logic.
CREATE OR REPLACE TABLE (not zero-copy clone) is used because the redaction requires actual data materialisation. Storage cost applies — a 5% sample of a 1 TB table costs ~50 GB. Retention is the same 14-day nightly cron.
This pattern is the exception, not the rule — most previews use zero-copy clones. Reserve the sampled-preview path for PII-sensitive schemas, and document which tables are redacted.

Output.

Table	Rows in prod	Rows in preview	Redacted cols
ORDERS	100M	~5M	customer_email → sha256
CUSTOMERS	10M	~500K	email, phone → sha256
SHIPMENTS	50M	~2.5M	(none)

Rule of thumb. Default to zero-copy clone; use sampled + redacted only for PII-sensitive schemas. Document the redaction list in a versioned config. Analysts querying the preview must know which columns are hashes.

Senior interview question on preview environments

A senior interviewer might ask: "Your dbt project has 800 models and 30 open PRs at any time. Design the preview-environment system: schema naming, creation, cleanup, per-PR cost bound, and the plan for PII-sensitive schemas. Include the runbook when the cleanup workflow fails."

Solution Using create + drop workflows + nightly safety-net + optional sampled seed

# .github/workflows/preview-envs.yml — full preview lifecycle
name: preview-envs
on:
  pull_request:
    types: [ opened, synchronize, reopened, closed ]
  schedule:
    - cron: '0 6 * * *'   # nightly safety-net at 06:00 UTC

jobs:
  create:
    if: github.event.action != 'closed' && github.event_name == 'pull_request'
    uses: ./.github/workflows/preview-schema-create.yml

  drop:
    if: github.event.action == 'closed'
    uses: ./.github/workflows/preview-schema-drop.yml

  nightly-safety-net:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install snowflake-connector-python
      - name: Drop pr_% schemas older than 14 days
        env:
          SNOWFLAKE_ACCOUNT:     ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_USER:        dbt_ci
          SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
        run: python scripts/preview_safety_net.py

# scripts/preview_safety_net.py — bounds preview-schema staleness
import os
import snowflake.connector

def main() -> None:
    conn = snowflake.connector.connect(
        account=os.environ["SNOWFLAKE_ACCOUNT"],
        user=os.environ["SNOWFLAKE_USER"],
        private_key=_load_pk(os.environ["SNOWFLAKE_PRIVATE_KEY"]),
        role="DBT_CI_ROLE",
        warehouse="DBT_CI_WH_XS",
        database="ANALYTICS",
    )
    with conn.cursor() as cur:
        cur.execute("""
            SELECT SCHEMA_NAME
            FROM   ANALYTICS.INFORMATION_SCHEMA.SCHEMATA
            WHERE  SCHEMA_NAME LIKE 'PR_%'
              AND  CREATED < DATEADD(day, -14, CURRENT_TIMESTAMP())
        """)
        stale = [r[0] for r in cur.fetchall()]
        for s in stale:
            cur.execute(f"DROP SCHEMA IF EXISTS ANALYTICS.{s} CASCADE")
            print(f"safety-net dropped {s}")
    print(f"OK — {len(stale)} schemas cleaned")


def _load_pk(pem: str):
    from cryptography.hazmat.primitives.serialization import (
        load_pem_private_key, Encoding, PrivateFormat, NoEncryption,
    )
    return load_pem_private_key(pem.encode(), password=None).private_bytes(
        encoding=Encoding.DER, format=PrivateFormat.PKCS8,
        encryption_algorithm=NoEncryption(),
    )


if __name__ == "__main__":
    main()

Step-by-step trace.

Event	Workflow	Result
PR open / sync	preview-schema-create	CREATE OR REPLACE SCHEMA … CLONE PROD; post URL
PR merged	preview-schema-drop	DROP SCHEMA IF EXISTS … CASCADE
PR rejected	preview-schema-drop	same
PR abandoned	nightly cron at day 14	DROP SCHEMA IF EXISTS … CASCADE
GitHub Actions outage	nightly cron	catches later
Analyst review	Snowsight URL from PR comment	reads preview schema
PII schema	scripts/seed_sampled_preview.py	5% sample + SHA-256 redaction

After deployment, the catalog holds exactly one schema per open PR plus prod; the November catalog audit shows zero orphaned pr_% schemas older than 14 days. Median preview creation time is 10 seconds; median compute cost per PR is $0.05 (analyst review session on the XS warehouse). Analysts open the preview URL in one click from the PR description.

Output:

Metric	Value
Preview creation latency	~10 s
Preview compute cost per PR	~$0.05
Preview storage cost	$0 (zero-copy)
Orphaned schemas (14-day window)	0
Cleanup coverage	100% (trigger + cron)
PII redaction	opt-in per schema

Why this works — concept by concept:

Deterministic schema naming — pr_<PR>_<branch_slug> is generated from GitHub context. Any script that knows the PR number can construct the name. This makes create + drop trivially symmetric.
Zero-copy CLONE — Snowflake and Databricks UC implement clone as metadata. Creation is a 3-second op; storage is $0 until write. The preview is a full mirror without a copy cost.
Idempotent CREATE OR REPLACE + DROP IF EXISTS — every re-run is safe. Race conditions between create + push don't corrupt state.
pull_request:closed trigger — fires on both merge and reject. Reclaims the schema within 30 seconds of PR close.
Nightly safety-net cron — bounds staleness at 14 days regardless of failure mode. Catches abandoned PRs, GHA outages, fork PRs.
Cost — $0 storage (zero-copy), ~$0.05 compute per PR review session (XS warehouse, AUTO_SUSPEND=60), $0.01 workflow overhead. Net cost per PR under $0.10. Compared to the "no preview + wait for merge" baseline the analyst-hour savings alone pay for the system in one week.

SQL
Topic — sql
SQL warehouse-preview and clone problems

Practice →

Design
Topic — design
Design problems on ephemeral environments

Practice →

Cheat sheet — CI/CD recipes for dbt + Airflow + Spark

Which gate when. dbt: dbt build --select state:modified+ --defer --state prod-manifest/ --target ci. Airflow: DagBag(dag_folder='airflow/dags', include_examples=False).import_errors == {} first; then operator pytest; then optional dag.test() on LocalExecutor + Postgres service. Spark: local SparkSession.builder.master("local[2]") pytest first, then optional docker-compose bitnami/spark:3.5.1 mini-cluster, then spark-submit --num-executors 0 dry-run against a dev cluster. Preview: CREATE OR REPLACE SCHEMA ANALYTICS.pr_<PR>_<branch> CLONE ANALYTICS.PROD on PR open; DROP SCHEMA IF EXISTS ... CASCADE on PR close; nightly cron drops pr_% older than 14 days.
Slim CI command template. dbt build --select state:modified+ --defer --state prod-manifest/ --target ci --fail-fast + dbt clone --state prod-manifest/ --resource-type model --resource-type seed --resource-type snapshot. Prod side uploads manifest.json + run_results.json + generated_at.txt to s3://dbt-artifacts/prod/latest/ at the end of every successful run. CI downloads all three; the generated_at.txt freshness guard warns if manifest is >48 h old.
Path-based short-circuit. Use dorny/paths-filter@v3 at the top of the workflow with dbt: ['dbt/**'], airflow: ['airflow/**'], spark: ['spark/**']. Never run a system's gates on a PR that did not touch that system. The cheapest CI optimisation ever invented.
DAG parse gate snippet. python -c "from airflow.models.dagbag import DagBag; b=DagBag('airflow/dags', include_examples=False); assert not b.import_errors, b.import_errors" with AIRFLOW_HOME=/tmp/airflow, AIRFLOW__CORE__LOAD_EXAMPLES=False, AIRFLOW__CORE__UNIT_TEST_MODE=True. Runs in 15-30 s; catches every import-time break.
pytest SparkSession fixture template. @pytest.fixture(scope="session") → SparkSession.builder.master("local[2]").appName("ci").config("spark.sql.shuffle.partitions", "2").config("spark.ui.enabled", "false").config("spark.sql.session.timeZone", "UTC").getOrCreate(). Session-scoped so JVM starts once. Under 200 ms per test after warm.
Preview schema naming + cleanup. Format pr_<PR>_<slug> where slug = branch|tr '/' '_' | tr -cd '[:alnum:]_' | tr '[:upper:]' '[:lower:]' | cut -c1-40. Create with CREATE OR REPLACE SCHEMA ... CLONE prod; drop with DROP SCHEMA IF EXISTS ... CASCADE. Nightly cron: SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME LIKE 'PR_%' AND CREATED < DATEADD(day, -14, CURRENT_TIMESTAMP()).
Deferral vs clone decision matrix. --defer alone: reads unbuilt refs from prod schema; preview schema contains only rebuilt models. Use when the CI job is the only consumer. --defer + dbt clone: preview schema is a full mirror of prod overlaid with rebuilt models. Use when humans (analysts, PMs) query the preview. Cost: identical (both are zero-copy on Snowflake / Databricks UC).
Cost guardrails. Snowflake: WAREHOUSE_SIZE=XSMALL, AUTO_SUSPEND=60, AUTO_RESUME=TRUE — an XS is $1/hr; 60-s suspend keeps typical CI session under $0.10. Databricks: SQL warehouse auto_stop_mins: 5; interactive cluster autotermination_minutes: 10. BigQuery: on-demand; every query priced independently. Publish per-gate cost budget as a contract (e.g. dbt-slim-ci ≤ $0.30, airflow-parse ≤ $0.02).
Airflow OCI image versioning contract. Build args → OCI labels: GIT_SHA → org.opencontainers.image.revision, GIT_BRANCH → org.opencontainers.image.ref.name, CI_RUN_ID → ai.pipecode.ci.run.id, AIRFLOW_VERSION → ai.pipecode.airflow.version. Immutable tag sha-<short> for deploys; mutable pr-<PR> / main for humans. docker inspect reveals provenance in one command.
Spark packaging contract. PySpark: pip wheel → job-<version>-py3-none-any.whl + entry_points = {"console_scripts": [...]}. Scala: sbt assembly → job-<version>-assembly.jar. Config never baked into artifact; selected by env var at submit time. --version flag prints git SHA. --dry-run flag constructs SparkSession + exits 0.
State-artifact freshness guard. dbt CI downloads generated_at.txt alongside manifest.json; if age_hours > 48 emit a ::warning:: in GitHub Actions. Stale manifest = slim CI over-selects (marks unchanged as modified) or under-selects (misses real changes). Warning surfaces the problem before it silently distorts CI output.
Full-build fallback triggers. dbt slim CI does not detect all impact classes. Fall back to full build when the PR touches: macros/**, dbt_project.yml, packages.yml, requirements.txt, or profiles.yml. Detect via git diff --name-only origin/main | grep -qE '(macros/|dbt_project\.yml|packages\.yml)'.
PII-sensitive preview pattern. Don't zero-copy clone PII schemas; instead materialise a 5% SAMPLE with SHA-256 redaction on sensitive columns (SHA2(email, 256) AS email). Storage cost applies (~5% of source); worth it for GDPR / SOC2. Document redaction list in versioned config.
On-call runbook order. (1) CI red on prod-blocking PR → check gate that failed; (2) dbt slim-CI failed on state comparison → verify s3://dbt-artifacts/prod/latest/manifest.json exists and is <48 h old; (3) Airflow parse failed → open the import_errors traceback; (4) Spark dry-run failed → check dev-EMR cluster health; (5) preview schema missing → check create workflow logs, re-run manually; (6) preview schema orphaned → nightly cron logs at 06:00 UTC.

Frequently asked questions

What is slim CI in dbt?

slim ci dbt is the pattern of running dbt build --select state:modified+ --defer --state prod-manifest/ in CI instead of a full dbt build. The prod manifest.json from the last successful production run is downloaded as an artifact; state:modified+ picks only the models whose SQL, config, or upstream refs have changed since that manifest, plus their descendants; --defer routes any unbuilt parent refs to the prod schema rather than re-materialising them in the CI schema. The result is that a CI run rebuilds 3-15 models on a typical PR instead of 200-1500, cutting the p95 CI time from about 45 minutes to about 2 minutes and the compute cost from $50-$500 per run to about $0.30. Every senior analytics engineer names slim CI in the first minute of any dbt CI/CD interview because it is the load-bearing CI optimisation for dbt projects.

How do I set up state comparison for dbt CI?

State comparison requires an artifact contract between production and CI: the prod dbt job uploads target/manifest.json (plus run_results.json and a generated_at.txt freshness marker) to a known object-store location (typically s3://dbt-artifacts/prod/latest/) at the end of every successful run. The CI workflow downloads those files into a local prod-manifest/ directory before running dbt build; the --state prod-manifest/ flag tells dbt where to look. Under the hood, dbt fingerprints every model (SHA-256 of SQL text + config + upstream refs + sources + contract), compares the CI manifest against the prod manifest, and marks any model with a differing fingerprint as state:modified. Ship the freshness guard alongside — warn if the prod manifest is more than 48 hours old, because stale prod state distorts the diff (over-selects or under-selects). Bootstrap by running one manual full build against the target-prod schema and uploading its manifest; subsequent runs incrementally refresh the artifact.

Do I need a full Spark cluster in CI?

No, and running one is the single most common CI-cost mistake in Spark shops. The CI matrix for Spark is three gates in ascending cost: (a) local SparkSession.builder.master("local[2]") pytest fixture, which runs in the GHA runner with zero cluster cost and covers pure transformations, UDFs, and schema assertions — this catches roughly 80% of Spark bugs; (b) docker-compose bitnami/spark:3.5.1 mini-cluster for jobs that exercise real shuffle, broadcast joins, or partitioning behaviour — adds ~2 minutes to CI when it runs but stays gated on spark/jobs/** path filter; (c) a spark-submit --num-executors 0 --dry-run submit against a small dev EMR cluster to validate config, JAR resolution, and cluster reachability without ever provisioning executors. The real prod cluster is never touched by CI. If you find your CI job submitting to the prod Spark cluster, you have a cost bug — refactor to the three-gate pattern.

How do I test Airflow DAGs in CI?

Three layers of gates in ascending cost. First, the DAG parse gate: instantiate DagBag(dag_folder='airflow/dags', include_examples=False) in CI and assert dagbag.import_errors == {}. Runs in 15-30 seconds and catches every import-time break — the single highest-impact class of Airflow bug because import failures pause the whole DagBag, not just the broken DAG. Second, pytest against custom operators and hooks with mocked backends: instantiate the operator, provide a mock context, assert on return values and mock call args. Coverage bar: 95% line coverage on everything in airflow/plugins/. Third, an optional integration test using dag.test(execution_date=...) (Airflow 2.5+) with a LocalExecutor and a Postgres service — runs the entire DAG synchronously without a scheduler. Reserve the integration layer for high-value DAGs (ones touching shared warehouse tables, dynamic task mapping, etc.) so the CI matrix stays under 6 minutes. Wrap all three in a labelled OCI image built with apache/airflow:2.10.x as the deployable artifact.

How do preview environments work with Snowflake?

Every open pull request maps to a throwaway schema named pr_<PR>_<branch> in Snowflake, created on PR open via zero-copy clone and dropped on PR close. The create workflow runs CREATE OR REPLACE SCHEMA ANALYTICS.pr_4712_add_orders_v2 CLONE ANALYTICS.PROD — Snowflake implements clone as a metadata operation, so a 10 TB source schema clones in about 3 seconds and costs $0 in storage until a write happens. The slim-CI dbt build runs against the preview schema; dbt clone fills in any unmodified tables so the preview is a full mirror. A comment is posted to the PR with a Snowsight deep-link that pre-selects the schema, so reviewers open the preview in one click. The drop workflow triggers on pull_request type closed (both merge and reject) and runs DROP SCHEMA IF EXISTS pr_<PR>_<branch> CASCADE. A nightly cron at 06:00 UTC drops any pr_% schema older than 14 days as a safety net for abandoned PRs or GHA outages. Compute uses an XS warehouse (WAREHOUSE_SIZE=XSMALL, AUTO_SUSPEND=60), keeping typical per-PR compute cost under $0.10.

What is dbt deferral vs dbt clone?

dbt deferral (--defer --state prod-manifest/) is a read-side optimisation: when a CI-built model runs {{ ref('unmodified_upstream') }}, dbt sees the ref was not rebuilt in this run, consults the state manifest, and rewrites the ref to point at the prod schema instead of the CI schema. No table is materialised for the deferred parent in the preview schema; the read is redirected. dbt clone (dbt clone --state prod-manifest/ --resource-type model --resource-type seed --resource-type snapshot) is a write-side operation: for every unmodified model in the state manifest, dbt executes CREATE OR REPLACE <preview>.<model> CLONE prod.<model>, producing a zero-copy metadata pointer in the preview schema. Use --defer alone when the CI job is the only consumer (fast, no schema footprint). Use --defer + dbt clone when humans (analysts, PMs) query the preview and expect all the joined tables to be there. Cost is identical (both are zero-copy on Snowflake and Databricks UC); the difference is whether the preview schema is a full mirror or just a delta.

Practice on PipeCode

Drill the ETL practice library → for the pipeline-CI, slim-build, and state-comparison problems senior interviewers love.
Rehearse on the SQL practice library → for warehouse-side clone semantics, incremental modeling, and preview-schema query patterns.
Sharpen the systems axis with the design practice library → for CI/CD architecture, ephemeral environment design, and deployable-artifact provenance scenarios.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis CI decision matrix against real graded inputs.

Lock in pipeline-CI muscle memory

Docs explain slim CI. PipeCode drills explain the decision — when `state:modified+` is enough versus when to fall back to a full build, when the DAG parse gate saves the scheduler versus when a `dag.test()` integration is worth the six minutes, when a docker-compose Spark cluster is worth the setup cost versus when `local[2]` pytest is the honest answer, when a zero-copy preview schema replaces "wait for merge" as the analyst review UX. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the CI/CD trade-offs senior analytics engineers and data platform engineers actually face.

Practice ETL problems →
Practice design problems →

Data SRE Playbook: SLO-Driven Pipelines, Error Budgets & On-Call Rotations for DE Teams

Gowtham Potureddi — Fri, 31 Jul 2026 14:38:09 +0000

A data sre playbook is the operating manual senior data teams reach for the day they stop shipping "the DAG turned green" as a reliability claim and start owning user-visible pipeline reliability the same way SRE teams own web-service reliability — with measurable SLIs, explicit SLOs, monthly error budgets, burn-rate paging, follow-the-sun on-call rotations, linked runbooks, and blameless postmortems that compound into a reliability roadmap. The reason every senior data engineer needs one in 2026 is that the number of downstream consumers of an operational data pipeline — dashboards, ML models, feature stores, reverse-ETL sync targets, customer-facing APIs — has grown to the point where a two-hour freshness gap on the orders table is a customer-visible outage, and "we'll fix it on the next DAG run" is not an incident response.

This guide is the senior-DE walkthrough you wished existed the first time an executive asked "what's our reliability commitment on the warehouse?", or an interviewer probed "walk me through the SLIs, SLOs, and error budgets you'd set for a streaming pipeline," or a director cornered you with "why did that data outage take three hours to notice and eleven hours to resolve?" It covers the four failure axes every data platform must instrument — data freshness slo (max age of the latest record), completeness (fraction of source rows landed), volume (row-count within an expected band), and quality (fraction of rows passing schema and constraint checks) — the error budget data engineering math that translates a 99.9% freshness target into a 43-minute monthly ceiling, the multi-window multi-burn-rate pipeline sli alerts modelled on the Google SRE workbook, the data on-call rotation shape and handoff ritual, the 5-field data runbook template every alert links to, and the blameless postmortem walk from detect through mitigate, resolve, and review. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the design practice library →, and sharpen the reliability axis with the data-validation practice library →.

On this page

Why data teams need SRE practices in 2026
SLIs and SLOs for data pipelines
Error budgets and burn-rate alerts
On-call rotations and incident runbooks
Post-incident review and reliability roadmap
Cheat sheet — Data SRE recipes
Frequently asked questions
Practice on PipeCode

1. Why data teams need SRE practices in 2026 — the four failure axes

Data pipelines fail differently from web services — the same SRE principles apply, the SLIs do not

The one-sentence invariant: a data platform is reliable when the datasets its consumers depend on are fresh, complete, correctly-sized, and quality-checked within an explicitly agreed-upon target — and pipeline reliability engineering is the practice of translating those four consumer-visible axes into measurable SLIs, monthly SLOs, error budgets, burn-rate alerts, on-call rotations, and blameless postmortems, exactly like SRE teams do for web services but with pipeline-shaped inputs instead of HTTP-shaped ones. Every senior data-engineering interview in 2026 probes this territory because the number of teams claiming "SRE for data" without actually shipping a single measurable SLI has exploded, and the interviewers you'll face have been burned by fluent-sounding candidates whose reliability practice starts and stops at "we monitor DAG status."

The four failure axes interviewers actually probe.

Freshness. The maximum age of the latest record in a target table relative to when it should have arrived. If a hourly orders pipeline runs at :05 past the hour, the freshness target might be "no consumer query should ever see MAX(ingested_at) < now() - 65 min." Freshness is the axis that hurts business dashboards first; it's the most common SLO to write.
Completeness. The fraction of expected source rows that have landed in the target within the SLO window. If the source produced 10,000 orders in the hour and only 9,850 arrived, completeness is 98.5% for that window. Completeness catches silent drops — filter bugs, schema mismatches, network truncation.
Volume. Whether the row count of the target table sits within an expected band (usually min_expected < count < max_expected from a rolling 7- or 28-day baseline). Volume catches "we ingested but the count is wrong" — a duplicated upstream feed doubling counts, a broken filter halving them.
Quality. The fraction of rows that pass schema, constraint, and business-rule checks (nulls in required columns, invalid enums, out-of-range values, orphan foreign keys). Quality catches "we ingested the right count but the data is garbage."

Why "did the DAG run green" is not an SLI.

DAG green ≠ data correct. An Airflow task can succeed while writing an empty file, writing to the wrong partition, or writing stale data because an upstream watermark stalled. The status of the job is a proxy for reliability at best; the status of the dataset is what consumers experience.
DAG green ignores latency-of-detection. A pipeline that fails at 09:00 and is caught at 15:00 has still burned six hours of business value. Job-status monitoring wakes you up when a task raises; SLI monitoring wakes you up when the dataset stops meeting its contract, which is often earlier and sometimes later.
DAG green ignores upstream reality. If the source system stopped producing rows at 08:00, every downstream DAG will still turn green (they read an empty delta and write an empty file). SLI-based monitoring on volume + freshness catches this class of "silent zero" failure that job monitoring cannot.
DAG green ignores partial success. A Spark job with spark.sql.files.ignoreCorruptFiles = true succeeds while silently dropping bad Parquet files. A dbt run with --fail-fast off finishes with 2 of 200 tests failing but a green run icon. Only per-dataset SLIs make partial-success failures visible.

The SRE contract: user-visible reliability, not job status.

User-visible metrics only. An SLI is measured on the output artifact (the target table, the served view, the exported file) — not on the process that produced it. This constraint forces the metric to align with what a downstream consumer cares about.
Target expressed as an SLO. "99.9% of hourly windows have freshness < 65 minutes." "99.5% of daily runs have completeness > 99% of expected count." SLOs are always a percentage of measurement windows meeting a threshold, never "we always meet it."
Error budget = 1 − SLO. A 99.9% freshness SLO means 0.1% of hourly windows can miss — that's 43 minutes per 30-day month. The budget is the operational contract with the rest of the organisation: within it, releases proceed; outside it, releases halt.
Alerting off the SLI, not the pipeline. Burn-rate alerts fire when the SLI degrades faster than the budget will tolerate. The alert wakes a human because the user-visible contract is at risk, not because a job status changed.

What interviewers listen for.

Do you name all four axes (freshness, completeness, volume, quality) without prompting? — senior signal.
Do you say "DAG status is not an SLI" when asked what to monitor? — required answer.
Do you cite the error-budget number (99.9% ≈ 43 min/month) without pulling out a calculator? — senior signal.
Do you describe a burn-rate alert as "multi-window multi-rate" and cite the Google SRE workbook? — senior signal.
Do you frame postmortems as "blameless, focused on contributing factors" rather than root-cause five-whys? — senior signal.

Worked example — the four-axis reliability table for an orders pipeline

Detailed explanation. The single most useful artifact for a data SRE interview is a memorised 4×4 axis table. Every senior reliability discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical hourly orders pipeline that lands into a Snowflake warehouse.

Source. production.public.orders (Postgres OLTP).
Target. analytics.orders (Snowflake, updated hourly by an Airflow DAG at :05 past the hour).
Consumers. Executive dashboard (Looker), ML fraud model (feature store), reverse-ETL sync to Salesforce.
User contract. All three consumers depend on data being no more than ~1 hour behind, complete against the source, correctly-sized, and quality-checked.

Question. Build the four-axis SLI/SLO table for the analytics.orders pipeline and pick a numeric target for each.

Input.

Axis	Measurement	SLO target	Budget / month
Freshness	max(`ingested_at`) age at query time	99.9% of minutes < 65 min	~43 min
Completeness	target rowcount / source rowcount per hour	99.5% of hours ≥ 99%	~3.6 hours
Volume	rowcount vs 7-day median band ±30%	99.5% of hours within band	~3.6 hours
Quality	rows passing schema + constraint checks	99.9% of daily runs ≥ 99%	~1 day/quarter

Code.

-- Freshness SLI — computed on the target table at query time
SELECT
    CASE
        WHEN MAX(ingested_at) >= now() - INTERVAL '65 minutes' THEN 1
        ELSE 0
    END AS meets_freshness_slo,
    now() - MAX(ingested_at) AS current_age
FROM analytics.orders;

-- Completeness SLI — computed hourly by comparing target to source
WITH source_hour AS (
    SELECT date_trunc('hour', event_ts) AS h, COUNT(*) AS n_src
    FROM   production.orders
    WHERE  event_ts >= now() - INTERVAL '24 hours'
    GROUP  BY 1
),
target_hour AS (
    SELECT date_trunc('hour', event_ts) AS h, COUNT(*) AS n_tgt
    FROM   analytics.orders
    WHERE  event_ts >= now() - INTERVAL '24 hours'
    GROUP  BY 1
)
SELECT s.h,
       s.n_src,
       COALESCE(t.n_tgt, 0)                              AS n_tgt,
       COALESCE(t.n_tgt, 0)::float / NULLIF(s.n_src, 0)  AS completeness_ratio,
       CASE WHEN COALESCE(t.n_tgt, 0)::float / NULLIF(s.n_src, 0) >= 0.99
            THEN 1 ELSE 0 END                            AS meets_completeness_slo
FROM   source_hour s
LEFT JOIN target_hour t USING (h)
ORDER  BY s.h DESC;

-- Volume SLI — computed hourly against a 7-day rolling median band
WITH baseline AS (
    SELECT date_trunc('hour', event_ts)::time AS hour_of_day,
           percentile_cont(0.5) WITHIN GROUP (ORDER BY hourly_count) AS median_ct
    FROM (
        SELECT date_trunc('hour', event_ts) AS h,
               date_trunc('hour', event_ts)::time AS hour_of_day,
               COUNT(*) AS hourly_count
        FROM   analytics.orders
        WHERE  event_ts >= now() - INTERVAL '7 days'
          AND  event_ts <  now() - INTERVAL '1 hour'
        GROUP  BY 1, 2
    ) t
    GROUP BY 1
)
SELECT t.h,
       t.hourly_count,
       b.median_ct,
       CASE WHEN t.hourly_count BETWEEN b.median_ct * 0.7 AND b.median_ct * 1.3
            THEN 1 ELSE 0 END AS meets_volume_slo
FROM   (SELECT date_trunc('hour', event_ts) AS h,
               date_trunc('hour', event_ts)::time AS hour_of_day,
               COUNT(*) AS hourly_count
        FROM   analytics.orders
        WHERE  event_ts >= now() - INTERVAL '24 hours'
        GROUP  BY 1, 2) t
JOIN   baseline b USING (hour_of_day)
ORDER  BY t.h DESC;

Step-by-step explanation.

The freshness SLI is a single per-query check — MAX(ingested_at) on the target table. If the answer is younger than the SLO threshold, the minute counts as "meeting"; otherwise it burns budget. Computing this in the warehouse rather than in Airflow means it reflects the dataset state, not the pipeline state.
The completeness SLI joins hourly source counts against hourly target counts. Ratio < 0.99 for the hour means the hour "misses" its SLO. The 5-minute look-back safety window (not shown; add WHERE event_ts < now() - INTERVAL '5 min') prevents in-flight rows from being counted as missing.
The volume SLI compares each hour's target count to the median of the same hour-of-day across the last 7 days. The ±30% band absorbs normal weekly variance while catching the "3x volume" and "0.1x volume" outliers that indicate upstream duplication or a broken filter.
The quality SLI (not shown in code — computed via Great Expectations or dbt tests) is the fraction of rows passing a battery of expectations: orders.order_id NOT NULL, orders.status IN ('pending','shipped','delivered','cancelled'), orders.total_cents > 0, orders.customer_id REFERENCES customers.id. Aggregate to "% of rows passing all checks" for the day.
All four SLIs feed into a single reliability dashboard and a single error-budget accounting table. When any one SLI's budget approaches exhaustion, the on-call receives a burn-rate page. When any one SLI's budget is exhausted, feature releases against the pipeline are halted until budget replenishes at the start of the next month.

Output.

Consumer	Axis they care most about	SLO target	Why
Executive dashboard	Freshness	99.9% < 65 min	dashboards are read at any moment; stale = wrong decisions
ML fraud model	Completeness	99.5% ≥ 99%	model retrained hourly; missing rows = biased scoring
Reverse-ETL to Salesforce	Volume	99.5% within band	wrong volume = wrong sales pipeline forecast
All	Quality	99.9% daily ≥ 99%	broken schema = downstream crashes

Rule of thumb. Never pick an SLI based on "what the pipeline emits." Pick it based on (freshness × completeness × volume × quality) — the four axes a downstream consumer actually experiences. Write the table on a whiteboard first; the SLI selection falls out of the consumer contract.

Worked example — what interviewers actually probe

Detailed explanation. The senior data-SRE interview has a predictable structure: the interviewer opens with an ambiguous question ("how do you monitor a data pipeline?"), then progressively narrows to test whether you know the axes. The candidates who name the SLIs in sentence one score highest; the candidates who describe "Airflow alerts on failure" score lowest. Walk through the interview grading rubric.

Ambiguous opener. "How would you monitor an hourly ETL pipeline?" — invites you to name SLIs.
Follow-up 1. "What's an SLI vs an SLO?" — probes definitions.
Follow-up 2. "How much downtime does 99.9% freshness allow?" — probes error-budget math.
Follow-up 3. "When do you page vs ticket?" — probes burn-rate alerting.
Follow-up 4. "Walk me through your last data incident postmortem." — probes maturity.

Question. Draft a 5-minute senior data-SRE answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
SLI named	"we alert on Airflow failures"	"freshness, completeness, volume, quality — each with an SLO"
SLO stated	"we aim for 99% uptime"	"99.9% freshness < 65 min = 43 min/month error budget"
Alert semantics	"we page on task failure"	"multi-window multi-burn-rate on the freshness SLI"
On-call shape	"the person who wrote the DAG gets paged"	"primary + secondary weekly rotation with runbook links"
Postmortem	"we write a Slack thread"	"blameless review, contributing factors, tracked action items"

Code.

Senior data-SRE answer template (5 minutes)
============================================

Minute 1 — name the SLIs up front
  "I'd instrument four SLIs per pipeline: freshness (max age of latest row),
   completeness (target/source rowcount), volume (rowcount vs baseline band),
   and quality (rows passing schema + constraint checks). Each gets an SLO
   like 99.9% of hourly windows meeting the threshold."

Minute 2 — error-budget math
  "99.9% freshness means 0.1% of the month can miss, which is 43 minutes.
   That's the operational contract with the rest of the org: within the
   budget, releases proceed; outside it, releases halt until the budget
   resets at month start."

Minute 3 — burn-rate alerts
  "Instead of alerting on any missed minute, I use multi-window
   multi-burn-rate: a fast-burn window (1 h) at 14x normal spend pages
   immediately, a slow-burn window (6 h) at 6x tickets the team. This
   catches both cliffs and creeping degradation without false pages."

Minute 4 — on-call + runbook
  "Weekly primary + secondary rotation with follow-the-sun handoff for
   global teams. Every alert links to a 5-field runbook: symptom,
   detection, mitigation, escalation, owner. No runbook link = alert
   isn't allowed to fire."

Minute 5 — postmortem + roadmap
  "Blameless postmortem within 5 business days: timeline, contributing
   factors (not a single root cause), action items with owners and due
   dates. Action items feed the reliability roadmap; SLO trends drive
   the SLI backlog next quarter."

Step-by-step explanation.

Minute 1 is the crucial framing. Naming the four SLIs immediately — freshness, completeness, volume, quality — signals you've operated a data platform under real reliability targets, not just written DAGs. Weak candidates start with tools ("we use Airflow and Datadog") before naming a metric.
Minute 2 does the error-budget math out loud. Interviewers listen for whether you can convert an SLO percentage into an operational number without hesitation. "99.9% is 43 minutes per month" is the answer you should have burned into muscle memory.
Minute 3 introduces multi-window multi-burn-rate. This is the SRE-workbook-canonical alerting pattern; naming it, and citing the fast-vs-slow-burn distinction, is a senior signal that separates you from candidates who alert on task-failure.
Minute 4 addresses on-call operationally. The 5-field runbook (symptom, detection, mitigation, escalation, owner) is the concrete template; refusing to allow alerts without runbook links is the enforcement mechanism. Both are things you'd expect a mature platform to have.
Minute 5 covers postmortems and the reliability roadmap. "Blameless, contributing factors, tracked action items, feed the roadmap" is the four-part cycle that turns incidents into reliability improvements over quarters. This is the axis mid-level candidates skip entirely.

Output.

Grading criterion	Weak score	Senior score
Names SLIs in minute 1	rare	mandatory
States error-budget number	rare	required
Names burn-rate alerting	rare	senior signal
Names runbook fields	rare	senior signal
Names postmortem cycle	rare	senior signal

Rule of thumb. The senior data-SRE answer is a 5-minute monologue that covers SLIs, error budgets, burn-rate alerts, on-call rotation, and postmortems without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "SRE maturity" decision tree for a data team

Detailed explanation. Given a data team, the senior architect runs a 4-question maturity check in their head. Codifying the check makes the interview answer reproducible: any interviewer can hand you a team description and you can walk the tree out loud. Walk through the tree with three canonical scenarios: a startup with one DE and no SLIs, a mid-market team with green DAGs only, and a mature platform team with per-dataset SLOs.

Q1. Do the pipelines have measurable SLIs on the four axes (freshness, completeness, volume, quality)? → no = start there.
Q2. Are the SLOs written down and agreed with downstream consumers? → no = do the SLO-negotiation exercise.
Q3. Are burn-rate alerts wired to a rotation with runbook links? → no = the SLIs are decorative.
Q4. Are postmortems blameless and tracked as action items on a reliability roadmap? → no = incidents don't compound into improvements.

Question. Walk the maturity tree for the three scenarios and record the recommended next step for each.

Input.

Scenario	Q1 SLIs?	Q2 SLOs?	Q3 alerts + runbook?	Q4 postmortems?
Startup, 1 DE, no SLIs	no	no	no	no
Mid-market, green DAGs only	partial (freshness)	no	partial	no
Mature platform	yes	yes	yes	yes

Code.

# Data-SRE maturity classifier (illustrative)
def data_sre_maturity(has_slis: bool,
                       has_slos: bool,
                       has_burn_rate_alerts: bool,
                       has_blameless_postmortems: bool) -> str:
    """Return the recommended next step on the SRE maturity ladder."""
    if not has_slis:
        return "Level 0 — instrument SLIs on all four axes first"
    if not has_slos:
        return "Level 1 — negotiate SLO targets with downstream consumers"
    if not has_burn_rate_alerts:
        return "Level 2 — wire multi-window burn-rate alerts + runbook links"
    if not has_blameless_postmortems:
        return "Level 3 — adopt blameless postmortems and a reliability roadmap"
    return "Level 4 — mature; iterate on SLI selection and chaos drills"


# Walk the three scenarios
print(data_sre_maturity(False, False, False, False))
# → 'Level 0 — instrument SLIs on all four axes first'

print(data_sre_maturity(True, False, False, False))
# → 'Level 1 — negotiate SLO targets with downstream consumers'

print(data_sre_maturity(True, True, True, True))
# → 'Level 4 — mature; iterate on SLI selection and chaos drills'

Step-by-step explanation.

Scenario 1 — the one-DE startup has no SLIs, no SLOs, no alerts, no postmortems. The next step is not "adopt SRE" but "instrument the SLIs" — you cannot manage what you don't measure. Start by adding freshness on the top three most-consumed tables; expand from there.
Scenario 2 — the mid-market team has partial SLIs (freshness only, computed as a metric but never with an agreed target). The next step is negotiating SLO numbers with consumers: sit with the dashboard owner and agree "99.9% < 65 min," write it in a doc, and only then wire alerts.
Scenario 3 — the mature platform team is at Level 4 and iterating. The next step is not "climb the ladder" but "expand SLI coverage" (from 20 tables to 200 tables), "reduce false-positive alert rate," and "run chaos drills quarterly to verify the runbooks."
The maturity ladder is strict — you cannot jump levels. A team that adds burn-rate alerts without agreed SLOs will burn out on false pages. A team that writes blameless postmortems without burn-rate alerts will find nothing to write about because incidents are silently ignored.
If none of Q1-Q4 pass, the correct answer to "how do we adopt data SRE?" is "start with SLIs on your three most-consumed tables and iterate." Refuse to endorse tools ("buy Monte Carlo," "adopt Datadog") until Q1 is passing.

Output.

Scenario	Level	Next step
Startup, no SLIs	0	Instrument freshness on top 3 tables
Mid-market, green DAGs	1	Negotiate SLO targets with consumers
Mature platform	4	Expand SLI coverage; chaos drills

Rule of thumb. The four-level maturity ladder is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any team description and get a level + next step in under 60 seconds.

Senior interview question on data SRE adoption

A senior interviewer often opens with: "You join a 40-engineer data platform team that has 200 Airflow DAGs, no SLIs, and monitors reliability by 'did the DAG turn green.' Design the 90-day plan to introduce SRE practices without breaking the team's ability to ship. Cover the SLI you'd start with, the SLO negotiation, the alerting rollout, the on-call bootstrapping, and the metric you'd use to measure your own success."

Solution Using a 90-day SLI-first rollout with staged SLO negotiation and phased alert cutover

# 90-day data-SRE rollout plan (illustrative)
rollout:
  weeks_0_2:
    goal: instrument SLIs on top 10 tables
    steps:
      - identify top 10 tables by downstream query count (via warehouse audit logs)
      - implement freshness SLI (SELECT MAX(ingested_at) FROM t)
      - implement completeness SLI (source rowcount vs target rowcount per hour)
      - export both as Prometheus gauges (label: table_name)
      - build a "Data reliability" Grafana dashboard (no alerts yet)

  weeks_2_6:
    goal: negotiate SLO targets with downstream consumers
    steps:
      - schedule 30-min meetings with owner of each top-10 dashboard/service
      - propose default SLOs (99.9% freshness < 2x pipeline cadence)
      - agree per-table SLO in writing (Confluence doc)
      - publish SLO catalogue link from every table in the data catalogue

  weeks_6_10:
    goal: wire burn-rate alerts + runbooks
    steps:
      - draft 5-field runbook per SLO (symptom, detection, mitigation, escalation, owner)
      - configure multi-window multi-burn-rate alert rules in Prometheus/Alertmanager
      - route to Slack (warning) and PagerDuty (critical) with runbook link
      - run a game-day simulation before enabling paging

  weeks_10_13:
    goal: on-call rotation + first postmortems
    steps:
      - set up weekly primary + secondary rotation in PagerDuty
      - write handoff-doc template (open incidents, budget status, changes)
      - schedule blameless postmortem for every SLO breach within 5 business days
      - track action items in Jira with "reliability" label

success_metric:
  primary: fraction of top-10 tables with a live SLI + agreed SLO + alerting = 10/10 by day 90
  secondary: mean time to detect (MTTD) reduced from ~4 h to < 15 min on top-10 tables
  guardrail: false-positive alert rate < 20% of total pages

# Freshness SLI exporter — Python + prometheus_client
# Run every 60 s as a sidecar or scheduled task
import psycopg2
import time
from prometheus_client import start_http_server, Gauge

g_age_seconds = Gauge('data_freshness_age_seconds',
                      'age in seconds of newest row',
                      ['table_name'])

TABLES = [
    'analytics.orders',
    'analytics.customers',
    'analytics.shipments',
    'analytics.payments',
    'analytics.sessions',
    'analytics.pageviews',
    'analytics.events',
    'analytics.line_items',
    'analytics.refunds',
    'analytics.reviews',
]

def scrape(conn):
    with conn.cursor() as cur:
        for t in TABLES:
            cur.execute(f"SELECT extract(epoch FROM (now() - MAX(ingested_at))) FROM {t}")
            (age,) = cur.fetchone()
            g_age_seconds.labels(t).set(age or 0)

if __name__ == '__main__':
    start_http_server(9200)
    conn = psycopg2.connect('host=warehouse port=5432 dbname=analytics user=slo_reader')
    conn.autocommit = True
    while True:
        scrape(conn)
        time.sleep(60)

Step-by-step trace.

Week	Milestone	Deliverable
0–2	SLIs live on top 10 tables	freshness + completeness gauges + dashboard
2–6	SLO targets agreed	10 SLO docs signed by consumer + owner
6–10	Burn-rate alerts wired	Prometheus rules + runbook links
10–13	On-call rotation + first postmortems	rotation live; first postmortem doc
Day 90	Success metrics reported	MTTD < 15 min; false-page < 20%

After the rollout, the team has 10 tables under measurable reliability contracts, a weekly on-call rotation with runbook-linked alerts, and a blameless postmortem cadence. The next 90 days extend SLI coverage to the top 50 tables and start iterating on false-positive alert reduction.

Output:

Metric	Baseline	Day 90
Tables with live SLI	0	10
Tables with agreed SLO	0	10
Mean time to detect	~4 h	< 15 min
Blameless postmortems / month	0	1–3
False-positive alert rate	n/a	< 20%

Why this works — concept by concept:

SLI-first rollout — measurement before alerting. Instrumenting the metric first, running a dashboard for a few weeks, then adding alerts on the same metric is the SRE-canonical sequence. It gives you baseline data to set realistic SLO targets and avoids the "alerts firing before we knew normal" trap.
SLO negotiation — the SLO is a contract, not a target unilaterally set by the platform team. Sitting with the consumer of each dataset and agreeing the number in writing turns "we aim for 99.9%" into "the fraud team accepted 99.5% freshness with 65-minute recovery." Writing it down is what makes it enforceable.
Multi-window multi-burn-rate alerts — the standard SRE-workbook pattern for turning "we burnt some budget in the last window" into a paging decision. Fast-burn wakes you at cliff-fall; slow-burn wakes you at creep. Both fire on the same SLI.
5-field runbook link per alert — no runbook, no alert. Enforcing this at the alert-config level is the strongest lever for reducing "what do I do now?" MTTR. Every alert that pages a human must link to a document that tells them what to do.
Cost — one platform engineer week 0-2 to build the exporter, half an engineer weeks 2-6 for SLO negotiation meetings, one engineer weeks 6-10 to wire alerts and runbooks, half an engineer weeks 10-13 to bootstrap on-call. Net ~2.5 engineer-months for a 90-day rollout that puts 10 tables under reliability contracts. Compared to a $200k/year "data observability" tool that doesn't do SLO negotiation, this is 3–4x more valuable in interviewer eyes.

Design
Topic — design
Design problems on data platform reliability

Practice →

ETL Topic — etl ETL problems on pipeline observability

Practice →

2. SLIs and SLOs for data pipelines — freshness, completeness, volume, quality

The four SLIs every data platform must instrument — one per axis of user-visible reliability

The mental model in one line: an SLI for a data pipeline is a numeric measurement of a single user-visible reliability axis — freshness, completeness, volume, or quality — computed on the output artifact (the target table, the served view, the exported file) rather than on the process that produced it, and an SLO is the target percentage of measurement windows in which the SLI meets its threshold. Every senior data engineer must know how to derive an SLI from a consumer contract, express it as a rolling percentage, and translate it into a monthly error budget without hesitation.

The four axes as concrete SLIs.

Freshness SLI. Measurement: now() - MAX(ingested_at) on the target table, or MAX(ingested_at) - MAX(source.event_ts) for end-to-end lag. SLO: e.g. 99.9% of one-minute observation windows have age < 65 min for an hourly pipeline. Freshness is the axis every dashboard user notices first.
Completeness SLI. Measurement: target rowcount / source rowcount, computed per hour or per day (or per micro-batch for streaming). SLO: e.g. 99.5% of hours have completeness_ratio ≥ 0.99. Completeness catches silent drops from schema mismatches, filter bugs, or truncated network transfers.
Volume SLI. Measurement: target rowcount vs a rolling 7-day median band. SLO: e.g. 99.5% of hours have count ∈ [0.7 × median, 1.3 × median]. Volume catches duplication (2× median) and silent zero (0.1× median) failures that completeness alone misses when both source and target are wrong.
Quality SLI. Measurement: fraction of rows passing a battery of schema and business-rule checks (nulls in required columns, invalid enums, out-of-range values, orphan foreign keys). SLO: e.g. 99.9% of daily runs have pass_rate ≥ 0.99. Quality catches "we ingested the right count but the data is garbage."

Why each axis is separate.

Freshness alone is insufficient. A pipeline that runs on time but writes an empty file has freshness = 0 (age of latest row = 0 seconds… because there is no latest row). Volume + completeness catch this.
Completeness alone is insufficient. A pipeline that mirrors the source 1:1 but the source itself is broken (upstream feed cut in half) will pass completeness. Volume catches this.
Volume alone is insufficient. A pipeline whose count matches the baseline exactly but whose rows are semantically wrong (all total_cents = 0) will pass volume. Quality catches this.
Quality alone is insufficient. A pipeline that passes every quality check but arrives 4 hours late fails freshness. Freshness catches this.

The measurement window — the second axis of every SLI.

Observation cadence. How often the SLI is measured. Freshness is typically measured per minute (cheap query). Completeness is per hour. Volume per hour. Quality per daily run.
SLO window. The period over which the percentage is computed. Standard: rolling 28 days ("SLO calculated over the trailing 28-day window").
Error budget window. Same as SLO window. 99.9% over 28 days = 40.32 minutes of budget.
Alert window. Shorter periods used for burn-rate detection: 1 hour (fast burn), 6 hours (slow burn).

How to derive an SLI target from a consumer contract.

Ask the consumer. "How stale is too stale?" The dashboard owner says "I check it every 5 minutes; if it's more than an hour behind, my meeting is derailed." That's your freshness target: 65 minutes (add a small buffer above the natural 60-minute cadence).
Sanity-check against current performance. Compute the SLI over the last 28 days. If current performance is 99.7%, target 99.9% requires a 3x improvement in bad-minute count. If current is 99%, target 99.9% requires a 10x improvement — that's a project, not a target.
Trade cost against reliability. Higher SLOs cost more (extra headroom in compute, faster recovery, more on-call attention). Explicit trade-off: "we could hit 99.99% freshness by adding a hot-standby DAG, at a cost of 2x compute." The business decides.
Publish and version the SLO. Confluence doc + Git-tracked SLO catalogue. Every SLO has an owner (platform team) and a consumer (dashboard or service owner). Reviewed quarterly.

Common interview probes on SLIs and SLOs.

"What's the difference between an SLI and an SLO?" — required answer: SLI is the measurement, SLO is the target.
"How do you pick a freshness target?" — required answer: derive from consumer contract; sanity-check against current performance.
"What's an SLA?" — required answer: SLA is the external customer-facing commitment (with penalties); SLO is the internal target (typically stricter).
"How do you handle multiple consumers with different needs?" — required answer: pick the strictest requirement; publish that as the SLO; slower consumers benefit for free.

Worked example — freshness SLI over a Snowflake table

Detailed explanation. The canonical freshness SLI setup: a per-minute query that compares MAX(ingested_at) against the SLO threshold, exported as a Prometheus gauge and rolled up into a 28-day SLO percentage. Walk through the pieces.

Metric definition. data_freshness_meets_slo{table="analytics.orders"} = 1 if now() - MAX(ingested_at) < 65 min, else 0.
Cadence. Every 60 seconds.
SLO percentage. sum_over_time(data_freshness_meets_slo[28d]) / count_over_time(data_freshness_meets_slo[28d]).
Target. 99.9% (= 40.32 min budget over 28 days).

Question. Implement the freshness SLI query, the Prometheus exporter, and the SLO rollup rule.

Input.

Component	Value
Target table	analytics.orders
Threshold	65 minutes
Cadence	60 s
SLO window	28 days
SLO target	99.9%

Code.

-- Freshness check query — runs every 60 s from an exporter
SELECT
    'analytics.orders'::text AS table_name,
    extract(epoch FROM (now() - MAX(ingested_at))) AS age_seconds,
    CASE WHEN now() - MAX(ingested_at) < INTERVAL '65 minutes'
         THEN 1 ELSE 0 END AS meets_slo
FROM analytics.orders;

# Freshness exporter — psycopg2 + prometheus_client
import psycopg2
import time
from prometheus_client import start_http_server, Gauge

g_meets = Gauge('data_freshness_meets_slo', '1 if within threshold', ['table_name'])
g_age   = Gauge('data_freshness_age_seconds', 'age of newest row',   ['table_name'])

CFG = [
    # (table_name, threshold_seconds)
    ('analytics.orders',    65 * 60),
    ('analytics.customers', 24 * 3600),
    ('analytics.events',    15 * 60),
]

def scrape(conn):
    for tbl, threshold in CFG:
        with conn.cursor() as cur:
            cur.execute(f"SELECT extract(epoch FROM (now() - MAX(ingested_at))) FROM {tbl}")
            (age,) = cur.fetchone()
            g_age.labels(tbl).set(age or 0)
            g_meets.labels(tbl).set(1 if (age or 0) < threshold else 0)

if __name__ == '__main__':
    start_http_server(9200)
    conn = psycopg2.connect('host=warehouse dbname=analytics user=slo_reader')
    conn.autocommit = True
    while True:
        scrape(conn)
        time.sleep(60)

# Prometheus recording rule — roll up to a 28-day SLO percentage
groups:
- name: data_slo_rollups
  interval: 60s
  rules:
    - record: slo:freshness:28d
      expr: |
        sum_over_time(data_freshness_meets_slo[28d])
          / count_over_time(data_freshness_meets_slo[28d])

    - record: slo:freshness:budget_remaining_28d
      expr: |
        (slo:freshness:28d - 0.999) / (1 - 0.999)
      # value > 0 means budget remaining; 0 means exhausted; < 0 means overspent

Step-by-step explanation.

The SLI query runs on the warehouse and computes now() - MAX(ingested_at) — the age of the newest row. This is a cheap query when ingested_at is indexed; sub-100ms on any modern warehouse. Running it every 60 seconds is negligible cost.
The exporter converts the raw age into two Prometheus gauges: data_freshness_age_seconds (the continuous measurement) and data_freshness_meets_slo (the 0/1 binary check against the threshold). Both are labelled by table name.
The recording rule slo:freshness:28d averages the binary meets_slo gauge over the trailing 28 days. If 99.9% of one-minute observations were "1", the SLO percentage is 0.999. Recording it as a rule (rather than computing at query time) makes dashboards and alerts cheap.
The budget_remaining_28d rule normalises the SLO into a "how much budget is left" number: (current − target) / (1 − target). A value of 0.5 means half the budget remains; 0 means exhausted; negative means overspent. This is the number you show in a big-bold-number Grafana panel.
The 28-day window is a rolling window — new minutes enter, old minutes fall off. A team that recovers reliability today sees the budget replenish gradually over the next 28 days, not resurrect instantly. This aligns incentives correctly: no "we'll fix it Monday and pretend nothing happened."

Output.

Table	age_seconds	meets_slo	28-day SLO	Budget remaining
analytics.orders	1247	1	0.9994	60%
analytics.customers	43200	1	1.0000	100%
analytics.events	923	1	0.9987	30%

Rule of thumb. Every table under a freshness SLO gets two gauges (age_seconds, meets_slo), one recording rule (slo:freshness:28d), and one big-bold-number Grafana panel showing budget remaining. This is the minimum viable freshness SLO stack.

Worked example — completeness SLI via source-target reconcile

Detailed explanation. The completeness SLI compares source and target row counts per hour and computes a ratio. It requires read access to both source and target and a shared time-column for windowing. Walk through the pattern for the orders pipeline.

Metric definition. completeness_ratio{table="orders", hour="…"} = target_rowcount / source_rowcount.
SLO. 99.5% of hours have ratio ≥ 0.99.
Cadence. Compute at the top of each hour, once the safety window has passed.
Storage. Persist per-hour to a slo_measurements table for the 28-day rolling calculation.

Question. Write the completeness reconcile SQL and the daily rollup that produces the SLO percentage.

Input.

Component	Value
Source	production.orders (event_ts)
Target	analytics.orders (event_ts, ingested_at)
Safety window	5 minutes
SLO threshold	ratio ≥ 0.99
SLO target	99.5%

Code.

-- Completeness reconcile — computed hourly, one row per (table, hour)
INSERT INTO slo_measurements(sli_name, table_name, window_start, value_num, meets_slo)
WITH windows AS (
    SELECT generate_series(
        date_trunc('hour', now() - INTERVAL '2 hours'),
        date_trunc('hour', now() - INTERVAL '1 hour'),
        INTERVAL '1 hour'
    ) AS window_start
),
src AS (
    SELECT w.window_start,
           COUNT(*) AS n_src
    FROM   windows w
    LEFT JOIN production.orders o
      ON o.event_ts >= w.window_start
     AND o.event_ts <  w.window_start + INTERVAL '1 hour'
    GROUP BY w.window_start
),
tgt AS (
    SELECT w.window_start,
           COUNT(*) AS n_tgt
    FROM   windows w
    LEFT JOIN analytics.orders o
      ON o.event_ts >= w.window_start
     AND o.event_ts <  w.window_start + INTERVAL '1 hour'
    GROUP BY w.window_start
)
SELECT
    'completeness'::text                             AS sli_name,
    'analytics.orders'::text                         AS table_name,
    src.window_start,
    tgt.n_tgt::numeric / NULLIF(src.n_src, 0)        AS value_num,
    CASE WHEN tgt.n_tgt::numeric / NULLIF(src.n_src, 0) >= 0.99
         THEN 1 ELSE 0 END                           AS meets_slo
FROM src
JOIN tgt USING (window_start);

-- Rollup — the 28-day completeness SLO percentage
SELECT
    table_name,
    AVG(meets_slo)::numeric(6, 5) AS slo_28d,
    COUNT(*) FILTER (WHERE meets_slo = 0)      AS bad_hours,
    COUNT(*)                                    AS total_hours,
    ROUND((AVG(meets_slo) - 0.995) / (1 - 0.995), 3) AS budget_remaining_frac
FROM slo_measurements
WHERE sli_name = 'completeness'
  AND window_start >= now() - INTERVAL '28 days'
GROUP BY table_name;

Step-by-step explanation.

The reconcile query uses generate_series to enumerate the specific hour windows to check — we deliberately go 1–2 hours back so the safety window (5 min) has already passed. This prevents counting in-flight rows as missing.
Left-joining source and target per hour gives us n_src and n_tgt even when a table is empty for the window. NULLIF(n_src, 0) avoids division-by-zero when the source itself was silent.
The ratio n_tgt::numeric / n_src is stored in value_num; the binary meets_slo records whether the ratio met the 0.99 threshold. Persisting both lets you back-fill dashboards and audit historical breaches.
The rollup computes AVG(meets_slo) over the trailing 28 days — that's the SLO percentage. bad_hours / total_hours gives you the human-readable version ("42 bad hours out of 672 = 93.75%").
The budget_remaining_frac column normalises the SLO into a "how much room is left" number. Positive = budget remaining; zero = exhausted; negative = you've overspent (and if this is a paying-customer SLA, credits are owed). Publish this to the Grafana dashboard next to the freshness budget.

Output.

table_name	slo_28d	bad_hours	total_hours	budget_remaining
analytics.orders	0.99702	2	672	40.4%
analytics.customers	0.99851	1	672	70.2%
analytics.events	0.99405	4	672	-19.0%

Rule of thumb. For every completeness SLO, persist the per-window measurement to a slo_measurements table — never recompute the SLO from raw source/target counts at query time. This gives you an auditable, backfill-tolerant, dashboard-cheap SLO history.

Worked example — quality SLI via dbt tests / Great Expectations

Detailed explanation. The quality SLI is the fraction of rows passing a battery of schema and business-rule checks. Modern data teams express these as dbt tests, Great Expectations expectations, or SQL data-diff assertions. Walk through wiring dbt tests to a quality SLO.

Test suite. unique(order_id), not_null(order_id, customer_id, total_cents, status), accepted_values(status), relationships(customer_id) → customers.id, range(total_cents, 0, 10_000_000).
SLI definition. quality_pass_rate = rows_passing_all / total_rows, computed per daily dbt run.
SLO. 99.9% of daily runs have pass_rate ≥ 0.99.

Question. Configure the dbt tests, compute the quality pass rate, and roll it up to a monthly SLO.

Input.

Component	Value
Test framework	dbt tests (Postgres/Snowflake adapter)
Test batch	daily post-build
SLI pass threshold	≥ 0.99
SLO target	99.9% of daily runs

Code.

# dbt schema.yml — quality tests on the orders model
version: 2
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('customers')
              field: id
      - name: total_cents
        tests:
          - not_null
          - dbt_utils.accepted_range:
              min_value: 0
              max_value: 10000000
      - name: status
        tests:
          - not_null
          - accepted_values:
              values: ['pending', 'shipped', 'delivered', 'cancelled', 'refunded']

-- Post-run macro — compute pass rate + insert into slo_measurements
{% macro record_quality_sli() %}
    INSERT INTO slo_measurements(sli_name, table_name, window_start, value_num, meets_slo)
    WITH results AS (
        SELECT model_name,
               SUM(CASE WHEN status = 'pass' THEN row_count ELSE 0 END) AS passed,
               SUM(row_count) AS total
        FROM   dbt_run_results
        WHERE  run_started_at >= date_trunc('day', now())
        GROUP  BY model_name
    )
    SELECT 'quality'                                 AS sli_name,
           'analytics.' || model_name                AS table_name,
           date_trunc('day', now())                  AS window_start,
           passed::numeric / NULLIF(total, 0)        AS value_num,
           CASE WHEN passed::numeric / NULLIF(total, 0) >= 0.99
                THEN 1 ELSE 0 END                    AS meets_slo
    FROM results;
{% endmacro %}

-- Monthly rollup
SELECT
    table_name,
    AVG(meets_slo)::numeric(6, 5) AS quality_slo_30d,
    COUNT(*) FILTER (WHERE meets_slo = 0) AS bad_days,
    COUNT(*)                              AS total_days
FROM slo_measurements
WHERE sli_name = 'quality'
  AND window_start >= now() - INTERVAL '30 days'
GROUP BY table_name;

Step-by-step explanation.

The dbt schema.yml declares the tests: uniqueness, null-ness, range, accepted values, and referential integrity. Each is a first-class dbt test that emits a pass or fail result with a row count of failing records.
The post-run macro (invoked via on-run-end in dbt_project.yml) aggregates the results into a per-model pass rate. Passed rows / total rows, computed per model, per day. This becomes the quality SLI.
The meets_slo binary records whether the day's pass rate cleared 99%. Persisting both the ratio (value_num) and the binary (meets_slo) lets you show both trend charts and SLO percentages.
The monthly rollup — same shape as freshness and completeness — computes AVG(meets_slo) over the trailing 30 days. A 99.9% SLO with 30 daily runs allows exactly 0.03 bad days per month — effectively "never miss." A 99% SLO allows ~1 bad day per quarter.
The reason quality SLOs are typically stricter (99.9% vs freshness's 99.9% of minutes) is that a daily run has far fewer observation windows than a per-minute freshness check. The same SLO percentage translates to very different bad-window counts based on cadence.

Output.

table_name	quality_slo_30d	bad_days	total_days
analytics.orders	1.00000	0	30
analytics.customers	0.96667	1	30
analytics.events	1.00000	0	30

Rule of thumb. For every table with a dbt or Great Expectations test suite, wire the pass-rate metric into the same slo_measurements table as freshness and completeness. Unified table + unified rollup rule = one dashboard for all four SLIs.

Senior interview question on SLIs and SLOs

A senior interviewer might ask: "You inherit a 500-table warehouse where 'monitoring' means Airflow email alerts on task failure. Design the SLI/SLO framework — how you'd pick the top tables to instrument, what SLIs you'd add per table, how you'd store measurements, and how you'd derive per-consumer SLO targets."

Solution Using a top-N table-prioritisation approach with a unified slo_measurements table and consumer-driven SLO derivation

-- 1. Prioritise tables by downstream query volume (warehouse audit log)
CREATE OR REPLACE VIEW slo_candidate_tables AS
SELECT
    table_schema || '.' || table_name AS table_name,
    COUNT(DISTINCT user_name)          AS distinct_users_7d,
    COUNT(*)                           AS queries_7d,
    ROW_NUMBER() OVER (ORDER BY COUNT(*) DESC) AS rn
FROM   snowflake.account_usage.access_history
WHERE  query_start_time >= now() - INTERVAL '7 days'
GROUP  BY 1;

-- The top 20 tables cover ~80% of downstream reads (Pareto)
SELECT * FROM slo_candidate_tables WHERE rn <= 20;

-- 2. Unified SLO measurement table
CREATE TABLE slo_measurements (
    sli_name         TEXT        NOT NULL,     -- 'freshness' | 'completeness' | 'volume' | 'quality'
    table_name       TEXT        NOT NULL,
    window_start     TIMESTAMPTZ NOT NULL,
    value_num        NUMERIC,                   -- raw measurement
    meets_slo        SMALLINT    NOT NULL,      -- 0 or 1
    recorded_at      TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (sli_name, table_name, window_start)
);

CREATE INDEX idx_slo_meas_rollup
  ON slo_measurements (table_name, sli_name, window_start DESC);

-- 3. SLO catalogue — one row per (table, sli) with the agreed target
CREATE TABLE slo_catalogue (
    table_name       TEXT NOT NULL,
    sli_name         TEXT NOT NULL,
    slo_target       NUMERIC NOT NULL,      -- e.g. 0.999
    consumer_team    TEXT NOT NULL,          -- who signed the SLO
    threshold_json   JSONB NOT NULL,          -- SLI-specific parameters
    approved_at      TIMESTAMPTZ NOT NULL,
    approved_by      TEXT NOT NULL,
    PRIMARY KEY (table_name, sli_name)
);

# 4. SLO rollup job — runs hourly; recomputes 28-day percentages
import psycopg2

def rollup_slos(conn):
    with conn.cursor() as cur:
        cur.execute("""
            SELECT c.table_name, c.sli_name, c.slo_target,
                   AVG(m.meets_slo)::numeric(6, 5) AS current_slo_28d
            FROM   slo_catalogue c
            LEFT JOIN slo_measurements m
              ON m.table_name = c.table_name
             AND m.sli_name   = c.sli_name
             AND m.window_start >= now() - INTERVAL '28 days'
            GROUP BY c.table_name, c.sli_name, c.slo_target
        """)
        rows = cur.fetchall()

    for table, sli, target, current in rows:
        budget_remaining = (current - target) / (1 - target) if current else 0
        # Write to Prometheus pushgateway (or expose as Grafana table)
        push_metric('slo_current_28d',    current, table=table, sli=sli)
        push_metric('slo_budget_remaining', budget_remaining, table=table, sli=sli)

        # Ticket if any SLO's budget has dropped below 20%
        if 0 < budget_remaining < 0.2:
            open_ticket(f"SLO warning: {sli} on {table} at {budget_remaining:.0%} budget")

Step-by-step trace.

Step	Value	Reasoning
Prioritisation	top-20 tables by 7-day query count	Pareto — 20 tables = 80% of consumption
Unified table	`slo_measurements` with (sli_name, table_name, window) PK	one query for all four SLIs
SLO catalogue	`slo_catalogue` with per-(table, sli) target + consumer	contract, not target
Rollup cadence	hourly	recompute 28-day percentages cheaply
Ticket trigger	budget remaining < 20%	early warning before breach
Alert wiring	separate — burn-rate on the per-SLI meets_slo gauge	see next H2

After the framework is in place, adding an SLI for a new table is a two-step operation: add a row to slo_catalogue (with the agreed target and consumer), and wire the SLI-specific measurement into slo_measurements. All dashboards, rollups, and burn-rate alerts pick up the new SLI automatically because they iterate the catalogue.

Output:

Metric	Value
Tables covered by any SLI	top 20 (rising to 100 by Q2)
SLIs per table (target)	4 (freshness, completeness, volume, quality)
Rollup query cost	< 500 ms per hour
Budget-warning ticket latency	~1 hour of degradation
SLO catalogue reviewed	quarterly

Why this works — concept by concept:

Top-N prioritisation — you cannot instrument 500 tables on day one. Ranking by downstream query volume (from the warehouse's own audit log) gives you the 20 tables that carry 80% of the reliability impact. Start there; extend as budget allows.
Unified slo_measurements table — one wide-format table with sli_name as a column beats one narrow-format table per SLI. It means one dashboard, one rollup query, one alerting scheme covers all four axes. Adding a new SLI type is a data change, not a schema change.
slo_catalogue as a contract — the catalogue is a Git-tracked, code-reviewed source of truth: which tables have which SLOs, what the targets are, who agreed to them, and when. An SLO that doesn't appear in the catalogue does not exist.
Consumer-signed targets — the consumer_team and approved_by columns enforce the contract-not-target discipline. The platform team cannot unilaterally set a 99.9% SLO on orders; the fraud model owner (or whoever depends on it) must agree in writing.
Cost — one 20-column slo_measurements row per (table, sli, window) — for 100 tables × 4 SLIs × per-hour cadence over 28 days, that's ~270k rows/month. Sub-gigabyte at rest, sub-second at query. The Prometheus/Grafana stack costs ~$50/month at this scale. Compared to $200k/year "data observability" tools that don't do SLO negotiation, this is 200x cheaper and 3x more valuable in interview answers.

Data validation
Topic — data-validation
Data validation and SLI problems

Practice →

SQL Topic — sql SQL reconcile and freshness problems

Practice →

3. Error budgets and burn-rate alerts

99.9% freshness = 43 minutes per month — spend it deliberately

The mental model in one line: an error budget is the operational contract that turns a percentage SLO into a concrete quantity of "bad time" the team is permitted to incur before feature releases must halt — 99.9% over a 30-day month is exactly 43 minutes and 12 seconds, and every SRE-style alerting decision (page now, ticket later, ignore) is grounded in how fast the team is spending against that budget, not on whether an individual event happened. Every senior data engineer must be able to derive the budget from the SLO percentage in their head, describe the multi-window multi-burn-rate alerting pattern, and defend the policy that halts releases when the budget is exhausted.

The budget math every senior DE knows without hesitation.

99% over 30 days = 30 × 24 × 60 × 0.01 = 432 minutes ≈ 7.2 hours.
99.5% = 216 minutes ≈ 3.6 hours.
99.9% = 43.2 minutes.
99.95% = 21.6 minutes.
99.99% = 4.32 minutes — "four nines" is aggressive for a data pipeline; usually reserved for the streaming layer where the source-of-truth is upstream.

Burn rate — the derivative of the budget.

Definition. Burn rate = the rate at which the SLI is spending budget divided by the sustainable rate that would exhaust the budget exactly at month-end.
Sustainable = 1x. If you spend budget at 1x, you'll use it all over 30 days — you'll hit exactly 99.9%.
14x = disaster. Spending at 14x means the whole month's budget would be gone in ~2 days. This is the fast-burn threshold that pages immediately.
6x = concerning. Spending at 6x means the budget is gone in ~5 days. This is the slow-burn threshold that tickets the team.
1x = normal. Spending at 1x is exactly on-plan — no alert.

The multi-window multi-burn-rate alerting pattern (Google SRE workbook).

Fast-burn (short window). Compute burn rate over a 1-hour window. If it exceeds 14x, page immediately. Detects cliffs quickly with minimal delay.
Slow-burn (long window). Compute burn rate over a 6-hour window. If it exceeds 6x, ticket the team. Detects creeping degradation without false pages.
Both windows use the same SLI. The alert doesn't fire on an event; it fires on the rate of budget consumption over a rolling window.
Why multi-window. A single 5-minute burn spike shouldn't page (transient); a 1-hour sustained burn should. A 6-hour slow burn shouldn't page but must be seen (the ticket).

Why not alert on every SLI dip?

Alert fatigue. A hair-trigger alert on any missed minute produces dozens of pages per week, each of which the on-call resolves by acknowledging and going back to sleep. The signal-to-noise ratio drops to zero.
False-positive cost. Every false page has a human cost (interrupted sleep, distraction, morale). The Google SRE book estimates false-page cost at ~$100 per event in engineering productivity; a team with 10 false pages per week costs the org $50k/year.
Missed-signal cost. Under-alerting means the team learns about incidents from the CEO, not the pager. Balance is via the multi-window burn-rate mechanism — page when the budget consumption rate is genuinely alarming.

The release-halt policy.

When the budget is exhausted. No feature releases against that pipeline until the budget replenishes at month-start. Only reliability fixes ship.
Why the halt is non-negotiable. The alternative — "release anyway" — invalidates every future SLO promise. Consumers cannot trust "99.9% freshness" if the platform team ships changes when the budget is empty and freshness is already at 98%.
How the halt is enforced. CI check that reads the SLO status; blocks the deploy if budget < 0. Emergency override with director-level approval only.
What "release" means. Any change to the pipeline code, DAG DSL, transformation SQL, source-schema contract, or infrastructure. Not just "new feature" — includes "innocuous refactor" too.

Common interview probes on error budgets.

"How many minutes of downtime does 99.9% allow per month?" — required answer: 43.
"What's a burn-rate alert?" — required answer: multi-window multi-rate on the SLI, from the SRE workbook.
"Why don't you page on every SLI dip?" — required answer: alert fatigue; false-page cost.
"What happens when the error budget is exhausted?" — required answer: release halt; only reliability fixes ship.

Worked example — freshness burn-rate PromQL against a freshness metric

Detailed explanation. The canonical multi-window multi-burn-rate alert setup: two rules, one fast (1-hour window, 14x threshold) and one slow (6-hour window, 6x threshold). Both reference the same data_freshness_meets_slo gauge. Walk through the PromQL.

SLI gauge. data_freshness_meets_slo{table="analytics.orders"} — 0/1 per minute.
Target. 99.9% (bad-minute rate = 0.001).
Fast alert. 1-hour window, 14x burn = 1.4% bad in the hour = page.
Slow alert. 6-hour window, 6x burn = 0.6% bad over the 6 hours = ticket.

Question. Write the Prometheus alert rules for the freshness SLI on analytics.orders.

Input.

Component	Value
SLI	`data_freshness_meets_slo`
SLO target	0.999
Fast window	1 h
Slow window	6 h
Fast burn threshold	14× normal = ≥ 1.4% bad
Slow burn threshold	6× normal = ≥ 0.6% bad

Code.

# Prometheus alerting rules — multi-window multi-burn-rate for freshness
groups:
- name: freshness_burn_rate
  rules:
    # Fast burn — page
    - alert: FreshnessBudgetFastBurn
      expr: |
        (
          1 - avg_over_time(data_freshness_meets_slo{table="analytics.orders"}[1h])
        ) > 0.014       # 14 × 0.001
        and
        (
          1 - avg_over_time(data_freshness_meets_slo{table="analytics.orders"}[5m])
        ) > 0.014
      for: 2m
      labels:
        severity: critical
        table: analytics.orders
        sli: freshness
      annotations:
        summary: "Freshness burning 14× normal — page on-call"
        runbook: "https://runbooks.internal/data/freshness-orders"
        current_burn: "{{ $value | humanizePercentage }}"

    # Slow burn — ticket
    - alert: FreshnessBudgetSlowBurn
      expr: |
        (
          1 - avg_over_time(data_freshness_meets_slo{table="analytics.orders"}[6h])
        ) > 0.006       # 6 × 0.001
        and
        (
          1 - avg_over_time(data_freshness_meets_slo{table="analytics.orders"}[30m])
        ) > 0.006
      for: 15m
      labels:
        severity: warning
        table: analytics.orders
        sli: freshness
      annotations:
        summary: "Freshness burning 6× normal for 6h — file ticket"
        runbook: "https://runbooks.internal/data/freshness-orders"
        current_burn: "{{ $value | humanizePercentage }}"

Step-by-step explanation.

The avg_over_time(...meets_slo[1h]) computes the fraction of minutes in the last hour that met the SLO. Subtracting from 1 gives the bad fraction. If bad fraction > 1.4%, we're spending at 14x the sustainable rate — the fast-burn threshold.
The and clause with a 5-minute window is the "double window" guard: the burn rate must be high in both the 1-hour and the 5-minute view. This prevents an old spike (already resolved) from re-firing the alert as it slowly ages out of the 1-hour window.
for: 2m is the Prometheus stability duration — the alert must be true for 2 minutes before firing. This absorbs metric-scrape jitter and prevents flapping. Combined with the burn-rate math, the fast-burn alert fires ~7 minutes after a cliff-fall begins.
The slow-burn rule mirrors the structure but with a 6-hour window and 6x threshold. for: 15m is longer because slow burns are inherently slower-changing; the extra stability window prevents ticket noise.
The annotations.runbook link is mandatory — no runbook link = alert doesn't ship. The on-call reads the runbook the moment the page arrives; without it, MTTR balloons while they figure out what to do.

Output.

Burn rate	Time-to-fire	Alert path	Human action
14× sustained 7 min	~7 min after cliff	fast-burn → PagerDuty → phone	page; on-call runs runbook
6× sustained 15 min in 6h window	~15 min into slow burn	slow-burn → PagerDuty → Slack	ticket; team triages
< 6×	never	no alert	budget dashboard shows draw

Rule of thumb. Every freshness (and completeness, volume, quality) SLO ships with two burn-rate alerts: one fast, one slow, sharing the same SLI. Skipping either creates a gap — fast-only misses creeping degradation; slow-only misses cliff-falls.

Worked example — monthly budget accounting spreadsheet template

Detailed explanation. The team needs a monthly view of every SLO's budget consumption to hold quarterly reviews and decide next-quarter SLO adjustments. Build the accounting template.

Columns. Table, SLI, Target, Actual, Budget spent (min), Budget remaining (min), Status.
Cadence. Refreshed daily; reviewed weekly at the platform team standup; formally reviewed monthly.
Source. The slo_measurements + slo_catalogue tables from H2 2.

Question. Build the monthly budget-accounting SQL that produces one row per (table, SLI) with budget spent and remaining.

Input.

Component	Value
Source tables	slo_measurements, slo_catalogue
Window	rolling 30 days
Cadence	daily refresh
Output columns	table, sli, target, actual, spent_min, remaining_min, status

Code.

-- Monthly SLO budget accounting
WITH observed AS (
    SELECT
        c.table_name,
        c.sli_name,
        c.slo_target,
        AVG(m.meets_slo)::numeric AS actual_slo,
        COUNT(*)                    AS observation_windows,
        COUNT(*) FILTER (WHERE m.meets_slo = 0) AS bad_windows
    FROM   slo_catalogue c
    JOIN   slo_measurements m
      ON   m.table_name = c.table_name
     AND   m.sli_name   = c.sli_name
     AND   m.window_start >= now() - INTERVAL '30 days'
    GROUP  BY c.table_name, c.sli_name, c.slo_target
),
budget AS (
    SELECT
        table_name,
        sli_name,
        slo_target,
        actual_slo,
        observation_windows,
        bad_windows,
        -- convert to minutes assuming per-minute observation windows
        ROUND((1 - slo_target) * observation_windows) AS budget_min_total,
        bad_windows                                    AS budget_min_spent,
        GREATEST(0, ROUND((1 - slo_target) * observation_windows) - bad_windows) AS budget_min_remaining
    FROM observed
)
SELECT
    table_name,
    sli_name,
    slo_target,
    actual_slo,
    budget_min_total,
    budget_min_spent,
    budget_min_remaining,
    CASE
        WHEN budget_min_remaining = 0 AND actual_slo < slo_target THEN 'EXHAUSTED — halt releases'
        WHEN budget_min_remaining::float / budget_min_total < 0.2 THEN 'CRITICAL — 20% left'
        WHEN budget_min_remaining::float / budget_min_total < 0.5 THEN 'WARNING — 50% left'
        ELSE                                                            'HEALTHY'
    END AS status
FROM budget
ORDER BY budget_min_remaining::float / budget_min_total ASC;

Step-by-step explanation.

The observed CTE joins the catalogue against the measurements over the last 30 days, aggregating to per-(table, SLI) actual SLO percentage, count of observation windows, and count of bad windows.
The budget CTE computes three quantities: budget_min_total (total minutes of budget the SLO allows), budget_min_spent (already-consumed bad minutes), budget_min_remaining (what's left, floored at 0). The math is: total = (1 − target) × windows; if target is 0.999 and windows is 43200 minutes/month, total = 43.2 minutes.
The status column translates numeric budget-remaining into an operational label. EXHAUSTED triggers the release-halt; CRITICAL and WARNING are ticket-level heads-ups.
Ordering by "% budget remaining ASC" puts the most at-risk SLOs at the top of the table — the review meeting focuses on those first.
The monthly review meeting walks this table row by row: for each CRITICAL or EXHAUSTED row, discuss what happened, what action was taken, whether the SLO target is still reasonable. For each HEALTHY row, discuss whether the target could be tightened next quarter.

Output.

table	sli	target	actual	total	spent	remaining	status
analytics.orders	freshness	0.999	0.9993	43	30	13	CRITICAL — 20% left
analytics.events	freshness	0.999	0.9987	43	56	0	EXHAUSTED — halt releases
analytics.customers	freshness	0.999	1.0000	43	0	43	HEALTHY
analytics.orders	completeness	0.995	0.9970	216	130	86	WARNING — 50% left

Rule of thumb. The monthly budget accounting table is the single most useful reliability artifact for a senior data-platform team. Review it every Monday standup; walk it row-by-row every month-end; use it to decide whether to tighten or loosen each SLO next quarter.

Worked example — release-halt policy enforced in CI

Detailed explanation. When an SLO's budget is exhausted, releases against the affected pipeline must halt. The most effective enforcement is a CI check that reads the SLO status and fails the deploy if any relevant SLO is EXHAUSTED. Walk through the check.

Check inputs. Which tables the PR touches (from git diff of dbt models / DAG code), current SLO status of each.
Fail condition. Any touched table has an exhausted budget.
Emergency override. A director-level approver adds the budget-override label to the PR; the check is skipped for that PR.
Reliability fixes exempt. PRs labelled reliability-fix are exempt.

Question. Write the CI check script that halts a release when any relevant SLO is exhausted.

Input.

Component	Value
Trigger	PR to main branch
Data source	slo_measurements view or Prometheus
Fail condition	budget_min_remaining = 0 for touched tables
Override	PR label `budget-override` (director sign-off)
Exempt	PR label `reliability-fix`

Code.

# release_halt_check.py — runs in CI
import os
import subprocess
import sys
import psycopg2

DB_DSN     = os.environ['SLO_DB_DSN']
PR_LABELS  = os.environ.get('PR_LABELS', '').split(',')

# Exempt labels
if 'reliability-fix' in PR_LABELS or 'budget-override' in PR_LABELS:
    print('Skipping SLO check due to label')
    sys.exit(0)

# 1. Identify touched tables from the PR diff
changed = subprocess.check_output(['git', 'diff', '--name-only', 'main...HEAD']).decode()
touched_tables = set()
for path in changed.splitlines():
    # dbt models under models/analytics/orders.sql -> analytics.orders
    if path.startswith('models/'):
        parts = path.replace('models/', '').replace('.sql', '').split('/')
        if len(parts) == 2:
            touched_tables.add(f'{parts[0]}.{parts[1]}')

# 2. Query current SLO status for those tables
conn = psycopg2.connect(DB_DSN)
with conn.cursor() as cur:
    cur.execute("""
        SELECT table_name, sli_name, budget_min_remaining, status
        FROM   v_slo_budget_status
        WHERE  table_name = ANY(%s)
    """, (list(touched_tables),))
    rows = cur.fetchall()

# 3. Fail if any SLO is exhausted
exhausted = [r for r in rows if r[3] == 'EXHAUSTED — halt releases']
if exhausted:
    print('SLO release halt triggered:')
    for r in exhausted:
        print(f'  {r[0]} · {r[1]} · remaining={r[2]} min · {r[3]}')
    print('\nAdd label `reliability-fix` to ship a reliability fix,')
    print('or `budget-override` (director sign-off) for an emergency override.')
    sys.exit(1)

print('SLO check passed — all touched tables within budget.')
sys.exit(0)

# .github/workflows/deploy.yml — invoke the check
name: Deploy pipeline changes
on:
  pull_request:
    branches: [main]

jobs:
  slo-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: SLO release-halt gate
        env:
          SLO_DB_DSN: ${{ secrets.SLO_DB_DSN }}
          PR_LABELS:  ${{ join(github.event.pull_request.labels.*.name, ',') }}
        run: python3 release_halt_check.py

Step-by-step explanation.

The CI check first inspects PR labels — reliability-fix and budget-override are exempt. This is the escape hatch for legitimate reliability work and for director-approved emergencies. The label check is the first gate because the queries below are unnecessary if we're exempt.
The script parses the git diff to extract which tables the PR touches. dbt convention is models/<schema>/<table>.sql → <schema>.<table>. Extend for other frameworks (Airflow DAGs → touched table names via a dag_to_tables.yaml mapping).
The v_slo_budget_status view (built on top of the earlier budget-accounting query) returns the current status per SLO. Only the SLOs on touched tables are relevant to this PR.
Any EXHAUSTED SLO fails the check. The error message names the exact SLO(s) and points to the two override labels. This is the contract the release-halt policy makes with engineers — no surprises, always a documented path forward.
The workflow file wires the check into every PR. GitHub required-checks makes this gate mandatory for merge; only admin override can bypass. This is the mechanism by which the SLO becomes an operational constraint, not a decorative percentage.

Output.

PR touches	Any SLO exhausted?	Label	CI result
analytics.orders (freshness EXHAUSTED)	yes	(none)	fails; blocks merge
analytics.orders (freshness EXHAUSTED)	yes	reliability-fix	passes
analytics.orders (freshness EXHAUSTED)	yes	budget-override	passes; requires director
analytics.customers (all HEALTHY)	no	(none)	passes

Rule of thumb. The release-halt policy without CI enforcement is a suggestion. With CI enforcement — required check + labelled exemptions + audit log of overrides — it becomes the operational reality that keeps SLO promises credible.

Senior interview question on error budgets

A senior interviewer might ask: "You have a freshness SLO of 99.9% on the orders warehouse table. Halfway through the month, an upstream schema change caused a 6-hour outage, burning most of the monthly budget. What alerts should have fired earlier, what's the release policy for the rest of the month, and how do you use this incident to negotiate a saner SLO next quarter?"

Solution Using multi-window burn-rate alerts, immediate release-halt, and a postmortem-driven SLO renegotiation

# 1. Alerts that should have fired (see H2 3 above) — verify these are wired
alerts_that_should_fire:
  fast_burn_1h:
    window: 1h
    threshold: 14x normal
    action: PagerDuty (page)
    expected_fire_time: ~15 min into outage

  slow_burn_6h:
    window: 6h
    threshold: 6x normal
    action: Slack + Jira ticket
    expected_fire_time: ~1 h into outage

  budget_low_warning:
    trigger: budget_min_remaining < 20% of total
    action: ticket to platform-team lead
    expected_fire_time: on next hourly rollup after threshold crossed

  budget_exhausted:
    trigger: budget_min_remaining = 0
    action: CI release-halt engaged; email leadership
    expected_fire_time: on next hourly rollup after breach

# 2. Post-incident policy runner — activates the release halt automatically
import os
import psycopg2
import requests

def enforce_release_halt():
    conn = psycopg2.connect(os.environ['SLO_DB_DSN'])
    with conn.cursor() as cur:
        cur.execute("""
            SELECT table_name, sli_name, budget_min_remaining
            FROM   v_slo_budget_status
            WHERE  budget_min_remaining = 0
        """)
        exhausted = cur.fetchall()

    if not exhausted:
        return

    # Update the shared release-gate flag consumed by CI
    with conn.cursor() as cur:
        for table, sli, _ in exhausted:
            cur.execute("""
                INSERT INTO release_gate(table_name, sli_name, halted_at, reason)
                VALUES (%s, %s, now(), 'budget_exhausted')
                ON CONFLICT (table_name, sli_name) DO NOTHING
            """, (table, sli))
        conn.commit()

    # Notify leadership + on-call
    body = 'Release halt engaged: ' + ', '.join(f'{t}·{s}' for t, s, _ in exhausted)
    requests.post(os.environ['SLACK_WEBHOOK'], json={'text': body})

if __name__ == '__main__':
    enforce_release_halt()

<!-- 3. Postmortem-driven SLO renegotiation template -->
# Incident: 6h freshness breach on analytics.orders — 2026-07-14

## SLO status this month
| SLI | Target | Actual (day 15) | Budget spent | Remaining |
|---|---|---|---|---|
| freshness | 99.9% | 98.7% | 388 min | 0 min (overspent by 345 min) |

## Root policy question
The SLO was set at 99.9% based on the dashboard-owner's stated tolerance
(~1h/month). Actual pipeline reliability over the last 90 days averaged
99.85% — the target was already tighter than achievable.

## Recommendation
Renegotiate to 99.5% freshness (216 min/month = 3.6 hours), which:
  • Reflects observed pipeline reliability
  • Still constrains bad hours to < 4/month
  • Allows the team to invest in one reliability project per quarter
    rather than burning weekly cycles on false-alert cleanup

## Investments required to sustain 99.9%
  • Hot-standby DAG on backup schedule (~2 engineer-weeks build, +25% compute)
  • Automatic upstream-schema-drift detection (~1 sprint)
  • Recommend: only invest if executive requires 99.9%

Step-by-step trace.

Concern	Answer	Reasoning
Alerts fired	fast + slow + budget-low	multi-window catches both cliff and creep
Release halt	automatic on budget = 0	CI-enforced; policy = credible
Mid-month spend	388 min out of 43	overspent by 8x — SLO not achievable at current reliability
Postmortem action	renegotiate to 99.5% or invest to hit 99.9%	contract must be sustainable
Next quarter	99.5% freshness or 2 sprints of reliability work	executive trade-off

After the incident, the release halt is enforced automatically; leadership is notified; the postmortem produces a concrete SLO-renegotiation recommendation grounded in observed reliability, not aspiration. Next quarter's SLO catalogue is updated with the new target and the reasoning.

Output:

Metric	Value
Incident duration	6 h
Budget consumption	388 min (~9× monthly budget)
SLO breached	freshness on analytics.orders
Release halt engaged	yes; auto-lifted at month-start
Postmortem recommendation	renegotiate to 99.5% freshness
SLO catalogue updated	next quarter

Why this works — concept by concept:

Multi-window multi-burn-rate — the fast-burn alert paged within 15 minutes of the cliff-fall; the slow-burn alert ticketed within an hour. No incident should ever be "found" hours after it started when both alerts are wired.
Automatic release halt — the release_gate table + CI check enforces the policy without human intervention. This is the mechanism that turns the error budget from an accounting fiction into an operational reality.
Postmortem-driven renegotiation — the incident revealed that 99.9% freshness was not achievable at current reliability. The postmortem is the vehicle for surfacing this and forcing a decision: renegotiate the SLO or invest in reliability. Either is fine; the wrong answer is "keep the target and hope."
Overspend accounting — the budget-remaining column goes negative when overspent (-345 min in this case). This visible number is the argument in the renegotiation meeting: "we set a target we systematically miss by 8x."
Cost — one PagerDuty page, one Jira ticket, one leadership email, one 1-hour postmortem meeting, one SLO catalogue update. Compared to the alternative — silently missing the SLO every month while consumers lose trust — this cycle costs about half an engineer-day per incident and buys credibility with every downstream team.

Time series
Topic — time-series
Time-series and burn-rate problems

Practice →

SQL Topic — time-series/sql SQL time-series and window problems

Practice →

4. On-call rotations and incident runbooks for data

Rotation × escalation × runbook — the three anchors of data on-call that scale past 10 pipelines

The mental model in one line: a data on-call rotation is the scheduled, documented ownership of who wakes up when an SLI breach pages, structured as a primary + secondary weekly rotation (or follow-the-sun for global teams) with a defined handoff ritual, a three-tier escalation ladder for when the primary cannot mitigate alone, and a 5-field runbook linked from every alert that turns "what do I do now?" into a scripted playbook — the three anchors together are what make data reliability sustainable past 10 pipelines and 3 on-call engineers. Every senior data engineer must be able to describe the rotation shape, the handoff protocol, the escalation matrix, and the runbook template in a single interview breath.

The rotation shape.

Primary + secondary. Two people on rotation each week. Primary gets paged first (5 min); secondary escalated to at 15 min if primary doesn't ack. Both are compensated for on-call.
Weekly cadence. One-week shifts. Shorter (daily) creates handoff overhead; longer (bi-weekly) creates burnout.
Follow-the-sun for global teams. A US team + an EU team + an APAC team can each own their business-hours 8-hour block and hand off at the boundary. Sub-daily rotations only in this model.
Fair rotation length. Aim for on-call every 4–8 weeks per engineer. More frequent burns people out; less frequent means they forget the muscle memory.

The handoff ritual.

When. Monday morning at a fixed time (e.g. 10:00 local for the incoming on-call).
Duration. 15–20 minutes.
Attendees. Outgoing primary, incoming primary, incoming secondary. Team lead optional.
Content. Open incidents, this-week's SLO/budget status, upcoming known changes (planned deploys, schema migrations), any custom alerts snoozed and why.
Artefact. Handoff doc committed to a handoffs/YYYY-MM-DD.md folder — permanent record for postmortems and audit.

The escalation ladder — three tiers.

Tier 1 (immediate, 0–5 min). Primary on-call. Pager wakes them. They own initial triage and mitigation.
Tier 2 (15 min if unresolved). Secondary + team lead. Extra hands, additional context. Common when Tier 1 has diagnosed but needs help executing (privileged access, unfamiliar system).
Tier 3 (30 min if unresolved). Engineering manager + platform SRE + downstream consumer contact. When incident is affecting external commitments or requires cross-team coordination.
Tier 4 (60 min). VP-level. Reserved for outages affecting revenue-generating customer flows or regulatory obligations.

The runbook — 5 fields, no more.

Symptom. What the alert looks like when it fires. "Freshness burn rate > 14× on analytics.orders for 5+ minutes."
Detection. Where to look to confirm the symptom is real. "Check the Grafana freshness_age_seconds panel; check the Airflow DAG status; check upstream Postgres pg_stat_replication."
Mitigation. The steps to stop the bleeding. "Restart the DAG. If still failing, disable the upstream connector to prevent backlog. If disk full, page storage team."
Escalation. When and to whom. "If not mitigated in 15 min, escalate to secondary + team lead. If cross-team required, engage Tier 3."
Owner. Who owns this runbook and reviews it quarterly. "@platform-team, reviewed 2026-Q2."

Why the runbook is mandatory.

Fresh 3 AM brain. On-call brains are 30% as effective at 3 AM. The runbook turns novel problem-solving into script-following.
Newer team members. A rotation with 6 engineers means each person owns 1/6 of the deep knowledge. The runbook is how the other 5/6 gets used.
Post-incident continuity. Runbooks improve after every incident (see H2 5 postmortem). A team that treats runbooks as living documents converges on incident-proof playbooks over quarters.

Common interview probes on on-call.

"What does your on-call rotation look like?" — required answer: primary + secondary + weekly + handoff ritual.
"Walk me through your escalation ladder." — required answer: three tiers with times and roles.
"What's in a runbook?" — required answer: 5 fields (symptom, detection, mitigation, escalation, owner).
"How do you keep runbooks up to date?" — required answer: reviewed quarterly + updated after every postmortem.

Worked example — runbook for "freshness SLO breach on orders table"

Detailed explanation. Build a concrete runbook for the freshness SLO breach on analytics.orders. Every alert on that SLI links to this runbook.

Alert. FreshnessBudgetFastBurn{table="analytics.orders", sli="freshness"}.
On-call target. Primary DE. Escalate to secondary + platform lead after 15 min.
Common causes. Upstream Postgres connector stalled, DAG task failure, warehouse merge queue backed up.

Question. Write the 5-field runbook for the freshness breach.

Input.

Field	Content type
Symptom	one-line description of the alert
Detection	3–5 diagnostic commands or dashboard checks
Mitigation	ordered list of remediation steps
Escalation	when and who
Owner	team + last-reviewed date

Code.

# Runbook — Freshness SLO breach on analytics.orders

## Symptom
Alert: `FreshnessBudgetFastBurn{table="analytics.orders", sli="freshness"}`.
Freshness burning at 14× normal for 5+ minutes. Downstream dashboards
(Looker "Sales Ops") will display stale data older than 65 minutes.

## Detection
1. **Grafana** — open the "Data reliability · analytics.orders" dashboard.
   Confirm `freshness_age_seconds` is rising and above 3900 s (65 min).
2. **Airflow** — check `dag_orders_hourly` in Airflow UI. Any red tasks?
   Any tasks stuck in `queued` for > 10 min?
3. **Upstream Postgres** — SSH to db-primary and run:
   `SELECT * FROM pg_replication_slots WHERE slot_name = 'debezium_orders';`
   Check `active = true` and `lag_bytes < 100MB`.
4. **Warehouse** — check Snowflake `WAREHOUSE_LOAD_HISTORY` for the merge
   warehouse. Is queue depth > 20 for the last 10 minutes?

## Mitigation
Try in order; move to next step if previous doesn't resolve within 5 min:

1. **Restart the DAG.** Airflow → dag_orders_hourly → Clear failed task.
   If succeeds, watch freshness metric for 15 min to confirm recovery.
2. **Restart the Debezium connector** (if replication slot is stale):
   `kubectl -n data rollout restart deploy/debezium-orders`.
   Watch slot lag drop; freshness should catch up within 5 min.
3. **Scale the merge warehouse** (if Snowflake queue is deep):
   `ALTER WAREHOUSE merge_wh SET WAREHOUSE_SIZE = 'LARGE';` for 1 h,
   then revert. Cost is +$4/h for one hour.
4. **Disable the upstream connector to protect backlog** (last resort):
   `kubectl -n data scale deploy/debezium-orders --replicas=0`.
   This stops new WAL consumption but preserves the slot. Only do this
   after Tier 2 approval — the pipeline will require manual catch-up.

## Escalation
- 15 min without resolution → page **secondary** + **@platform-lead**.
- 30 min → engage **@eng-manager-data** + notify **@sales-ops-owner**
  (dashboard consumer) via Slack #sales-ops.
- 60 min affecting revenue reporting → engage **@vp-data**.

## Owner
`@platform-team`. Last reviewed: 2026-Q2. Next review: 2026-Q3.
Update this runbook after every postmortem involving analytics.orders.

Step-by-step explanation.

The Symptom section names the exact alert label and describes what the reader will see. On-call arrives at the runbook 30 seconds after the page fires; they need to confirm "yes, this is my situation" in one glance.
The Detection section is a sequential checklist of dashboards and commands to run. Each step is copy-pasteable. Ordered from most-likely-to-be-the-issue (Airflow, upstream Postgres) to least-likely (warehouse queue). This ordering compresses average diagnosis time.
The Mitigation section is ordered remediation steps, each with a "stop if this works" guard. The order goes from lowest-risk (restart the DAG) to highest-risk (disable the upstream connector). Each step includes success criteria — "watch freshness metric for 15 min."
The Escalation section is time-bounded — 15 min, 30 min, 60 min. Escalation targets are named with @-handles so the on-call can page them directly from the runbook. The 60-min VP-level escalation is only for revenue-affecting incidents; naming the criterion prevents over-escalation.
The Owner section records who maintains the runbook and when it was last reviewed. Runbooks that go unreviewed for a year are stale; quarterly review + post-postmortem update keeps them alive. A runbook with no owner is a runbook no-one improves.

Output.

Field	Content
Symptom	2-sentence alert description + consumer impact
Detection	4-step checklist with copy-paste commands
Mitigation	4-step ordered try-list with success criteria
Escalation	3 time-bounded tiers with named targets
Owner	team + last-reviewed date + review cadence

Rule of thumb. Every runbook fits on one screen. If the mitigation section runs past 5 numbered steps, split it into sub-runbooks. On-call brains at 3 AM cannot follow a 20-step checklist.

Worked example — runbook for "volume SLI < 50% of 7-day median"

Detailed explanation. The volume SLI catches "row count is wildly wrong." Build the runbook for the alert that fires when the target table's count drops below 50% of the rolling 7-day median for the hour of day.

Alert. VolumeBurnRateFastBurn{table="analytics.orders"} — sustained bad-volume ratio in the 1-hour window.
Common causes. Upstream feed truncated, source query WHERE clause bug, backfill job overwriting live data.

Question. Write the runbook for the volume alert.

Input.

Field	Content type
Symptom	volume dropped below 50% of baseline
Detection	compare source vs target counts
Mitigation	quarantine target; investigate upstream
Escalation	product-owner + platform-lead
Owner	platform-team

Code.

# Runbook — Volume SLI breach on analytics.orders

## Symptom
Alert: `VolumeBurnRateFastBurn{table="analytics.orders"}`.
Hourly row count on analytics.orders is < 50% of the 7-day median for
this hour-of-day. Dashboards showing "orders per hour" will display a
misleading dip; downstream ML models may retrain on biased data.

## Detection
1. **Grafana** — "Data reliability · analytics.orders volume" panel.
   Confirm actual count is < 50% of expected baseline.
2. **Warehouse count query** — run:
   `SELECT date_trunc('hour', event_ts), COUNT(*) FROM analytics.orders
    WHERE event_ts >= now() - INTERVAL '3 hours' GROUP BY 1 ORDER BY 1;`
   Confirm the drop is real (not a rendering issue).
3. **Source count query** — run against production.orders:
   `SELECT date_trunc('hour', event_ts), COUNT(*) FROM production.orders
    WHERE event_ts >= now() - INTERVAL '3 hours' GROUP BY 1 ORDER BY 1;`
   Is the drop in the source too?
4. **Recent deploys** — check the changelog for any deploy in the last
   24 h that touched the orders pipeline or the source query.

## Mitigation

### Case A: source is also low (upstream issue)
1. Notify **@service-orders-owner** (upstream service team).
2. Do NOT modify the pipeline — it's correctly reflecting reality.
3. Post to #incidents: "Upstream orders source is producing X% of normal;
   downstream is accurate; investigating with @service-orders-owner."

### Case B: source is normal, target is low (pipeline issue)
1. **Quarantine the affected partitions**:
   `ALTER TABLE analytics.orders RENAME PARTITION FOR (h='2026-07-30 14:00')
    TO analytics.orders_quarantine_20260730_14;`
2. **Backfill** by running:
   `airflow tasks run dag_orders_hourly backfill --start-date=... --end-date=...`
3. Verify count matches source before dropping the quarantine partition.

## Escalation
- 20 min without root cause → page **secondary** + **@platform-lead**.
- If Case A and upstream unresponsive within 30 min → engage
  **@eng-manager-orders** and **@product-orders**.
- If Case B backfill takes > 2 h → notify dashboard consumers via
  #sales-ops Slack.

## Owner
`@platform-team`. Last reviewed: 2026-Q2. Next review: 2026-Q3.

Step-by-step explanation.

The symptom section again names the exact alert and calls out the consumer impact — the "misleading dip" language forces the on-call to understand what dashboards will show, not just what a metric says. This impact framing drives urgency.
The detection section is Case-A-vs-Case-B in disguise: comparing source and target counts is the diagnostic that branches the mitigation. This is a common pattern for volume runbooks — is it a data issue or a pipeline issue?
The mitigation section is branched into two cases based on the diagnostic. Case A (upstream issue) requires no pipeline change — a common on-call trap is to "fix" a pipeline that's correctly reflecting a broken source. Case B (pipeline issue) requires quarantine + backfill.
The quarantine step (RENAME PARTITION ... TO ..._quarantine_...) is a reversible mitigation — it removes bad data from the live table without deleting it. Backfill then produces the correct data; the quarantine partition can be dropped after verification.
Escalation names different people for Case A vs Case B — Case A escalates to the service-owner (upstream), Case B stays in the platform team. This differentiation is what stops the wrong team from being paged repeatedly for an incident they can't fix.

Output.

Case	Diagnostic	Mitigation	Escalation
A: source low	source count ≈ target count	no pipeline change; notify service-owner	@eng-manager-orders
B: source normal	source count normal; target low	quarantine + backfill	@platform-lead

Rule of thumb. For any SLI with more than one common cause, branch the mitigation section by diagnostic. A single monolithic mitigation list encourages the on-call to run every step in order regardless of what actually broke.

Worked example — escalation matrix and comms template

Detailed explanation. The escalation matrix documents who is paged at each tier for each incident class. The comms template gives the on-call ready-made language for Slack, status pages, and executive updates. Build both.

Matrix. Per-service, per-tier, named on-call groups.
Comms template. Fill-in-the-blank Slack message + hourly update cadence.

Question. Build the escalation matrix for the data platform and the standard comms template for a Sev-1 incident.

Input.

Component	Value
Services	orders, customers, events, payments
Tiers	1 (primary), 2 (+lead), 3 (+eng-mgr), 4 (+VP)
Comms channels	#incidents (Slack), status.internal (page), executive email
Update cadence	every 30 min while incident is live

Code.

# escalation_matrix.yml — versioned in Git; referenced by every runbook
services:
  orders:
    tier_1: '@data-oncall-primary'
    tier_2: '@data-oncall-secondary,@platform-lead'
    tier_3: '@eng-manager-data,@platform-sre,@service-orders-owner'
    tier_4: '@vp-data,@vp-eng'
  customers:
    tier_1: '@data-oncall-primary'
    tier_2: '@data-oncall-secondary,@platform-lead'
    tier_3: '@eng-manager-data,@platform-sre,@service-customers-owner'
    tier_4: '@vp-data,@vp-eng'
  events:
    tier_1: '@streaming-oncall-primary'
    tier_2: '@streaming-oncall-secondary,@streaming-lead'
    tier_3: '@eng-manager-streaming,@platform-sre'
    tier_4: '@vp-data,@vp-eng'
  payments:
    tier_1: '@data-oncall-primary'
    tier_2: '@data-oncall-secondary,@platform-lead,@security-oncall'
    tier_3: '@eng-manager-data,@security-lead,@finance-ops'
    tier_4: '@vp-data,@vp-eng,@cfo'   # PCI-scope; finance leadership involved

<!-- comms_template.md — copy-paste for the on-call -->
## Incident: {sev} — {short_title}

**Start:** {utc_timestamp}
**Impact:** {who_is_affected_and_how}
**Current status:** {investigating|mitigating|monitoring|resolved}
**Next update:** {timestamp}
**On-call:** {@primary}, {@secondary}
**Ticket:** {link_to_incident_doc}

### Timeline
- {t+00:00} Alert fired: {alert_name}
- {t+00:04} On-call acked; runbook opened
- {t+00:12} Detection confirmed: {finding}
- {t+00:22} Mitigation applied: {action}
- {t+00:35} Recovery observed on {metric}
- {t+01:00} Incident resolved; postmortem scheduled for {date}

### Consumer notification
| Consumer | Notified | Method |
|---|---|---|
| Sales Ops | yes | #sales-ops Slack + email to VP-Sales |
| Fraud ML | yes | #ml-fraud Slack |
| Reverse ETL | yes | #revops Slack |

Step-by-step explanation.

The escalation matrix is a single YAML file per data platform. Every runbook references it by service name (escalation: matrix.orders.tier_1) so name changes propagate. When a person leaves the team, one edit updates every runbook.
The tier structure is the same across services — 4 tiers, defined times. Different services have different Tier 2/3/4 populations (payments adds security + finance because of PCI scope), but the tier concept is stable.
The comms template gives the on-call a fill-in-the-blank Slack post. This matters because at 3 AM, composing an incident comms message from scratch is error-prone; the template ensures nothing critical is omitted (impact, next update, on-call, ticket).
The timeline section is populated in real time during the incident, not written afterwards. This becomes the raw material for the postmortem. t+00:00 timestamps are relative to the alert firing.
The consumer notification table forces the on-call to think about downstream consumers. Even a fully-mitigated incident has a comms obligation: "sales-ops, your dashboard was stale between 09:04 and 10:15." Under-communication is the surest way to erode trust in the platform team.

Output.

Artefact	Purpose	Cadence
escalation_matrix.yml	who to page per service per tier	reviewed quarterly
comms_template.md	Slack/email template for incidents	updated after major incidents
runbook links	detailed diagnostic + mitigation	reviewed per postmortem
handoff_docs/YYYY-MM-DD.md	weekly rotation handoff	one per Monday

Rule of thumb. The escalation matrix, comms template, runbooks, and handoff docs are the four operational artefacts that make on-call sustainable. All four live in Git alongside the pipeline code; all four are reviewed at defined cadences; all four are updated after every incident.

Senior interview question on data on-call

A senior interviewer might ask: "You lead a 6-engineer data platform team. Design the on-call rotation, escalation matrix, and runbook coverage that supports 24×7 SLO enforcement across 50 pipelines. Cover the rotation shape, handoff protocol, runbook backlog prioritisation, and how you'd measure the on-call load on each engineer over time."

Solution Using a weekly primary+secondary rotation with follow-the-sun handoff, tiered escalation, prioritised runbook backlog, and on-call load tracking

# 1. Rotation shape — weekly primary + secondary; follow-the-sun over EU/US
rotation:
  cadence: weekly
  handoff_day: Monday
  handoff_time: 10:00 local (incoming)
  primary_count: 1
  secondary_count: 1
  min_gap_between_shifts: 4 weeks
  compensation: 1 day off per week on-call + on-call stipend

  regions:
    - name: US
      hours: 08:00-20:00 US-Central
      pool: [alice, bob, carol, dan]
    - name: EU
      hours: 08:00-20:00 CET
      pool: [emily, frank]
    - name: overnight
      hours: 20:00-08:00 US-Central
      strategy: secondary in EU takes overnight for the US half of the week; vice versa

# 2. Escalation matrix (see previous example)

# 3. Runbook backlog — prioritise by SLO coverage + incident frequency
runbook_backlog_priority:
  ranking_formula: (downstream_users × pageable_alerts_per_month) / runbook_maturity
  # runbook_maturity: 0=none, 1=draft, 2=reviewed, 3=battle-tested
  target: every pageable alert has a runbook at maturity 2+

-- 4. On-call load tracking — pages per engineer per week
CREATE OR REPLACE VIEW oncall_load AS
SELECT
    engineer,
    date_trunc('week', page_time) AS week,
    COUNT(*)                       AS total_pages,
    COUNT(*) FILTER (WHERE hour < 8 OR hour > 20) AS off_hours_pages,
    AVG(EXTRACT(EPOCH FROM (resolved_time - page_time)) / 60.0)::int AS avg_resolve_min
FROM (
    SELECT engineer,
           page_time,
           resolved_time,
           EXTRACT(HOUR FROM page_time) AS hour
    FROM   pagerduty_incidents
    WHERE  page_time >= now() - INTERVAL '90 days'
) t
GROUP BY engineer, date_trunc('week', page_time)
ORDER BY week DESC, total_pages DESC;

-- Alert on team-lead if any engineer exceeds 10 off-hours pages / week
SELECT engineer, week, off_hours_pages
FROM   oncall_load
WHERE  off_hours_pages > 10
  AND  week >= now() - INTERVAL '4 weeks';

# 5. Runbook backlog auto-updater — runs monthly
def prioritise_runbook_backlog(conn):
    with conn.cursor() as cur:
        cur.execute("""
            SELECT c.table_name, c.sli_name,
                   COALESCE(rb.maturity, 0) AS runbook_maturity,
                   COUNT(p.id) AS pages_last_90d,
                   COALESCE(ct.downstream_users, 1) AS users
            FROM   slo_catalogue c
            LEFT JOIN runbooks rb USING (table_name, sli_name)
            LEFT JOIN pagerduty_incidents p
              ON p.alert_labels @> jsonb_build_object('table', c.table_name, 'sli', c.sli_name)
             AND p.page_time >= now() - INTERVAL '90 days'
            LEFT JOIN table_consumers ct USING (table_name)
            GROUP BY 1, 2, 3, 5
        """)
        rows = cur.fetchall()

    scored = []
    for table, sli, maturity, pages, users in rows:
        score = (users * pages) / (maturity + 1)   # higher = higher priority
        scored.append((table, sli, maturity, pages, users, score))

    scored.sort(key=lambda r: -r[5])
    for row in scored[:20]:
        print(f'{row[0]:30s} {row[1]:15s} maturity={row[2]} pages={row[3]} users={row[4]} score={row[5]:.1f}')

Step-by-step trace.

Component	Design choice	Reasoning
Rotation cadence	weekly	balance handoff overhead vs burnout
Handoff	Monday 10:00 local	fresh brains; not weekend
Follow-the-sun	US + EU pools	reduces overnight pages by 50%
Escalation matrix	Git-tracked YAML	one source of truth
Runbook priority	(users × pages) / maturity	quantitative backlog
Load tracking	off-hours pages per engineer	detect burnout early

After the design is deployed, the team has a documented rotation, a searchable escalation matrix, a scored runbook backlog, and a load-tracking dashboard. The next quarterly review adjusts the rotation pool as engineers join and leave, retargets the runbook backlog based on incident patterns, and reviews the load-tracking dashboard for engineers approaching burnout thresholds.

Output:

Metric	Value
On-call engineers	6 (rotation every 6 weeks)
Handoff cadence	weekly Monday
Follow-the-sun coverage	US + EU pool splits overnight
Runbook backlog target	100% pageable alerts at maturity 2+
Off-hours page threshold	10/week per engineer
Escalation matrix format	Git YAML

Why this works — concept by concept:

Weekly primary + secondary — the rotation shape that trades off handoff overhead (frequent enough) against burnout (not too frequent). Adding secondary decouples "primary is asleep / in a meeting" from "no-one is responding."
Follow-the-sun overnight coverage — a US-only rotation means every on-call gets some overnight pages. Splitting overnight with an EU pool cuts off-hours pages roughly in half — the single biggest on-call quality-of-life lever.
Escalation matrix as YAML — human-readable, Git-tracked, referenced by name from every runbook. One edit updates every alert; roles are role-based (@platform-lead) not person-based, so team changes don't cascade.
Runbook backlog prioritised by (users × pages) / maturity — the runbook you write next should be the one whose gap costs the most and whose maturity is lowest. This is a quantitative backlog ranker; without it, teams write runbooks for the wrong alerts.
Cost — 6 engineers × ~2 hours/week on-call meta-work (handoff + runbook maintenance) = ~12 engineer-hours/week to sustain the rotation. Compared to the "hero on-call" alternative (one senior engineer paged every incident, burns out in 6 months), this is the operationally sane approach that keeps teams together for years.

Design
Topic — design
Design problems on operational tooling

Practice →

Log processing Topic — log-processing Log-processing and alert-signal problems

Practice →

5. Post-incident review and reliability roadmap

Blameless postmortem → action items → reliability roadmap — the loop that compounds

The mental model in one line: a post-incident review is the structured, blameless meeting held within 5 business days of every SLO breach that produces a written timeline, an explicit list of contributing factors (not a single root cause), a set of tracked action items with owners and due dates, and an update to the reliability roadmap — and the roadmap itself is the quarterly plan that turns incidents into a compounding investment in SLI coverage, runbook maturity, and chaos-drill practice. Every senior data engineer must be able to describe the postmortem template, defend the "contributing factors, not root cause" preference, and explain how action items feed the reliability roadmap.

The postmortem template — one document per incident.

Header. Incident title, severity, start time, end time, MTTA (mean time to acknowledge), MTTM (mitigate), MTTR (resolve).
Impact. Which SLIs breached, budget consumed, downstream consumers affected, revenue impact (if any).
Timeline. Minute-by-minute log of what happened, ideally populated in real time from the comms template.
Contributing factors. A list — not one root cause. Every incident has 3–7 factors that together enabled it; ignoring the softer contributors misses future prevention.
What went well. Non-negotiable section. Reinforces successful mitigations; motivates the team.
What went badly. What we wish we'd done differently.
Action items. Concrete tickets, each with owner + due date + SLI impact.

Blameless — what it actually means.

Focus on systems, not people. "The on-call missed the alert" → "The alert routing was mis-configured and did not reach the on-call phone." The system is the target of improvement, not the human.
Assume rational actors. Everyone involved was doing their best with the information they had. If a decision looks bad in hindsight, ask "what information was missing?" not "who was careless?"
Language matters. Ban words like "should have" ("the on-call should have checked X"). Prefer "we observed that…" and "in future we can…"
Author is the incident owner. Not the on-call themselves — someone one step removed writes the postmortem to reduce self-blame framing.

"5 whys" vs "contributing factors" — the SRE preference.

5 whys. Sequential drill-down to a single root cause. Popular in classical incident response.
Why SRE prefers contributing factors. Real incidents have multiple simultaneous causes: an upstream schema change and a stale runbook and a missing alert and a confused escalation. Picking one "root" misses the other three; fixing only the root leaves the other three primed to re-fire.
The alternative. List 3–7 contributing factors; assign an action item per factor. Some may be "no action" (accepted risk), but all are named.
Example. Freshness breach: (1) upstream connector stalled on schema change, (2) burn-rate alert threshold too high to catch early, (3) runbook didn't include the schema-change check, (4) on-call was in a meeting and secondary was on PTO, (5) dashboard consumer wasn't notified for 45 min. Fix all five.

The reliability roadmap.

Cadence. Reviewed quarterly at platform-team offsite.
Inputs. Postmortem action items, SLI-coverage backlog, on-call load metrics, chaos-drill results.
Structure. One-quarter view (in-flight), next-quarter view (planned), 6-month view (aspirational).
Sizing. Each item sized as small (1 sprint), medium (2 sprints), large (a quarter). Aim for ~30% capacity on reliability work per engineer.
Owner. Platform-team lead; reviewed with engineering director.

Common interview probes on postmortems.

"What's in a postmortem template?" — required answer: header, impact, timeline, contributing factors, what went well, what went badly, action items.
"Why blameless?" — required answer: focus on systems not people; assume rational actors; language matters.
"Root cause vs contributing factors — which do you prefer?" — required answer: contributing factors; explain why single root misses the softer factors.
"How do action items become reliability improvements?" — required answer: feed the quarterly reliability roadmap.

Worked example — postmortem for a 2-hour freshness SLO breach

Detailed explanation. A concrete postmortem for a real-shape incident on analytics.orders. The freshness SLO (99.9% < 65 min) breached at 09:15 and did not recover until 11:12 — 117 minutes over the threshold, consuming ~117 min of the 43 min monthly budget (overspent by ~2.7x).

Root event. An upstream Postgres schema change added a discount_cents column; the Debezium connector required a schema-history sync that briefly stalled.
Compounding factors. Fast-burn alert threshold was too coarse; runbook did not include the schema-change check; on-call was in a meeting; consumer was not notified until minute 45.

Question. Write the postmortem for the incident using the standard template.

Input.

Field	Value
Incident	freshness breach on analytics.orders
Duration	117 min above threshold
Budget consumed	117 min (277% of month's total)
Severity	Sev-2 (business-hours; single dashboard affected)
Consumers affected	Sales Ops dashboard

Code.

# Postmortem — Freshness SLO breach on analytics.orders (2026-07-30)

## Header
- **Incident ID:** INC-2026-07-30-001
- **Severity:** Sev-2
- **Start:** 2026-07-30 09:15 UTC
- **End:** 2026-07-30 11:12 UTC
- **Duration:** 117 min above SLO threshold
- **MTTA:** 12 min (alert fired 09:15, acked 09:27)
- **MTTM:** 42 min (mitigation applied 09:57)
- **MTTR:** 117 min (fully recovered 11:12)

## Impact
- **SLIs breached:** freshness on analytics.orders
- **Budget consumed:** 117 min (277% of monthly 43-min budget; overspent by 74 min)
- **Downstream consumers:** Sales Ops dashboard (staleness visible to VP-Sales)
- **Revenue impact:** none directly; delayed sales-pipeline update by 2 h

## Timeline
- **09:12** Upstream deployed `ALTER TABLE production.orders ADD COLUMN discount_cents INT`
- **09:15** Debezium connector paused for schema-history sync
- **09:15** Freshness metric began rising above threshold
- **09:27** FreshnessBudgetFastBurn fired; on-call (Alice) acked (was in team meeting)
- **09:35** Alice opened runbook; ran Airflow + Postgres checks — nothing obvious
- **09:47** Alice noticed Debezium connector was paused
- **09:57** Alice restarted connector; freshness metric began recovering
- **10:00** Alice posted to #sales-ops (45 min after alert): "Data delay affecting orders dashboard"
- **10:45** Freshness reached below threshold intermittently; still burning budget
- **11:12** Freshness stable under threshold; incident resolved
- **11:30** Alice paged secondary + platform-lead to write postmortem

## Contributing factors (not root cause)
1. **Upstream schema change was not coordinated with the platform team.** The service team deployed the ALTER TABLE without notifying data platform. Downstream systems (including Debezium) require a heads-up for schema changes.
2. **Debezium connector required manual restart after schema-history sync.** The connector paused for ~40 min waiting for schema propagation; auto-restart on schema change is not configured.
3. **Runbook did not include the "check Debezium connector" step.** The on-call spent 20 min investigating Airflow and Postgres before checking the connector — this should have been step 1.
4. **On-call was in a team meeting when alert fired.** Meeting was internal (skippable), but the on-call did not have "leave meeting on page" muscle memory.
5. **Consumer (Sales Ops) was notified 45 min into the incident.** Comms template exists but was not invoked until after mitigation was underway.

## What went well
- Fast-burn alert fired within 12 min of SLI degradation — the alerting worked.
- Alice correctly identified Debezium as the cause once she looked at it.
- Mitigation (connector restart) was fast and clean once identified.
- Team switched to secondary/lead promptly to write the postmortem.

## What went badly
- MTTM (42 min) was too high because the runbook did not point at Debezium.
- Consumer comms delayed 45 min — dashboard viewers had no explanation for the stale data.
- Monthly error budget was consumed 2.7x in a single incident.

## Action items
| ID | Owner | Due | SLI impact |
|---|---|---|---|
| AI-1: Coordinate schema-change protocol with service teams (RFC + Confluence page) | @platform-lead | +14 d | prevents recurrence |
| AI-2: Add "check Debezium connector" as step 1 in freshness runbook | @alice | +3 d | reduces MTTM to ~15 min |
| AI-3: Configure Debezium auto-restart on schema-history sync | @bob | +21 d | eliminates cause |
| AI-4: Add "leave meetings on page" to on-call onboarding doc | @platform-lead | +7 d | reduces MTTA |
| AI-5: Add consumer-comms trigger to fast-burn alert (auto-Slack to dashboard channel) | @carol | +14 d | consumer comms lag → 0 |
| AI-6: Re-negotiate freshness SLO from 99.9% to 99.7% (achievable at current infra) | @platform-lead | +30 d | sustainable SLO |

## Reliability roadmap update
- Add "Debezium schema-change hardening" to Q3 platform roadmap (from AI-3).
- Add "schema-change protocol" to Q3 cross-team RFC list (from AI-1).
- Track AI-2, AI-4, AI-5 in weekly platform standup until closed.

Step-by-step explanation.

The header captures the four operational metrics — MTTA, MTTM, MTTR, duration — that every postmortem should quantify. These become the trend metrics the team watches over quarters ("MTTM is dropping — the runbooks are getting better").
The impact section names which SLIs breached and by how much. "Freshness on analytics.orders, 117 min over threshold, 277% of monthly budget." This is the language leadership understands; vague "the pipeline was slow" is uselessly abstract.
The timeline is populated from the comms template — the Slack messages posted during the incident become the raw timeline. This is why the comms template matters: it produces documentation as a byproduct of communication.
The contributing factors section lists 5 concrete factors — not one root cause. "The upstream schema change" is a factor, but "the runbook didn't include the check" and "the on-call was in a meeting" are equally load-bearing. Fixing only the schema-change misses the softer factors.
Every action item is a Jira ticket-shape: ID, owner, due date, SLI impact. This is what turns a postmortem from a document into an operational improvement. The postmortem author's job is to enforce that action items are entered into Jira before the postmortem meeting closes.

Output.

Postmortem section	Word count target	Purpose
Header	~50 words	quantitative facts
Impact	~100 words	consumer + budget framing
Timeline	~200 words	minute-by-minute
Contributing factors	~200 words	3–7 named factors
What went well	~100 words	motivation + reinforcement
What went badly	~100 words	honest self-critique
Action items	table of 5–10 rows	traceable improvement backlog

Rule of thumb. Every postmortem is authored by someone one step removed from the on-call, sized to fit on 2 pages, and finalised within 5 business days of the incident. Every action item is entered into Jira before the review meeting adjourns.

Worked example — action-item tracker with owner + due date + SLI impact

Detailed explanation. Action items from every postmortem land in a single tracker that the platform-team lead reviews weekly. The tracker feeds the quarterly reliability roadmap. Build the tracker.

Schema. ID, source postmortem, owner, due date, status, SLI impact, size.
Cadence. Reviewed at weekly platform standup.
Escalation. Overdue items surfaced to eng-manager at the +2 week mark.

Question. Design the action-item tracker table and the weekly-review query.

Input.

Component	Value
Storage	Jira project `RELIAB` with custom fields
Or	Postgres `reliability_action_items` table
Weekly review	Monday standup
Overdue escalation	+2 weeks past due date

Code.

-- reliability_action_items — tracker table
CREATE TABLE reliability_action_items (
    id              BIGSERIAL   PRIMARY KEY,
    postmortem_id   TEXT        NOT NULL,      -- e.g. INC-2026-07-30-001
    title           TEXT        NOT NULL,
    owner           TEXT        NOT NULL,       -- @-handle
    created_at      TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    due_date        DATE        NOT NULL,
    status          TEXT        NOT NULL DEFAULT 'open',   -- open|in-progress|done|dropped
    sli_impact      TEXT        NOT NULL,       -- 'prevents recurrence' | 'reduces MTTM by 30 min' | ...
    size            TEXT        NOT NULL,       -- 'small' | 'medium' | 'large'
    closed_at       TIMESTAMPTZ,
    dropped_reason  TEXT
);

CREATE INDEX idx_action_items_status_due
  ON reliability_action_items (status, due_date);

-- Weekly review query — what's open, what's overdue
SELECT
    id,
    postmortem_id,
    title,
    owner,
    due_date,
    (due_date - CURRENT_DATE) AS days_until_due,
    status,
    sli_impact,
    CASE
        WHEN status = 'done' THEN 'closed'
        WHEN due_date < CURRENT_DATE - INTERVAL '14 days' THEN 'ESCALATE'
        WHEN due_date < CURRENT_DATE THEN 'overdue'
        WHEN due_date < CURRENT_DATE + INTERVAL '7 days' THEN 'due soon'
        ELSE 'on-track'
    END AS review_status
FROM   reliability_action_items
WHERE  status IN ('open', 'in-progress')
ORDER  BY due_date ASC;

# Overdue-escalation cron — emails the eng-manager weekly
import psycopg2
import os
from email.message import EmailMessage
import smtplib

def escalate_overdue():
    conn = psycopg2.connect(os.environ['DB_DSN'])
    with conn.cursor() as cur:
        cur.execute("""
            SELECT id, postmortem_id, title, owner, due_date
            FROM   reliability_action_items
            WHERE  status IN ('open', 'in-progress')
              AND  due_date < CURRENT_DATE - INTERVAL '14 days'
        """)
        rows = cur.fetchall()

    if not rows:
        return

    body = 'The following reliability action items are >2 weeks overdue:\n\n'
    for aid, pm, title, owner, due in rows:
        body += f'  [{aid}] {title} · owner {owner} · due {due} · from {pm}\n'
    body += '\nPlease follow up in Monday standup.'

    msg = EmailMessage()
    msg['Subject'] = f'{len(rows)} reliability action items overdue >2w'
    msg['From']    = 'reliability-bot@internal'
    msg['To']      = 'eng-manager-data@internal'
    msg.set_content(body)

    with smtplib.SMTP('mail-relay:25') as s:
        s.send_message(msg)

if __name__ == '__main__':
    escalate_overdue()

Step-by-step explanation.

The tracker table stores one row per action item with the four fields that make follow-through possible: owner, due date, status, and SLI impact. Every item comes from a postmortem, so postmortem_id links back to the source incident document.
The weekly review query surfaces open items by due date. The review_status column translates raw dates into review categories: on-track / due soon / overdue / ESCALATE. The Monday standup walks the query results top-down.
The ESCALATE status (>2 weeks past due) triggers the escalation cron — an email to the eng-manager. This is the mechanism that prevents action items from silently rotting; overdue items get management visibility on a fixed cadence.
The sli_impact column forces the postmortem author to name why the action item matters. "Prevents recurrence" is stronger than "improves reliability"; "reduces MTTM by 30 min on freshness alerts" is stronger still. Vague impacts predict items that never get done.
The dropped_reason column exists for items that get explicitly abandoned. Sometimes a postmortem action turns out to be wrong ("we thought we needed to buy X, but we found a cheaper fix"). Recording why it was dropped preserves the reasoning for future postmortems.

Output.

Item ID	Title	Owner	Due	Status	Review
42	Add Debezium check to runbook	@alice	2026-08-02	in-progress	on-track
43	Configure Debezium auto-restart	@bob	2026-08-20	open	on-track
41	Schema-change protocol RFC	@platform-lead	2026-07-25	in-progress	overdue
38	Volume runbook v2	@carol	2026-07-12	open	ESCALATE

Rule of thumb. Every postmortem action item must have (owner, due date, SLI impact, size) fields populated before the postmortem meeting closes. Weekly review + 2-week auto-escalation is what keeps the tracker alive.

Worked example — quarterly reliability roadmap tied to SLO trend

Detailed explanation. The reliability roadmap is a quarterly plan that turns the action-item backlog into a prioritised sequence of platform investments. Build the Q3 roadmap for the data platform team.

Inputs. Open action items, SLO-trend data, on-call load, chaos-drill gaps.
Output. Prioritised list of ~10 items sized to consume 30% of team capacity.
Cadence. Reviewed at quarter-start offsite; revisited at mid-quarter checkpoint.

Question. Draft the Q3 2026 reliability roadmap for the data platform team.

Input.

Component	Value
Team size	6 engineers
Reliability capacity	30% = 1.8 engineer-quarters
Roadmap items	~10
Prioritisation input	SLO trend, incident frequency, on-call load

Code.

# Q3 2026 — Data Platform Reliability Roadmap

## Context
- Q2 saw 4 Sev-2 incidents; 2 were freshness-related on analytics.orders.
- SLO trend: freshness slipping from 99.94% → 99.87% quarter-over-quarter.
- On-call load: 2 engineers over 10 off-hours pages/week threshold.
- Q3 goal: hit 99.9% freshness on top-10 tables, reduce off-hours pages 30%.

## Q3 Investments (prioritised — 30% team capacity ≈ 1.8 engineer-quarters)

### Tier 1 — must ship this quarter (0.8 EQ)
1. **Debezium schema-change hardening** (@bob, medium, from AI-3 INC-2026-07-30-001)
   → SLI impact: eliminates freshness cliffs from upstream ALTER TABLE.
2. **Runbook coverage pass on top-10 tables** (@alice, medium)
   → SLI impact: raises runbook maturity from 1.4 avg to 2.5+.
3. **Consumer-comms auto-trigger on fast-burn alerts** (@carol, small, from AI-5)
   → SLI impact: consumer comms lag 45 min → 0.

### Tier 2 — plan to ship (0.6 EQ)
4. **Hot-standby DAG for orders pipeline** (@dan, large)
   → SLI impact: freshness recovery time p99 from 30 min to 5 min.
5. **Quarterly chaos drills — kill Debezium mid-hour** (@emily, small)
   → SLI impact: verifies runbooks + on-call muscle memory.
6. **On-call load rebalance — recruit 2 more into pool** (@platform-lead, medium)
   → SLI impact: off-hours pages per engineer 12/wk → 7/wk.

### Tier 3 — best-effort (0.4 EQ)
7. **Volume SLI on top-20 tables** (@frank, medium)
   → SLI coverage: 10 → 20 tables under volume SLO.
8. **Automated backfill on volume breach** (@bob, medium)
   → SLI impact: volume MTTR 90 min → 30 min.
9. **Postmortem author rotation** (@platform-lead, small)
   → team practice: spread postmortem-writing skill.

### Stretch (if capacity allows)
10. **Great Expectations integration for quality SLI** (@carol, large)
    → SLI coverage: quality SLI on all top-20 tables (currently 5).

## Success metrics for Q3
- Freshness SLO on top-10 tables: 99.87% → ≥ 99.9%.
- Off-hours pages per engineer: 12/wk → ≤ 8/wk median.
- Runbook coverage on pageable alerts: 60% → 100% at maturity 2+.
- Reliability action-item completion rate: 60% → 80% within due date.

## Mid-quarter checkpoint
- Week 6 offsite: review Tier 1 progress; re-prioritise Tier 2/3 if needed.

Step-by-step explanation.

The context section grounds the roadmap in quantitative Q2 data: incident count, SLO trend, on-call load. Prioritisation reasoning is transparent — a director reviewing the roadmap can see why Debezium hardening is Tier 1 (2 incidents in Q2, causing 74% of budget overspend).
Investments are sized in engineer-quarters (EQ) and grouped into three tiers by priority. Tier 1 (0.8 EQ) is roughly half a person's quarter — must-ship items where cutting is not an option. Tier 2 is planned; Tier 3 is best-effort. Stretch is opportunistic.
Every item names owner, size, source postmortem (where applicable), and SLI impact. The SLI-impact line is the "why this matters" that survives roadmap review. Items without a clear SLI impact are candidates for de-prioritisation.
The success metrics section commits to measurable outcomes for the quarter. "Freshness SLO 99.87% → 99.9%" is testable; "improve reliability" is not. These become the Q4-review inputs.
The mid-quarter checkpoint is non-negotiable — real quarters produce surprises (a new incident, a person leaving, a business priority shift). The checkpoint is when the roadmap is re-scoped honestly rather than silently over-committing.

Output.

Roadmap section	Items	EQ	Success metric
Tier 1 must-ship	3	0.8	freshness ≥ 99.9%, comms lag = 0
Tier 2 planned	3	0.6	recovery ≤ 5 min, pages ≤ 8/wk
Tier 3 best-effort	3	0.4	volume coverage → 20
Stretch	1	opportunistic	quality SLI on 20 tables

Rule of thumb. The reliability roadmap is quantitative (EQ, SLI impact, target metrics), time-bounded (one quarter, with a mid-quarter checkpoint), and traceable back to postmortems (every Tier 1 item cites a source incident). Roadmaps that lack these three properties become wish lists that never ship.

Senior interview question on postmortems

A senior interviewer might ask: "You had a 4-hour freshness SLO breach that consumed the entire monthly budget in one incident. Walk me through your postmortem process, the contributing factors you'd expect to find, the action items you'd extract, and how those action items become platform reliability improvements over the next quarter."

Solution Using a blameless postmortem with contributing-factors analysis, tracked action items, and quarterly-roadmap integration

# Postmortem — 4h Freshness SLO Breach (INC-2026-08-14-001)

## Header
- **Severity:** Sev-1 (revenue-affecting)
- **Duration:** 4 h 12 min above threshold
- **MTTA / MTTM / MTTR:** 8 min / 2 h 15 min / 4 h 12 min
- **Budget consumed:** 252 min (585% of monthly)

## Impact
- Freshness SLO on analytics.orders and analytics.line_items breached.
- Executive dashboard, ML fraud model, and reverse-ETL to Salesforce all
  showed stale data.
- Sales-ops meeting at 14:00 was held with stale numbers; revised the
  next morning.

## Contributing factors
1. **Postgres primary failed over unexpectedly.** Hardware issue on the
   AZ; failover to replica took 90 s but Debezium slot did not follow.
2. **Debezium reconnect logic did not handle failover cleanly.** Slot on
   the new primary did not exist; connector entered CrashLoopBackoff.
3. **Fast-burn alert fired quickly (8 min MTTA) but runbook did not
   include the "failover" case.** On-call spent 90 min investigating
   other causes.
4. **Debezium slot re-creation required schema-registry manual sync.**
   Documented procedure existed but was in an unfamiliar Confluence page.
5. **Consumer notifications were manual and delayed.** Reverse-ETL owner
   found out via customer complaint, not our comms.

## What went well
- MTTA of 8 min shows the fast-burn alert is properly tuned.
- Team collaboration was fast — 3 engineers on the call within 30 min.
- Root cause identified correctly once on-call opened the connector logs.

## What went badly
- MTTM of 2h 15m was way too high; the runbook missed the failover case.
- Consumer comms delayed; reverse-ETL owner learned via customer.
- 252 min of budget consumed = 5.85× monthly allowance.

## Action items
| ID | Title | Owner | Due | Size |
|---|---|---|---|---|
| AI-1 | Add "check for Postgres failover" as detection step 1 in freshness runbook | @primary-oncall | +3d | small |
| AI-2 | Configure Debezium slot re-creation on connect failure | @platform-lead | +21d | medium |
| AI-3 | Move slot re-creation procedure from Confluence into runbook itself | @primary-oncall | +7d | small |
| AI-4 | Auto-Slack the consumer channel when fast-burn fires | @carol | +14d | small |
| AI-5 | Add "Postgres failover" scenario to Q3 chaos drills | @emily | +30d | medium |
| AI-6 | Post-mortem: renegotiate freshness SLO given failover risk | @platform-lead | +45d | small |

## Roadmap update
- Elevate AI-2 to Q3 Tier 1 (was Tier 2).
- Move AI-5 chaos-drill to Q3 Tier 1 (was Tier 2).
- Adjust Q3 success metric: freshness p99 recovery ≤ 15 min (was 30 min).

-- Reliability roadmap-update trigger: any Sev-1 auto-elevates related items
UPDATE reliability_action_items
SET    tier = 'Tier 1'
WHERE  postmortem_id = 'INC-2026-08-14-001'
  AND  tier IN ('Tier 2', 'Tier 3');

-- Auto-notify the roadmap owner
NOTIFY roadmap_update, 'INC-2026-08-14-001 auto-elevated to Tier 1';

Step-by-step trace.

Step	Value	Reasoning
Postmortem written	within 5 business days	fresh memory
Contributing factors	5 named	avoids single-root reductionism
Action items	6 with owner + due date	traceable improvement
Roadmap elevation	AI-2, AI-5 → Q3 Tier 1	Sev-1 forces re-prioritisation
SLO renegotiation	tabled as AI-6	data-driven, not reactive

After the postmortem, the platform team has 6 concrete tracked items, two of which have been elevated to the current-quarter must-ship tier. The Q3 roadmap is republished with the updated priorities; the SLO catalogue is scheduled for renegotiation in 45 days when AI-2 (Debezium hardening) is complete and we have new failover-recovery data.

Output:

Metric	Value
Postmortem authored	day +3
Contributing factors identified	5
Action items opened	6
Roadmap items elevated	2
SLO renegotiation scheduled	day +45
Q3 goal adjusted	recovery p99 30 → 15 min

Why this works — concept by concept:

Blameless framing — no individual is named as at-fault. The contributing factors are all system properties (missing runbook step, missing Debezium logic, missing comms automation). This preserves psychological safety while producing sharper improvements.
Contributing factors, not root cause — the Postgres failover is a factor, but the Debezium reconnect logic, the missing runbook step, the buried procedure, and the manual comms are equally load-bearing. Fixing all five is what prevents the next 4-hour breach.
Action items with owner + due date — every factor becomes a concrete ticket. AI-1 and AI-3 are small (3–7 days); AI-2 and AI-5 are medium (2–4 weeks). The distribution keeps the postmortem operational: small items ship fast; medium items feed the roadmap.
Roadmap auto-elevation — the postmortem doesn't just recommend roadmap changes; it executes them via the SQL update. This is the operational mechanism that prevents postmortem findings from being ignored between quarterly offsites.
Cost — one postmortem meeting (~90 min), one 2-page document, 6 Jira tickets, one roadmap re-shuffle. Compared to the alternative — quiet Slack apology, informal Slack thread, no tracked improvements — this is what turns a bad incident into a compounding reliability investment over quarters.

Design
Topic — design
Design problems on reliability engineering

Practice →

API integration
Topic — api-integration
API-integration and comms-flow problems

Practice →

Cheat sheet — Data SRE recipes

The four axes as the SLI menu. Every pipeline gets up to four SLIs: freshness (now() - MAX(ingested_at) on the target table), completeness (target rowcount / source rowcount per hour), volume (target rowcount vs 7-day median band ±30%), quality (fraction of rows passing schema + constraint checks). Freshness is per-minute; completeness and volume are per-hour; quality is per-daily-run. Skip any axis that doesn't have a downstream consumer who cares — SLIs cost engineer-time to maintain.
Freshness SLI PromQL. data_freshness_meets_slo{table="..."} — 1 if within threshold, 0 otherwise. Roll up: sum_over_time(...[28d]) / count_over_time(...[28d]). Budget remaining: (current_slo − target) / (1 − target). Show the last one as a big-bold-number Grafana panel per top-10 table.
Completeness SLI SQL. WITH src AS (SELECT hour, COUNT(*) n_src FROM source GROUP BY hour), tgt AS (SELECT hour, COUNT(*) n_tgt FROM target GROUP BY hour) SELECT hour, n_tgt::float / n_src FROM src LEFT JOIN tgt USING (hour) WHERE hour < now() - INTERVAL '5 min'; — persist per-hour to slo_measurements.
Error-budget monthly template. Total = (1 − slo_target) × observation_windows_per_month. For per-minute observations: 99% = 432 min, 99.5% = 216 min, 99.9% = 43 min, 99.95% = 21.6 min, 99.99% = 4.3 min. Track remaining as total − spent. Show as a fuel-tank icon on the reliability dashboard; halt releases at 0.
Multi-window multi-burn-rate alert. Fast burn: (1 - avg_over_time(sli[1h])) > 14 × (1 - slo_target) AND (1 - avg_over_time(sli[5m])) > 14 × (1 - slo_target) for 2m → page. Slow burn: [6h] > 6× AND [30m] > 6× for 15m → ticket. Both wired against the same SLI; both link to the same runbook.
5-field runbook template. ## Symptom (2 sentences), ## Detection (3–5 commands or dashboards), ## Mitigation (ordered try-list with success criteria), ## Escalation (time-bounded tiers with @-handles), ## Owner (team + last-reviewed date). One screen max. No runbook link = alert doesn't ship.
On-call rotation shape. Weekly primary + secondary; handoff Monday 10:00 local (incoming); minimum 4 weeks between shifts; follow-the-sun with EU/US pools for overnight coverage. Compensate with one day off per week on-call plus stipend. Track pages per engineer per week; escalate to eng-manager if any engineer exceeds 10 off-hours pages weekly.
Escalation matrix as YAML. One escalation_matrix.yml in Git; per-service tier 1/2/3/4 as @-handles. Every runbook references by name (matrix.orders.tier_1). Change once, propagates everywhere. Role-based (@platform-lead) not person-based to survive team changes.
Comms template. Fill-in-the-blank Slack post: {sev}, {start}, {impact}, {status}, {next_update}, {on_call}. Timeline populated real-time (t+00:00 alert fired, t+00:04 acked, ...). Consumer notification table forces the on-call to acknowledge downstream consumers exist.
Postmortem sections. Header (MTTA/MTTM/MTTR), Impact (SLIs + budget + consumers), Timeline (from comms template), Contributing factors (3–7, not one root), What went well, What went badly, Action items (owner + due + SLI impact + size). 2 pages; finalised within 5 business days.
Contributing factors, not root cause. Real incidents have multiple simultaneous causes. Listing all 3–7 and assigning an action per factor catches the "soft" contributors (missing runbook step, missed comms) that a single-root-cause analysis silently ignores.
Action-item tracker fields. ID, postmortem_id, title, owner, due_date, status, sli_impact, size (small/medium/large). Weekly standup review; auto-escalate to eng-manager at +2 weeks past due. Track completion rate; aim for >80% within due date.
Reliability roadmap. Quarterly, sized in engineer-quarters, prioritised Tier 1/2/3/stretch. Tier 1 ≈ 40% of reliability capacity (must-ship); Tier 2 ≈ 35% (planned); Tier 3 ≈ 25% (best-effort). Mid-quarter checkpoint mandatory. Every Tier 1 item cites a source postmortem.
SLO renegotiation trigger. If the current 28-day SLO is below target for three consecutive months and the platform team has already shipped two Tier 1 reliability fixes without moving the metric, renegotiate the SLO downward. The alternative — keeping an aspirational SLO the team cannot hit — destroys the credibility of every SLO in the catalogue.

Frequently asked questions

What is a data SLO in one sentence?

A data SLO (service-level objective) is an explicit, agreed-upon target for a data SLI (service-level indicator) computed over a rolling measurement window — for example, "99.9% of one-minute freshness observations on analytics.orders have MAX(ingested_at) within 65 minutes of now() over the trailing 28 days." The SLO is a contract between the platform team producing the dataset and the downstream consumers depending on it; the SLO percentage translates to an error budget (1 − target) that governs the operational trade-off between shipping features and investing in reliability. Every senior data-engineering interview probes SLOs because they are the load-bearing mechanism that turns "we monitor pipelines" into "we own user-visible reliability."

SLI vs SLO vs error budget — what's the difference?

An SLI (service-level indicator) is the measurement — a numeric fact about the pipeline's output artifact, computed on the dataset rather than on the job that produced it. Examples: "age in seconds of the newest row in analytics.orders," "target rowcount / source rowcount for the last hour," "fraction of rows passing dbt tests today." An SLO (service-level objective) is the target expressed as a percentage of measurement windows in which the SLI meets a threshold — for example, "99.9% of one-minute windows have freshness under 65 minutes." An error budget is the mathematical complement of the SLO: 1 − SLO, expressed in minutes or days per month. A 99.9% freshness SLO is a 43-minute monthly error budget. The three concepts stack: SLI → SLO → error budget → burn-rate alerts → release-halt policy. Getting each definition right is the first thing senior interviewers probe.

How do I pick freshness targets for my pipelines?

Derive the target from the downstream consumer who cares most, then sanity-check against current observed performance. Start with an interview: sit with the dashboard owner or service owner who consumes the dataset and ask "how stale is too stale?" Their answer is usually anchored to a business cadence ("I check it every 5 minutes; more than an hour behind and my meeting is derailed") — that becomes your freshness target with a small buffer (add ~5 minutes above the natural cadence). Then compute the SLI over the last 28 days: if current freshness is 99.7% and the consumer wants 99.9%, the gap is a 3× improvement in bad-minute count — that's a project, not a target. If the gap is 10× or larger, either invest in specific reliability improvements first, or renegotiate the target down to something achievable. Publish the SLO in a Git-tracked catalogue with the consumer's name against it; review quarterly. The wrong pattern is to unilaterally invent a 99.99% target because it "sounds professional" — an aspirational SLO no team can hit destroys the credibility of every SLO in your catalogue.

What does a data on-call rotation actually do?

A data on-call rotation is the documented schedule of who receives pages when a data SLI breaches, structured around three anchors: rotation shape (primary + secondary weekly cadence with follow-the-sun handoff for global teams), escalation ladder (tier 1 primary in 5 min → tier 2 secondary+lead in 15 min → tier 3 eng-manager+SRE in 30 min → tier 4 VP in 60 min for revenue-affecting incidents), and linked runbooks (every alert points to a 5-field document: symptom, detection, mitigation, escalation, owner). The on-call's job during a shift is to acknowledge pages within 5 minutes, open the linked runbook, execute the detection and mitigation steps, escalate on the defined timeline if unresolved, communicate to affected consumers using the standard comms template, and author (or receive) the postmortem within 5 business days if an SLO is breached. Between shifts, the on-call attends the Monday handoff standup, contributes to runbook updates from the last week's incidents, and participates in the quarterly reliability-roadmap review. Rotation should recur no more often than every 4 weeks per engineer to prevent burnout; follow-the-sun EU/US splits cut off-hours pages roughly in half — the single biggest quality-of-life lever available.

What is a burn-rate alert and why do I need multi-window?

A burn-rate alert fires when an SLI is spending its error budget faster than the sustainable rate that would exhaust the budget over the full SLO window. For a 99.9% freshness SLO with 43 minutes of monthly budget, the sustainable burn rate is 1× (1.43 min/day). A burn rate of 14× means the whole month's budget would be gone in about 2 days — that's the fast-burn threshold that pages the on-call immediately. A burn rate of 6× means the budget is gone in about 5 days — the slow-burn threshold that files a ticket. Multi-window multi-burn-rate (from the Google SRE workbook) computes burn on both a short window (typically 1 hour) and a long window (typically 6 hours), and alerts only when both windows agree the burn is high — this catches cliff-falls quickly via the short window while catching creeping degradation via the long window, and it prevents old resolved spikes from re-firing as they age out of a single long window. Without multi-window burn-rate, teams either over-alert on transient spikes (alert fatigue, false-page cost) or under-alert on slow creeping degradations (silent SLO breaches, discovered from consumer complaints). Multi-window is the standard SRE pattern; naming it in a senior interview is a strong signal.

How do I run a blameless postmortem for a data incident?

A blameless postmortem is a structured written review of an incident, authored within 5 business days by someone one step removed from the on-call, following a template that captures impact (which SLIs breached, budget consumed, downstream consumers affected), timeline (populated in real-time from the incident comms template — t+00:00 alert fired, t+00:04 acked, etc.), contributing factors (a list of 3–7 named factors, not a single root cause — a schema change and a missing runbook step and a delayed comms and an on-call in a meeting are all load-bearing), what went well (reinforces successful mitigations), what went badly (honest self-critique), and action items (each with owner, due date, SLI impact, and size — small, medium, or large). The blameless discipline is enforced by focusing every finding on system properties rather than individual people ("the alert routing didn't reach on-call" not "the on-call missed the alert"), banning "should have" language, and assuming rational actors doing their best with available information. Action items become Jira tickets before the postmortem meeting closes; the tracker is reviewed weekly; overdue items auto-escalate to the eng-manager at +2 weeks; and the reliability roadmap absorbs the medium/large items into the next quarter's plan. The whole loop — detect → mitigate → resolve → review → improve → roadmap — is what turns individual incidents into a compounding reliability investment over quarters.

Practice on PipeCode

Drill the ETL practice library → for pipeline observability, incremental-load, and reconcile problems senior interviewers love.
Rehearse on the design practice library → for the reliability, on-call, and platform-architecture scenarios that show up in senior data-platform loops.
Sharpen the reliability axis with the data-validation practice library → for the SLI-shaped quality and schema-check questions.
Add the time-series practice library → and the SQL time-series subset → for burn-rate, rolling-window, and freshness-metric SQL patterns.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis SLI framework against real graded inputs.

Own reliability like an SRE, not a task-runner

Docs explain SLOs. PipeCode drills explain the decision — how to pick freshness targets a consumer will actually sign, when a burn-rate alert should page vs ticket, how to write a runbook the on-call reads at 3 AM, and how to convert a postmortem into a quarter of reliability improvement. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice design problems →

Pytest for Data Engineering: Fixtures, Parametrization & Docker-Compose Integration Tests

Gowtham Potureddi — Fri, 31 Jul 2026 14:33:59 +0000

pytest for data engineering is the load-bearing test framework for every serious 2026 data platform — the one tool that lets a senior engineer wire dbt model contracts, Airflow DAG smoke tests, Spark transform property tests, and end-to-end Postgres+Kafka+MinIO integration suites into a single pytest invocation that runs on a laptop, on a Codespace, and on GitHub Actions with the exact same behaviour. Every ingestion pipeline you ship pushes rows through code paths whose failure modes (schema drift, null explosion, off-by-one window boundaries, transaction leakage) can only be caught by tests that exercise the real behaviour against a real database — and pytest's fixture model, parametrization primitives, and testcontainers ecosystem are what turn that from a Herculean manual chore into a pytest -n auto one-liner.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through your pytest fixtures scope strategy for a suite with 300 DB-touching tests," or "how do you pytest parametrize a Spark UDF across the null / edge / valid axis without exploding your test count," or "your integration suite spins up pytest docker-compose per-worker under pytest-xdist — how do you keep the Postgres port from colliding?" It walks through the four canonical test layers — unit (pure Python transforms), contract (dbt/Spark schema + not-null assertions), component-with-DB (pytest-postgresql transactional fixtures), and integration-with-compose (pytest testcontainers running Postgres + Kafka + MinIO) — the four axes senior interviewers actually probe (fixture scope, parametrization discipline, container hygiene, CI reproducibility), the canonical config for each, and the CI wiring that keeps a 1000-test data-engineering suite under three minutes on GitHub Actions. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the Python practice library →, rehearse on the ETL practice library →, and sharpen the validation axis with the data-validation practice library →.

On this page

Why pytest is the load-bearing test framework for data engineering in 2026
Fixtures — session scope, factory patterns, DB seed lifecycles
Parametrization — table-driven tests over dbt/Spark transforms
Docker-Compose + Testcontainers integration tests
CI wiring — coverage, xdist parallelism, flaky-test discipline
Cheat sheet — pytest recipes for data engineering
Frequently asked questions
Practice on PipeCode

1. Why pytest is the load-bearing test framework for data engineering in 2026

Four test layers, one framework — why pytest wins the DE stack over unittest, nose, and hand-rolled runners

The one-sentence invariant: pytest for data engineering is the framework where every meaningful test layer (unit transforms, dbt/Spark schema contracts, component tests against a real Postgres, and end-to-end integration suites spun up via pytest docker-compose or pytest testcontainers) shares one runner, one fixture graph, one parametrization primitive, and one CI story — which is what makes a 1000-test data-engineering suite feasible on a laptop, in a Codespace, and on GitHub Actions with identical behaviour. The choice of pytest over unittest, nose2, or Bazel-native runners is not aesthetic; it is structural. Fixtures compose. Parametrization is declarative. Plugins (pytest-xdist, pytest-cov, pytest-postgresql, pytest-mock, pytest-rerunfailures) plug in without ceremony. Every other framework in the Python ecosystem has receded, and in 2026 there is no serious data-platform team that does not run pytest as the entry point.

The four test layers every serious DE team ships.

Layer 1 — unit tests. Pure-Python transforms with no I/O — a normalize_email function, a parse_iso_timestamp helper, a chunk_iterable utility. Sub-millisecond per test; hundreds per file; run on every save. This is the base of the pyramid.
Layer 2 — contract tests. dbt schema tests (not_null, unique, accepted_values), Great Expectations suites, JSON-schema validators for API payloads, pydantic model validation. Fast; deterministic; catch schema drift before it reaches production.
Layer 3 — component tests with a real DB. SQL-emitting Python code executed against a real Postgres (via pytest-postgresql template DB) or DuckDB (in-process). Catch off-by-one window boundaries, transaction-leakage bugs, and index-usage regressions that unit tests cannot. ~100 ms per test.
Layer 4 — integration tests with docker-compose. Airflow scheduler + Postgres + Redis + a Kafka broker + MinIO, all spun up via docker-compose or testcontainers-python, then a DAG runs end-to-end. ~5–30 s per test; run on PR and nightly. Catches the failure modes that only appear when networked services interact.

The four axes senior interviewers probe.

Fixture scope discipline. Do you know when to use session (expensive setup like a container) versus module (per-file setup) versus function (per-test isolation)? Do you know that autouse=True at session scope is a footgun that couples every test to that fixture whether it needs it or not? This is the single most-probed pytest topic.
Parametrization hygiene. Do you know how to use ids= to keep parametrized test names readable? Do you know when to use indirect=True to run params through a fixture? Do you know that pytest_generate_tests lets you parametrize dynamically from a manifest? Senior teams reject test files where parametrized test IDs are test_transform[test_transform0] instead of test_transform[null-input].
Container hygiene. Do you spin one container per session (fast, shared state) or per-test (slow, hermetic)? Do you know the wait-for-ready health-check idiom? Do you clean up volumes at teardown? Do you know how to make containers survive pytest-xdist parallelism without port collisions? Senior signal: name testcontainers-python and the network= parameter unprompted.
CI reproducibility. Does the suite pass on your laptop, in Codespaces, and in GitHub Actions the same way — or does it flake on CI only? Do you have a coverage gate (--cov-fail-under)? Do you have flaky-test discipline (pytest-rerunfailures + quarantine + nightly retriage)? Do you know how to shard across xdist workers without breaking DB fixtures?

The 2026 reality — pytest + testcontainers is the default; the alternatives are legacy.

unittest ships in the stdlib and remains for legacy codebases, but nobody starts a greenfield DE project with it in 2026. The lack of fixture composition, the boilerplate setUp / tearDown, and the absence of parametrization make it a dead end for the kind of matrix-driven testing DE work demands.
nose / nose2 are historical curiosities; the maintainers have moved on. Any codebase still on nose should schedule a migration.
pytest + pytest-xdist + pytest-cov + pytest-postgresql (or testcontainers-python) is the canonical stack. Every dbt-adapter, every Airflow provider, every managed-Spark toolkit publishes fixtures and helpers targeting this stack.
Bazel / pants / native build runners wrap pytest rather than replacing it — even in monorepos, the actual test execution goes through pytest.

What interviewers listen for.

Do you name all four test layers without prompting? — senior signal.
Do you say "session-scope for the expensive container, function-scope for the test data" in the first sentence when scope comes up? — required answer.
Do you push back on "just use unittest.mock everywhere" with the argument that "mocks don't catch schema drift; component tests against a real Postgres do"? — senior signal.
Do you name testcontainers-python (or pytest-docker-compose) as the integration primitive rather than "we shell out to docker-compose"? — senior signal.
Do you describe pytest as "a fixture graph plus a parametrization matrix" rather than as "a test framework"? — required answer.

Worked example — the four-layer test pyramid for a dbt + Airflow + Spark platform

Detailed explanation. The single most useful artifact for a pytest-for-DE interview is a memorised four-layer pyramid table. Every senior test-strategy discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical retail data platform that runs dbt in Snowflake, Airflow on Kubernetes, and Spark on EMR.

Platform. dbt-core targeting Snowflake, Airflow 2.9 on Kubernetes, Spark 3.5 on EMR.
Test count budget. ~800 unit + ~300 contract + ~150 component + ~40 integration = ~1,290 tests total.
Speed budget. Unit ~10 s total, contract ~30 s, component ~2 min, integration ~5 min. Full suite under 8 minutes on CI.
Coverage target. ≥85% line coverage, ≥70% branch coverage on the Python packages.

Question. Build the four-layer test pyramid for the platform, name the pytest plugin for each layer, and pick the run-frequency.

Input.

Layer	Tests	Speed	Plugins	Frequency
Unit	800	~10 s	pytest, pytest-mock	on save
Contract	300	~30 s	pytest, pydantic, dbt-core	on save
Component	150	~2 min	pytest-postgresql, pytest-mock	on PR
Integration	40	~5 min	testcontainers-python, pytest-xdist	on PR + nightly

Code.

# tests/conftest.py — one file wires all four layers
import pytest
from pathlib import Path

# ------------------------------------------------------------------
# Layer 1 (unit) — no fixtures needed; pure functions
# ------------------------------------------------------------------


# ------------------------------------------------------------------
# Layer 2 (contract) — pydantic + manifest-driven parametrization
# ------------------------------------------------------------------
@pytest.fixture(scope="session")
def dbt_manifest():
    """Loaded once per session; used by parametrized contract tests."""
    import json
    manifest_path = Path("target/manifest.json")
    return json.loads(manifest_path.read_text())


# ------------------------------------------------------------------
# Layer 3 (component-with-DB) — pytest-postgresql template
# ------------------------------------------------------------------
from pytest_postgresql import factories

postgresql_proc = factories.postgresql_proc(port=None, unixsocketdir="/tmp")
postgresql = factories.postgresql("postgresql_proc", dbname="test_db")


# ------------------------------------------------------------------
# Layer 4 (integration) — testcontainers-python
# ------------------------------------------------------------------
@pytest.fixture(scope="session")
def compose_stack():
    """Spin up Postgres + Kafka + MinIO once per session."""
    from testcontainers.compose import DockerCompose
    with DockerCompose(".", compose_file_name="docker-compose.test.yml") as compose:
        compose.wait_for("http://localhost:9001/minio/health/ready")
        yield compose

Step-by-step explanation.

Layer 1 (unit) needs no fixtures — pure Python transforms are called directly with in-memory args. This is the fastest layer and where the bulk of the tests live. Sub-millisecond per test.
Layer 2 (contract) uses one session-scoped dbt_manifest fixture loaded once, then parametrized tests iterate every model in the manifest. This gives you 300 tests for the cost of ~30 lines of code — the parametrization amplifier is what makes contract testing tractable.
Layer 3 (component) uses pytest-postgresql's postgresql_proc (session-scoped process fixture that starts one Postgres binary) and postgresql (function-scoped database that gets a fresh template-cloned DB per test). This gives per-test DB isolation at the cost of ~100 ms per test — template-DB cloning is what makes the pattern fast.
Layer 4 (integration) uses testcontainers-python's DockerCompose fixture at session scope — the compose stack starts once per pytest run, and every integration test shares it. The wait_for health check is non-optional; without it, tests race the container startup and flake.
The four layers are declared in a single conftest.py at the repo root and automatically discovered by pytest — you don't import them, you don't wire them; they just appear in every test file that names them. This is fixture-graph magic.

Output.

Layer	Fixture scope	Per-test cost	Isolation
Unit	none	<1 ms	pure functions
Contract	session (manifest)	~1 ms per model	in-memory
Component	session (proc) + function (db)	~100 ms	fresh DB per test
Integration	session (compose)	~5–30 s	shared containers, isolated schemas

Rule of thumb. Never mix scopes across layers by accident. If a component test starts to depend on a session-scoped compose fixture, you have promoted it to Layer 4 — accept the ~5 s per-test cost or refactor the dependency out.

Worked example — what interviewers actually probe about pytest for DE

Detailed explanation. The senior data-engineering pytest interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you test a dbt project?"), then progressively narrows to test whether you know the axes. The candidates who name the four layers in sentence one score highest; the candidates who describe "we write unit tests with mocks for everything" score lowest. Walk through the interview grading rubric.

Ambiguous opener. "How would you build a test suite for our dbt + Airflow + Spark platform?" — invites you to name the four layers.
Follow-up 1. "How do you make the Postgres fixture fast enough for 150 tests?" — probes fixture scope + template-DB knowledge.
Follow-up 2. "How do you parametrize a schema-contract test over every table in the schema?" — probes pytest_generate_tests.
Follow-up 3. "How do you run the suite in parallel without Postgres fixtures colliding?" — probes xdist + port allocation.
Follow-up 4. "One test in the integration suite flakes 5% of the time. What's your discipline?" — probes rerunfailures + quarantine.

Question. Draft a 5-minute senior pytest-for-DE answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Layers named	"we write pytest tests"	"unit, contract, component-with-DB, integration-with-compose"
Fixture scope	"we use setUp"	"session for containers, function for data, template-DB clone per test"
Parametrize discipline	"we loop inside the test"	"@pytest.mark.parametrize with ids=; pytest_generate_tests for dynamic"
Container story	"we shell out to docker-compose"	"testcontainers-python with wait_for health checks; --xdist-group by port"
Flaky discipline	"we rerun the pipeline"	"pytest-rerunfailures with reruns=2; quarantine marker; nightly retriage"

Code.

Senior pytest-for-DE answer template (5 minutes)
================================================

Minute 1 — name the four layers up front
  "I'd build four layers: unit for pure transforms, contract for
   dbt/pydantic schema, component-with-DB via pytest-postgresql for
   SQL-emitting code, and integration-with-compose via
   testcontainers-python for the end-to-end Airflow DAG."

Minute 2 — fixture scope discipline
  "Session-scoped fixtures for the expensive things — the Postgres
   process, the compose stack, the dbt manifest. Function-scoped
   fixtures for the isolation-critical things — the DB itself
   (template-cloned per test), the tmp_path, the monkeypatched env."

Minute 3 — parametrization discipline
  "Every contract test uses @pytest.mark.parametrize with explicit
   ids= so test names read like test_not_null[orders.customer_id]. For
   dynamic axes — every model in the dbt manifest — I use
   pytest_generate_tests to expand at collection time."

Minute 4 — container hygiene
  "testcontainers-python for integration; DockerCompose fixture at
   session scope with a wait_for health-check. Under xdist, I group
   by --xdist-group so all tests hitting the same container land on
   the same worker; container port is randomised via published_port=0."

Minute 5 — CI reproducibility + flaky discipline
  "GitHub Actions matrix on 3.11/3.12; pytest -n auto with
   --cov-fail-under=85; --junit-xml uploaded to a flaky-test dashboard.
   Flaky tests get @pytest.mark.flaky(reruns=2) plus a quarantine
   label; a nightly job retriages them so nothing stays flaky forever."

Step-by-step explanation.

Minute 1 is the crucial framing. Naming the four layers immediately — "unit, contract, component-with-DB, integration-with-compose" — signals you're a strategy-shaper, not a task-runner. Weak candidates dive into tools ("we'd use pytest-mock and…") before naming the layers.
Minute 2 addresses fixture scope before the interviewer asks. This preempts the common trap where you commit to "all fixtures at function scope" and then have to defend a 40-minute test run. Naming session/module/function up front is senior signal.
Minute 3 is the parametrization probe. Every senior DE test suite is 80% parametrized cases; the fluent answer names ids=, indirect=True, and pytest_generate_tests unprompted.
Minute 4 is the container hygiene question. Naming testcontainers-python and the wait_for health-check pattern shows you've built the pattern, not read about it. Naming --xdist-group shows you've run it under parallel CI.
Minute 5 covers CI and flakes — the reliability axis. The rerunfailures + quarantine + nightly retriage story shows you've inherited a flaky suite and rescued it, which is the operational credential senior teams hire for.

Output.

Grading criterion	Weak score	Senior score
Names four layers in minute 1	rare	mandatory
Names fixture scope discipline	rare	required
Names parametrize + ids	occasional	mandatory
Names testcontainers-python	rare	senior signal
Names flaky discipline	rare	senior signal

Rule of thumb. The senior pytest-for-DE answer is a 5-minute monologue that covers all four axes without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the test layer" decision tree

Detailed explanation. Given a new piece of code, the senior engineer runs a 4-question decision tree to place it on the right test layer. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a code snippet and you can walk the tree out loud. Walk through the tree with three canonical inputs: a pure normalize_phone function, a SQL-emitting find_orders_in_window helper, and an Airflow DAG that reads from Postgres and writes to S3.

Q1. Does the code touch any I/O (network, disk, DB)? → no = Layer 1 unit; yes = go to Q2.
Q2. Is the I/O purely SQL against one database? → yes = Layer 3 component with pytest-postgresql; no = go to Q3.
Q3. Does the code coordinate multiple services (broker, DB, storage)? → yes = Layer 4 integration with compose; no = mock the one external dependency with pytest-mock and stay at Layer 1.
Q4 (parallel branch). Is the code a schema definition (dbt model, pydantic model, JSON schema)? → yes = Layer 2 contract with parametrized assertion; N/A to Q1.

Question. Walk the decision tree for the three code snippets and record the layer each ends up on.

Input.

Code	Q1 (I/O?)	Q2 (SQL only?)	Q3 (multi-service?)	Layer
normalize_phone	no	—	—	Layer 1 unit
find_orders_in_window	yes	yes	—	Layer 3 component
ingest_daily_orders DAG	yes	no	yes	Layer 4 integration

Code.

# Decision-tree helper (illustrative)
def pick_test_layer(touches_io: bool,
                     is_sql_only: bool,
                     multi_service: bool,
                     is_schema_def: bool) -> str:
    """Return the pytest layer for a code artifact."""
    if is_schema_def:
        return "Layer 2 contract"
    if not touches_io:
        return "Layer 1 unit"
    if is_sql_only:
        return "Layer 3 component"
    if multi_service:
        return "Layer 4 integration"
    return "Layer 1 unit (mock the one external)"


# Walk the three code snippets
print(pick_test_layer(False, False, False, False))
# -> Layer 1 unit

print(pick_test_layer(True,  True,  False, False))
# -> Layer 3 component

print(pick_test_layer(True,  False, True,  False))
# -> Layer 4 integration

print(pick_test_layer(False, False, False, True))
# -> Layer 2 contract

Step-by-step explanation.

Scenario 1 — normalize_phone(raw: str) -> str is a pure function with no I/O. The tree short-circuits at Q1 → Layer 1 unit. Sub-millisecond, hundreds of these, run on save.
Scenario 2 — find_orders_in_window(conn, start, end) executes SQL against Postgres and returns rows. Q1 = yes, Q2 = yes → Layer 3 component with pytest-postgresql. Per-test template DB clone; ~100 ms; catches window-boundary bugs unit tests cannot.
Scenario 3 — ingest_daily_orders_dag reads from Postgres, writes JSONL to MinIO, publishes a completion event to Kafka. Q1 = yes, Q2 = no, Q3 = yes → Layer 4 integration with testcontainers-python. Session-scoped compose; ~10 s; catches inter-service failures.
Scenario 4 — a pydantic.BaseModel for the API payload → Layer 2 contract. Parametrized test_payload_shape[orders-good], test_payload_shape[orders-missing-total], etc. Fast; deterministic.
If the code fails Q1-Q3 but interacts with exactly one external dependency (say, an HTTP API), mock the dependency with pytest-mock's mocker.patch(...) and keep it at Layer 1. This preserves the pyramid shape — thousands of unit tests, dozens of integration tests, not the inverted pyramid that kills CI budgets.

Output.

Code	Layer	Fixture needed	Speed
normalize_phone	Layer 1 unit	none	<1 ms
find_orders_in_window	Layer 3 component	postgresql (function-scope)	~100 ms
ingest_daily_orders_dag	Layer 4 integration	compose_stack (session-scope)	~10 s
OrdersPayload (pydantic)	Layer 2 contract	none	<1 ms

Rule of thumb. The four-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any code snippet and get a layer name in under 30 seconds.

Senior interview question on pytest test-layer strategy

A senior interviewer often opens with: "You inherit a data platform with 900 tests, most of them Layer 1 unit tests with unittest.mock mocking Postgres, and a CI run that takes 25 minutes because the 40 integration tests each spin up their own docker-compose. Walk me through the test-layer strategy you'd migrate to, the fixture-scope changes, and the CI wiring you'd expect to bring the run under 8 minutes."

Solution Using a four-layer pytest pyramid with session-scoped compose + template-DB fixtures + xdist parallelism

# 1. tests/conftest.py — one fixture graph for all four layers
import pytest
from pathlib import Path
from pytest_postgresql import factories

# Layer 3 — pytest-postgresql: one process per session, fresh DB per test
postgresql_proc = factories.postgresql_proc(
    port=None,                      # OS-allocated port (xdist-safe)
    unixsocketdir="/tmp",
    load=[Path("schema.sql").read_text()],  # template DB is pre-seeded
)
postgresql = factories.postgresql(
    "postgresql_proc",
    dbname="test_db",
    load=[],                        # per-test DB is a template clone
)


# Layer 4 — testcontainers: one compose stack per session
@pytest.fixture(scope="session")
def compose_stack():
    from testcontainers.compose import DockerCompose
    with DockerCompose(
        ".",
        compose_file_name="docker-compose.test.yml",
        pull=True,
    ) as compose:
        compose.wait_for("http://localhost:9001/minio/health/ready")
        # Also wait for Postgres + Kafka
        compose.wait_for("http://localhost:9092/kafka-health")
        yield compose

# 2. tests/unit/test_normalize.py — Layer 1
import pytest
from mypkg.transforms import normalize_phone

@pytest.mark.parametrize("raw,expected", [
    ("+1 415-555-1212", "+14155551212"),
    ("415.555.1212",     "+14155551212"),
    ("(415) 555-1212",   "+14155551212"),
], ids=["plus-space", "dot-sep", "paren-sep"])
def test_normalize_phone(raw, expected):
    assert normalize_phone(raw) == expected

# 3. tests/component/test_find_orders.py — Layer 3
import pytest
from mypkg.orders import find_orders_in_window

def test_find_orders_in_window_inclusive_start(postgresql):
    """Window is [start, end) — start is inclusive, end is exclusive."""
    with postgresql.cursor() as cur:
        cur.execute("INSERT INTO orders(id, created_at) VALUES (1, '2026-01-01 00:00:00')")
        cur.execute("INSERT INTO orders(id, created_at) VALUES (2, '2026-01-02 00:00:00')")

    rows = find_orders_in_window(
        postgresql,
        start="2026-01-01 00:00:00",
        end  ="2026-01-02 00:00:00",
    )
    assert [r.id for r in rows] == [1]   # id=1 is inside; id=2 is boundary-exclusive

# 4. tests/integration/test_dag_end_to_end.py — Layer 4
import pytest

@pytest.mark.integration
def test_ingest_daily_orders_dag_end_to_end(compose_stack):
    """Full DAG: read Postgres, write MinIO, publish Kafka completion."""
    from mypkg.dags.ingest import ingest_daily_orders_dag

    # (arrange test data in compose_stack.postgres; elided for brevity)
    result = ingest_daily_orders_dag.test()

    assert result.state == "success"
    # Assert MinIO object landed
    assert compose_stack.get_service_host("minio", 9000) is not None
    # Assert Kafka completion event
    # ...

# 5. .github/workflows/tests.yml — CI matrix
name: tests
on: [push, pull_request]
jobs:
  pytest:
    runs-on: ubuntu-24.04
    strategy:
      matrix:
        python: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ matrix.python }} }
      - run: pip install -e '.[test]'
      - run: |
          pytest \
            -n auto \
            --dist=loadgroup \
            --cov=mypkg \
            --cov-branch \
            --cov-fail-under=85 \
            --junit-xml=junit.xml \
            tests/
      - if: always()
        uses: actions/upload-artifact@v4
        with: { name: junit-${{ matrix.python }}, path: junit.xml }

Step-by-step trace.

Step	Before (mock-heavy, per-test compose)	After (four-layer pyramid, session compose + xdist)
Unit tests	900 heavy mocks	800 pure-function tests
Contract tests	0	300 parametrized schema tests
Component tests	0 (mocked)	150 pytest-postgresql tests
Integration tests	40 per-test compose	40 sharing one session compose
Postgres startup	none (all mocked)	one per session (~1 s)
Compose startup	40 × ~10 s = 6.7 min	1 × ~10 s
Parallelism	serial	xdist -n auto (8 workers)
Total CI time	~25 min	~7 min

After the migration, the suite shape has inverted from mock-heavy (unit tests full of mocker.patch("psycopg2.connect") that never caught the off-by-one window bug) to real-behaviour-heavy (component tests against a real Postgres that catch it every time), and the CI budget has dropped from 25 minutes to 7 minutes despite adding 450 tests. The pytest-xdist parallelism amortises the compose startup across all workers.

Output:

Metric	Before	After
Total tests	940	1,290
CI runtime	25 min	7 min
Postgres coverage	mocked (0% real)	100% real behaviour
Compose spin-ups	40 per run	1 per run
Flakes / week	~15	~1
Bug catch rate on PR	~40%	~85%

Why this works — concept by concept:

Four-layer pyramid — Unit / Contract / Component-with-DB / Integration-with-compose is the shape the industry converged on. Each layer catches a different class of bug at a different cost. Mixing scopes across layers by accident is the anti-pattern this shape prevents.
Session-scoped compose — The compose stack starts once per pytest run, shared across all Layer-4 tests. This is the ~40-minute-to-40-second win. The wait_for health check makes the fixture reliable.
Template-DB cloning — pytest-postgresql runs one Postgres process per session, but every test gets a fresh DB by cloning the template. Cloning is ~100 ms; starting a fresh Postgres is ~1 s. This is what makes 150 Layer-3 tests feasible.
xdist parallelism — -n auto spawns one worker per CPU. Session-scoped fixtures are per-worker, so each worker gets its own Postgres process and the compose stack is shared across workers via --dist=loadgroup. This is the ~7-minute-to-~2-minute latent win when hardware allows.
Cost — one Postgres binary (~50 MB), one compose stack (~2 GB with Postgres + Kafka + MinIO), coverage tooling (pytest-cov ~3% CPU overhead), and xdist (one worker per CPU). The eliminated cost is the 6.7 minutes of compose spin-ups and the false confidence of mock-heavy unit tests. Net O(1) startup per run versus O(N) startup per test.

Python
Topic — python
Python testing and pytest patterns

Practice →

ETL Topic — etl ETL problems on pipeline testing

Practice →

2. Fixtures — session scope, factory patterns, DB seed lifecycles

`@pytest.fixture` is a dependency-injection graph — and scope discipline is the difference between a 2-minute suite and a 20-minute one

The mental model in one line: pytest fixtures are a directed dependency-injection graph where every fixture declares its own construction cost via a scope (function / class / module / session) and its own teardown via yield or a finalizer, and the difference between a fast suite and a slow one is almost always a fixture that was silently promoted from session to function scope by a well-meaning refactor. Every senior data engineer has debugged the "why did the test suite go from 90 s to 12 min overnight" mystery, and roughly nine times out of ten the answer is that someone changed scope="session" to the default scope="function" on the Postgres process fixture, and now every one of 400 tests starts its own Postgres.

The four scopes for pytest fixtures.

function (default). Constructed and torn down for every test. Best for data that mutates during the test (a fresh DB row, a temp file, a monkeypatched env var). Cheapest to reason about; most expensive at scale.
class. Constructed once per test class, torn down after. Rarely used in modern DE codebases because pytest style discourages class TestFoo: grouping.
module. Constructed once per test file. Useful when a group of related tests share the same setup — e.g. all tests in test_orders_analytics.py reuse the same seeded orders dataset.
session. Constructed once per pytest run, torn down at exit. Mandatory for expensive setup: the Postgres process, the compose stack, the dbt manifest, an HTTP mock server. The rule of thumb: if setup takes >1 second, it belongs at session scope.

Yield-teardown vs finalizer — pick one style and stick to it.

yield-teardown. The modern, Pythonic style. yield the_thing at the point where the test receives the fixture; anything after the yield runs at teardown. Reads top-to-bottom like a contextmanager. This is what >95% of 2026 pytest code uses.
request.addfinalizer(fn). The older style. Register a callback that runs at teardown. Useful when teardown depends on runtime state that doesn't exist yet at yield time (e.g. you need to tear down N containers where N is determined mid-test).
autouse=True. The footgun. Marks a fixture as automatically active for every test in scope without being explicitly requested. Legitimate use: cross-cutting concerns like resetting a global logger, seeding a UTC timezone, patching datetime.now. Illegitimate use: silently coupling every test to a Postgres fixture — which then breaks every unit test that doesn't need a DB.

The factory-as-fixture pattern — the single most useful pattern for DE test data.

The problem. Tests need "an order row for customer X with N line items" — but every test needs a different shape. A single orders_seed fixture that returns a hardcoded row is either too specific (only works for one test) or too general (returns a dict every test has to reshape).
The pattern. The fixture returns a factory function that tests call with the shape they need. def make_order(customer_id=1, total_cents=1000, status='pending'): ... — every test calls make_order(status='shipped') or make_order(customer_id=99) and gets exactly what it needs.
Why it wins. Test data stays close to the test that needs it (readability), the factory centralises the DB-write logic (DRY), and the factory can produce multiple rows in one test without re-invoking the fixture.
Composability. Factories compose — make_shipment can call make_order internally. This is how you build hierarchical test data without ceremony.

The DB-seed lifecycle — three patterns, one right answer per use case.

Truncate-per-test. Fixture truncates every table at teardown. Slow (~50 ms per test); simple; works for small schemas. Suitable for <50 tests total.
Rollback-per-test (savepoint). Fixture opens a savepoint at setup; rolls back at teardown. Fast (<5 ms per test); requires the test to not commit; works only when the code under test doesn't call COMMIT. Suitable for read-only tests and code that uses the injected connection.
Template-DB clone-per-test. pytest-postgresql clones a pre-seeded template DB for every test (~100 ms per test); test gets a fresh DB it can COMMIT into; teardown drops the clone. Suitable for tests that must exercise real commit behaviour. This is the pattern for the majority of DE component tests.

Common interview probes on pytest fixtures.

"What scope should the Postgres fixture be?" — required answer is "the Postgres process at session; the DB itself at function via template-clone."
"When would you use autouse=True?" — cross-cutting concerns only; never for I/O fixtures.
"How do you avoid teardown order bugs?" — declare explicit dependencies (fixture A requests fixture B); pytest tears down in reverse dependency order.
"How do factory fixtures compose?" — one factory calls another; both are yielded as callables from separate fixtures.

Worked example — session-scoped Postgres + function-scoped template DB

Detailed explanation. The canonical DE Postgres fixture setup: one Postgres process per pytest session (~1 s startup, amortised across N tests) plus one template-cloned database per test (~100 ms clone, hermetic). This gives real Postgres behaviour with per-test isolation, at ~1/10th the cost of spinning up a fresh Postgres per test. Walk through the fixture wiring.

Session fixture. postgresql_proc — one binary, one datadir, one Postgres process. Started once, killed at exit.
Session fixture. template_db — a database created inside postgresql_proc, pre-seeded with schema + reference data. Cloned by every function-scoped postgresql.
Function fixture. postgresql — a fresh database cloned from template_db for every test.
Teardown. Function-scoped fixture drops its clone; session-scoped fixture kills the process.

Question. Wire the three fixtures and write one Layer-3 component test that inserts a row and asserts the schema is present.

Input.

Component	Scope	Setup cost	Teardown
postgresql_proc	session	~1 s	kill process
template_db	session	~500 ms (schema load)	drop DB
postgresql	function	~100 ms (clone)	drop DB

Code.

# tests/conftest.py — the three-fixture pattern
import pytest
from pathlib import Path
from pytest_postgresql import factories

# Session — one Postgres process for the entire pytest run
postgresql_proc = factories.postgresql_proc(
    port=None,                          # OS-allocated (xdist-safe)
    unixsocketdir="/tmp",
    load=[],                            # no schema on the process itself
)

# Session — the template database, pre-seeded with schema.sql
postgresql_template = factories.postgresql(
    "postgresql_proc",
    dbname="template_db",
    load=[Path(__file__).parent / "schema.sql"],
)

# Function — a fresh clone of template_db for every test
postgresql = factories.postgresql(
    "postgresql_proc",
    dbname="test_db",
    load=[Path(__file__).parent / "schema.sql"],
)

-- tests/schema.sql — the schema every test starts with
CREATE TABLE public.orders (
    id            BIGSERIAL PRIMARY KEY,
    customer_id   BIGINT       NOT NULL,
    total_cents   BIGINT       NOT NULL,
    status        TEXT         NOT NULL DEFAULT 'pending',
    created_at    TIMESTAMPTZ  NOT NULL DEFAULT now()
);

CREATE INDEX idx_orders_customer_id ON public.orders (customer_id);
CREATE INDEX idx_orders_status      ON public.orders (status);

CREATE TABLE public.customers (
    id      BIGSERIAL PRIMARY KEY,
    email   TEXT      NOT NULL UNIQUE,
    name    TEXT      NOT NULL
);

# tests/component/test_orders_schema.py — Layer-3 test
def test_orders_schema_has_indexes(postgresql):
    """Every test gets a fresh DB with the schema pre-loaded."""
    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT indexname
            FROM   pg_indexes
            WHERE  schemaname = 'public'
              AND  tablename  = 'orders'
            ORDER  BY indexname
        """)
        indexes = [r[0] for r in cur.fetchall()]

    assert "idx_orders_customer_id" in indexes
    assert "idx_orders_status"      in indexes
    assert "orders_pkey"            in indexes


def test_orders_insert_visible(postgresql):
    """Fresh DB per test — no leakage from previous tests."""
    with postgresql.cursor() as cur:
        cur.execute("""
            INSERT INTO public.orders(customer_id, total_cents)
            VALUES (1, 1000)
            RETURNING id
        """)
        order_id = cur.fetchone()[0]
    postgresql.commit()

    with postgresql.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM public.orders")
        assert cur.fetchone()[0] == 1     # exactly one — no leakage

Step-by-step explanation.

postgresql_proc starts one real Postgres binary on an OS-allocated port with a temp datadir. The port=None is critical for xdist safety — every worker gets a distinct port. This runs once per pytest session.
postgresql_template (session) creates one database inside that process, loads schema.sql, and stays around for the duration. It is never mutated during tests — it's the "clean stamp" that every function-scoped clone starts from.
postgresql (function) creates a fresh database per test by cloning the template (CREATE DATABASE test_db TEMPLATE template_db). Cloning is a Postgres-native O(1) operation that copies file blocks; ~100 ms even for a large template.
Every test receives a psycopg2.Connection to its own fresh DB. Tests can INSERT / COMMIT / DELETE freely; teardown drops the entire database, so leakage between tests is impossible.
The alternative — one long-lived DB with a rollback-per-test savepoint — is faster (~5 ms per test) but breaks any code under test that itself calls COMMIT (nested savepoints don't behave the same as top-level transactions). The template-clone pattern is the more forgiving default.

Output.

Test	Postgres process	Template DB	Function DB	Duration
test_orders_schema_has_indexes	reused	reused	freshly cloned	~110 ms
test_orders_insert_visible	reused	reused	freshly cloned	~115 ms
(subsequent tests)	reused	reused	freshly cloned	~100–120 ms
First test in run	started	seeded	cloned	~1.6 s

Rule of thumb. Postgres process at session scope; the database at function scope via template clone. Never start a Postgres per test; never share a database across tests.

Worked example — factory fixture for dbt seed data

Detailed explanation. DE tests routinely need to seed varied data shapes: "an order with 3 line items", "a customer with a 5-year history", "an inventory row at the safety-stock boundary". A single orders_seed fixture can't cover all shapes without becoming a config-driven monster. The factory-as-fixture pattern solves this: the fixture yields a callable that tests invoke with the shape they need. Walk through the pattern.

The factory function. make_order(customer_id=1, total_cents=1000, status='pending', **kwargs) -> Order. Every kwarg has a default; tests override only what they care about.
The fixture that yields the factory. Function-scoped — the fixture closes over the DB connection and returns the factory. Test invokes the factory zero or more times.
Cleanup. The DB connection is torn down by the underlying postgresql fixture; created rows disappear with the cloned DB. No manual cleanup needed.
Composability. make_order_with_items(customer_id, items) can call make_order internally, then insert line items.

Question. Write the make_order factory fixture and a test that uses it to seed three orders with distinct statuses.

Input.

Component	Value
Factory signature	make_order(customer_id=1, total_cents=1000, status='pending')
Return type	dataclass Order(id, customer_id, total_cents, status)
DB	function-scoped postgresql fixture
Cleanup	automatic (template-clone drops at teardown)

Code.

# tests/conftest.py — factory-as-fixture pattern
import pytest
from dataclasses import dataclass


@dataclass
class Order:
    id: int
    customer_id: int
    total_cents: int
    status: str


@pytest.fixture
def make_order(postgresql):
    """Return a factory function that inserts an order and returns it."""
    def _make(customer_id: int = 1,
              total_cents: int = 1000,
              status: str = "pending") -> Order:
        with postgresql.cursor() as cur:
            cur.execute("""
                INSERT INTO public.orders(customer_id, total_cents, status)
                VALUES (%s, %s, %s)
                RETURNING id
            """, (customer_id, total_cents, status))
            order_id = cur.fetchone()[0]
        postgresql.commit()
        return Order(order_id, customer_id, total_cents, status)

    return _make        # yield the callable, not the row


@pytest.fixture
def make_customer(postgresql):
    """Composable partner factory — used by higher-level make_order_full."""
    counter = {"n": 0}

    def _make(email: str | None = None, name: str = "Test User"):
        counter["n"] += 1
        email = email or f"user-{counter['n']}@example.com"
        with postgresql.cursor() as cur:
            cur.execute("""
                INSERT INTO public.customers(email, name)
                VALUES (%s, %s)
                RETURNING id
            """, (email, name))
            cid = cur.fetchone()[0]
        postgresql.commit()
        return cid

    return _make

# tests/component/test_orders_factory.py — using the factories
def test_status_filter_returns_only_matching(postgresql, make_order):
    """Seed three orders; query by status; assert selectivity."""
    make_order(status="pending")
    make_order(status="shipped")
    make_order(status="cancelled")

    with postgresql.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM public.orders WHERE status = 'shipped'")
        assert cur.fetchone()[0] == 1


def test_customer_order_join(postgresql, make_customer, make_order):
    """Compose two factories — customer + order — in one test."""
    alice = make_customer(email="alice@example.com", name="Alice")
    make_order(customer_id=alice, status="pending")
    make_order(customer_id=alice, status="pending")

    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT c.name, COUNT(o.id)
            FROM   public.customers c
            JOIN   public.orders   o ON o.customer_id = c.id
            GROUP  BY c.name
        """)
        assert cur.fetchall() == [("Alice", 2)]

Step-by-step explanation.

make_order (fixture) opens a closure over postgresql (the function-scoped DB connection) and returns an inner _make function. The fixture itself runs once per test that requests it; the inner function can be called any number of times by the test.
Every call to _make(...) executes one INSERT with defaulted-or-overridden values, commits, and returns an Order dataclass carrying the generated id. The test can compose arbitrary sequences of factory calls.
make_customer shows the composability pattern — it also uses a counter closure to generate unique emails so tests don't collide on the UNIQUE(email) constraint. This is a common factory idiom: side-effect-free defaults with per-call uniqueness.
In test_customer_order_join, both factories are requested; pytest resolves both against the same underlying postgresql connection because that fixture is function-scoped and the two factory fixtures both request it. Fixture-graph sharing is automatic.
Cleanup is trivial: the underlying postgresql fixture tears down the entire cloned database at test end, so every row inserted by every factory call vanishes with it. Factories never need explicit cleanup.

Output.

Test	Factory calls	DB rows	Duration
test_status_filter_returns_only_matching	make_order × 3	3 orders	~115 ms
test_customer_order_join	make_customer × 1, make_order × 2	1 cust + 2 orders	~120 ms
(fresh DB per test)	inputs isolated	outputs isolated	(no leakage)

Rule of thumb. Every DE test suite needs at least one factory per major table. Return a callable, defaulted-and-overridable, closing over the DB fixture. This is the pattern the industry converged on because it scales: 5 factories cover 500 tests worth of data-shape variation.

Worked example — `tmp_path` + `monkeypatch` for filesystem and env isolation

Detailed explanation. DE code routinely reads env vars (AWS_REGION, DATABASE_URL, KAFKA_BOOTSTRAP), writes to disk (parquet dumps, JSONL landing), and shells out to external tools (dbt run). Tests must isolate these side effects: an env var set in test A cannot leak into test B; a file written in test C cannot pollute test D. Pytest ships two built-in fixtures for this: tmp_path (per-test temp directory that auto-cleans) and monkeypatch (per-test env-var / attribute patcher that auto-restores).

tmp_path. Yields a pathlib.Path to a fresh directory. Auto-deleted after the test. Alternative: tmp_path_factory at session scope for cross-test reuse.
monkeypatch. Method-rich — monkeypatch.setenv, monkeypatch.setattr, monkeypatch.delenv. Every change auto-reverts at teardown.
monkeypatch.chdir. Changes the working directory for the test; reverts at teardown. Critical for dbt tests that assume dbt_project.yml in cwd.
When to reach for these. Any test that touches os.environ, os.chdir, pathlib.Path for writes, or global module-level attributes. Never modify these directly — always through the fixture.

Question. Write a test that reads env vars for its DB config, writes a JSONL landing file, and asserts both without leaking to sibling tests.

Input.

Concern	Fixture	Restoration
DATABASE_URL env var	monkeypatch.setenv	automatic
KAFKA_BOOTSTRAP env var	monkeypatch.setenv	automatic
/tmp/landing directory	tmp_path	auto-deleted
cwd for dbt run	monkeypatch.chdir	reverted

Code.

# tests/component/test_env_and_fs.py — tmp_path + monkeypatch
import json
from pathlib import Path
from mypkg.landing import write_orders_landing


def test_write_orders_landing_creates_jsonl(tmp_path, monkeypatch):
    """Isolated env + filesystem; every assertion is post-test cleaned."""
    # 1. Env — inject config the code reads via os.environ
    monkeypatch.setenv("LANDING_ROOT", str(tmp_path))
    monkeypatch.setenv("AWS_REGION", "us-west-2")

    # 2. Filesystem — everything under tmp_path is auto-cleaned
    orders = [
        {"id": 1, "total_cents": 1000},
        {"id": 2, "total_cents": 2500},
    ]

    # 3. Exercise the code under test
    output_path = write_orders_landing(orders)

    # 4. Assert the file materialised where we expected
    assert output_path == tmp_path / "orders.jsonl"
    lines = output_path.read_text().splitlines()
    assert len(lines) == 2
    assert json.loads(lines[0]) == {"id": 1, "total_cents": 1000}


def test_env_var_does_not_leak_to_next_test(monkeypatch):
    """Sanity check — env from the previous test must be gone."""
    import os
    assert os.environ.get("LANDING_ROOT") is None      # cleared by monkeypatch teardown
    assert os.environ.get("AWS_REGION")   != "us-west-2"  # or wasn't set at all


def test_dbt_test_uses_chdir(tmp_path, monkeypatch):
    """dbt requires cwd == project root; monkeypatch.chdir reverts."""
    # Create a fake dbt_project.yml in tmp_path
    (tmp_path / "dbt_project.yml").write_text("name: test_project\nversion: 1.0\n")

    monkeypatch.chdir(tmp_path)
    # Anything the code does with Path.cwd() sees tmp_path
    from pathlib import Path
    assert Path.cwd() == tmp_path
    # After the test, cwd reverts automatically

Step-by-step explanation.

tmp_path is a function-scoped built-in fixture that yields a fresh temp dir. Pytest deletes the directory at test end (subject to the --basetemp retention setting, typically "keep the last 3"). This makes filesystem tests hermetic by default.
monkeypatch.setenv("LANDING_ROOT", str(tmp_path)) injects the env var for the duration of this test only. Pytest records the original value at set time and restores it at teardown — no test can leak an env var into a sibling.
Step 3 exercises write_orders_landing, which reads LANDING_ROOT from os.environ and writes orders.jsonl under it. Because both the env var and the target directory are per-test, this test can be re-run under pytest-xdist in parallel with 10 copies of itself without collision.
test_env_var_does_not_leak_to_next_test is the sanity check — proves monkeypatch cleaned up. In CI, this pattern is worth its weight in gold for catching accidental global-state leakage.
test_dbt_test_uses_chdir shows the monkeypatch.chdir idiom for tools that require a specific working directory. Without this, sibling tests would start in the wrong cwd and fail cryptically.

Output.

Test	tmp_path	env vars set	env vars visible after
test_write_orders_landing	/tmp/pytest-.../test_0	LANDING_ROOT, AWS_REGION	(cleared)
test_env_var_does_not_leak	/tmp/pytest-.../test_1	(none)	(still cleared)
test_dbt_test_uses_chdir	/tmp/pytest-.../test_2	(none, chdir only)	cwd restored

Rule of thumb. For any test that touches env vars, cwd, or filesystem writes, use monkeypatch and tmp_path — never modify globals directly and never rely on test-ordered cleanup. Test isolation is non-negotiable.

Senior interview question on fixture scope discipline

A senior interviewer might ask: "You inherit a suite with 400 pytest tests where every test spins up its own Postgres via docker.run(...) in a function-scoped fixture. Full CI run takes 42 minutes. Walk me through the fixture-scope migration you'd do, the template-DB pattern, the factory pattern for test data, and the exact time budget you'd expect on 8-core CI."

Solution Using session-scoped Postgres process + template-clone per test + factory fixtures

# 1. tests/conftest.py — session Postgres, function template clone, factories
import pytest
from dataclasses import dataclass
from pathlib import Path
from pytest_postgresql import factories


# ------------------------------------------------------------------
# Session — one Postgres binary for the whole test run
# ------------------------------------------------------------------
postgresql_proc = factories.postgresql_proc(
    port=None,                          # OS-allocated port; xdist-safe
    unixsocketdir="/tmp",
    postgres_options="-c fsync=off "    # turn off durability for speed
                     "-c synchronous_commit=off "
                     "-c full_page_writes=off",
)


# ------------------------------------------------------------------
# Function — a fresh clone of the schema-loaded template
# ------------------------------------------------------------------
postgresql = factories.postgresql(
    "postgresql_proc",
    dbname="test_db",
    load=[Path(__file__).parent / "schema.sql"],
)


# ------------------------------------------------------------------
# Factory — reusable data shape generator
# ------------------------------------------------------------------
@dataclass
class Order:
    id: int
    customer_id: int
    total_cents: int
    status: str


@pytest.fixture
def make_order(postgresql):
    """Callable that inserts an order with defaulted overrides."""
    def _make(customer_id=1, total_cents=1000, status="pending") -> Order:
        with postgresql.cursor() as cur:
            cur.execute("""
                INSERT INTO public.orders(customer_id, total_cents, status)
                VALUES (%s, %s, %s) RETURNING id
            """, (customer_id, total_cents, status))
            oid = cur.fetchone()[0]
        postgresql.commit()
        return Order(oid, customer_id, total_cents, status)
    return _make

# 2. A representative Layer-3 test using both fixtures
def test_bulk_status_update(postgresql, make_order):
    """Seed 5 pending orders; bulk-update to shipped; assert count."""
    for _ in range(5):
        make_order(status="pending")

    with postgresql.cursor() as cur:
        cur.execute("""
            UPDATE public.orders
            SET    status = 'shipped'
            WHERE  status = 'pending'
        """)
        rowcount = cur.rowcount
    postgresql.commit()

    assert rowcount == 5

    with postgresql.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM public.orders WHERE status='shipped'")
        assert cur.fetchone()[0] == 5

# 3. pytest.ini — test-discovery + speed-critical options
[pytest]
addopts = -ra -q --strict-markers --strict-config
markers =
    integration: end-to-end tests that require docker-compose
    slow: tests > 1 s that we want to skip in the fast lane
testpaths = tests

Step-by-step trace.

Step	Before (function-scope Postgres)	After (session + template clone)
Postgres startup	400 × ~1 s = 6.7 min	1 × ~1 s
DB clone	none	400 × ~100 ms = ~40 s
Schema load	400 × ~200 ms = 80 s	1 × ~200 ms (into template)
Test body	400 × ~50 ms = 20 s	400 × ~50 ms = 20 s
Full run (serial)	~42 min	~2 min
Full run (8-core xdist)	~6 min	~30 s

After the migration, the same 400 tests run in 2 minutes serially and 30 seconds with 8-core xdist. The Postgres process starts once per session (per xdist worker), the template DB is created once with the schema, and every test gets a fresh clone via Postgres's native CREATE DATABASE ... TEMPLATE machinery.

Output:

Metric	Before	After
CI runtime (serial)	42 min	2 min
CI runtime (8-core xdist)	~6 min	~30 s
Postgres binary starts	400	1 (× N workers)
Docker image pulls	400	0 (pytest-postgresql uses local binary)
Test isolation	full (own container)	full (own DB)
Memory pressure	~400 × 100 MB = 40 GB peak	~1 × 100 MB per worker

Why this works — concept by concept:

Session-scoped process fixture — the Postgres binary starts once and is reused. Amortises ~1 s of startup across all N tests. This is the O(1) versus O(N) win.
Template-DB clone — Postgres's CREATE DATABASE ... TEMPLATE is a file-block copy at the OS level. ~100 ms even for GB-scale schemas. Every test gets a fresh DB with the schema pre-loaded, no per-test schema loading needed.
Speed-critical Postgres flags — fsync=off, synchronous_commit=off, full_page_writes=off trade durability for speed. Safe in tests because the DB is thrown away at exit. Cuts commit latency by ~5×.
Factory-as-fixture — one factory per major table, closure over postgresql, callable defaults. Scales to arbitrary data shapes without exploding the fixture count.
Cost — one Postgres process per xdist worker (~100 MB RAM), one template DB (~50 MB), ~100 ms per test for the clone. The eliminated cost is 400 container startups + 400 schema loads = ~7 minutes. Net O(1) startup per worker versus O(N) startup per test.

Python
Topic — oop/python
OOP + fixtures + factories in Python

Practice →

SQL Topic — sql SQL problems on schema seeding and test data

Practice →

3. Parametrization — table-driven tests over dbt/Spark transforms

`@pytest.mark.parametrize` turns a 100-test suite into a 3-file suite — as long as your `ids=` are readable

The mental model in one line: pytest parametrize is a decorator that runs the same test function once per row of a parameter table, giving you N logical tests for the cost of one function definition — but the readability of the resulting test IDs (test_transform[null-input] vs test_transform[0]) is the single largest predictor of whether your team maintains the parametrized style or refactors it into 30 copy-pasted test functions six months later. Every senior data engineer has inherited a suite with test_transform[0] through test_transform[47] — impossible to grep for the failing case — and every senior data engineer has replaced it with ids=["null-input", "empty-str", "unicode-emoji", ...] in a two-hour refactor that pays for itself the first time a single case fails.

The four flavours of parametrization every DE engineer uses.

Static @pytest.mark.parametrize. The classic — a list of tuples decorating the test function. Best when the test cases are known at collection time and small enough to inline. This covers ~70% of DE parametrized tests.
@pytest.mark.parametrize(..., indirect=True). Passes the parameter through a fixture instead of directly to the test. Useful when the parameter needs setup — e.g. parametrizing which DB backend a test runs against (Postgres vs DuckDB).
Fixture-level parametrization (@pytest.fixture(params=...)). Parametrizes the fixture itself; every test that requests the fixture runs once per param. Powerful for "run the whole suite against multiple backends" — but easy to over-use.
Dynamic pytest_generate_tests. The hook — pytest calls it during collection and you supply the params programmatically. Essential for "run this contract test once per model in the dbt manifest" or "once per table in the schema" — where the axes are discovered at runtime.

Axes senior DEs parametrize over — the canonical five.

Null / empty / edge / valid / invalid. The base axis for any transform. Every column that can be null must have a null-case test; every string must have empty and unicode-emoji cases; every numeric must have zero, negative, and overflow cases.
Backend. For code that must work on Postgres, MySQL, Snowflake, DuckDB — parametrize the fixture; run the same test across all backends. Catches dialect-specific SQL bugs.
Time boundary. For window/session logic — [start-inclusive, end-exclusive, span-crossing-midnight, span-crossing-DST, single-instant]. This axis catches roughly half of all off-by-one bugs.
Volume. For batch code — [1-row, 100-rows, 10k-rows] cases parametrized over the same transform. The 10k case catches memory-explosion bugs the 100 case doesn't.
Manifest. For contract tests — parametrize over every model / column / test / source in the dbt manifest, or every table in the information_schema. Amplifies one test into hundreds.

The ids= discipline — the single most important pytest hygiene rule.

Bad. @pytest.mark.parametrize("x,y", [(1, 2), (3, 4), (5, 6)]) → test IDs test_foo[1-2], test_foo[3-4], test_foo[5-6]. Grep-friendly only if your values are unique and semantic.
Better. @pytest.mark.parametrize("x,y", [(1, 2), (3, 4)], ids=["simple-case", "boundary-case"]) → readable IDs; the failing case's meaning is obvious from the test name.
Best. Use a helper that generates IDs from a dataclass — ids=lambda p: f"{p.name}-{p.expected}". Scales to hundreds of cases without hand-writing IDs.
Never. Do not omit ids= for parametrizations with more than 3 cases. Every failing test that reports test_foo[test_foo0] is a debugging tax.

Common interview probes on pytest parametrize.

"How do you parametrize a test?" — required answer: @pytest.mark.parametrize with ids=.
"What does indirect=True do?" — routes the param through a fixture, useful for parameterizing setup.
"How do you parametrize dynamically at collection time?" — pytest_generate_tests hook.
"What's the tradeoff between fixture-level params and decorator-level params?" — fixture-level multiplies every test in scope; decorator-level scopes to the specific test.

Worked example — Spark UDF parametrized across null/edge/valid inputs

Detailed explanation. A Spark UDF that normalises phone numbers is a canonical parametrization target. The correctness contract has ~6 axes: normal input, leading whitespace, embedded punctuation, null input, empty string, malformed input. Six tests × one function = six lines of parametrize data. Walk through the pattern.

Function under test. normalize_phone(raw: str | None) -> str | None.
Axes. null → null; empty → null; valid → digits-only; punctuation → digits-only; unicode → error; too-short → error.
ID strategy. Human-readable names describing the axis.
Assert. Exact-match or exception; no wiggle room.

Question. Write the parametrized test covering all six axes with human-readable IDs.

Input.

Axis	raw input	expected output	id
null	None	None	null-in
empty	""	None	empty-str
valid	"+1 415-555-1212"	"+14155551212"	valid-with-punct
digits	"4155551212"	"+14155551212"	digits-only
unicode	"📱4155551212"	ValueError	unicode-emoji
short	"555"	ValueError	too-short

Code.

# tests/unit/test_normalize_phone.py — parametrized Spark UDF test
import pytest
from mypkg.udfs.phone import normalize_phone


@pytest.mark.parametrize("raw,expected", [
    (None,                 None),
    ("",                   None),
    ("+1 415-555-1212",    "+14155551212"),
    ("4155551212",         "+14155551212"),
], ids=[
    "null-in",
    "empty-str",
    "valid-with-punct",
    "digits-only",
])
def test_normalize_phone_valid_inputs(raw, expected):
    """Six valid-or-null inputs; exact-match against expected."""
    assert normalize_phone(raw) == expected


@pytest.mark.parametrize("raw", [
    "📱4155551212",
    "555",
    "not-a-phone",
], ids=[
    "unicode-emoji",
    "too-short",
    "totally-garbage",
])
def test_normalize_phone_raises_on_invalid(raw):
    """Three invalid inputs; must raise ValueError."""
    with pytest.raises(ValueError):
        normalize_phone(raw)

Step-by-step explanation.

Pytest collects test_normalize_phone_valid_inputs four times — once per row in the parametrize table. Each collection produces a distinct test node with a distinct ID: test_normalize_phone_valid_inputs[null-in], test_normalize_phone_valid_inputs[empty-str], etc.
When a test fails, the pytest report shows the ID — so a regression in the punctuation-handling case shows up as FAILED tests/unit/test_normalize_phone.py::test_normalize_phone_valid_inputs[valid-with-punct]. You know exactly which axis broke without running the debugger.
Splitting valid-vs-invalid into two parametrize decorators is idiomatic — the assertion style differs (equality vs pytest.raises), so one function per assertion style keeps each test focused.
pytest.raises(ValueError) is a context manager that asserts the block raises that exception (or a subclass). Using it inside a parametrized loop tests three invalid inputs with one function.
If the UDF later gains a "return None for whitespace-only input" behaviour, adding a row (" ", None, "whitespace-only") takes one line — no new test function, no fixture change. This is the parametrization amplifier at work.

Output.

Test ID	raw	expected	actual	pass/fail
test_normalize_phone_valid_inputs[null-in]	None	None	None	pass
test_normalize_phone_valid_inputs[empty-str]	""	None	None	pass
test_normalize_phone_valid_inputs[valid-with-punct]	"+1 415-555-1212"	"+14155551212"	"+14155551212"	pass
test_normalize_phone_valid_inputs[digits-only]	"4155551212"	"+14155551212"	"+14155551212"	pass
test_normalize_phone_raises_on_invalid[unicode-emoji]	"📱..."	ValueError	ValueError	pass
test_normalize_phone_raises_on_invalid[too-short]	"555"	ValueError	ValueError	pass

Rule of thumb. Every non-trivial transform gets a null/empty/valid/edge/invalid parametrization with ids=. Six lines of table data beats six copy-pasted test functions every time.

Worked example — dbt model parametrized over source-fixture rows

Detailed explanation. A dbt model that computes dim_customer_lifetime_value from raw orders needs to be tested against several source-row patterns: customer with one order, customer with 100 orders, customer with zero orders (should return zero LTV), customer with a returned order (should net out). A single parametrized test can seed the source data and assert the model output per case. Walk through the pattern.

Source rows. Parametrized dicts describing customer + order data.
Model. dim_customer_lifetime_value.sql — SUM(revenue) - SUM(returns) per customer.
Assertion. Materialised model row equals expected LTV.
Cleanup. Each parametrized case starts with a fresh DB (template clone).

Question. Write the parametrized dbt model test covering four source-data shapes.

Input.

Case	Customer orders	Expected LTV	id
single order	1 × $100	$100	one-hundred
bulk orders	100 × $10	$1000	thousand
no orders	0	$0	zero-orders
returned order	1 × $100 + 1 × -$50	$50	with-return

Code.

# tests/component/test_dim_customer_ltv.py — parametrized dbt model test
import pytest
from dataclasses import dataclass
from mypkg.dbt_helpers import run_dbt_model


@dataclass
class CustomerLTVCase:
    name: str
    orders: list[int]     # cents; negative = return
    expected_ltv_cents: int


CASES = [
    CustomerLTVCase("one-hundred",   [10_000],                                10_000),
    CustomerLTVCase("thousand",      [1_000] * 100,                          100_000),
    CustomerLTVCase("zero-orders",   [],                                          0),
    CustomerLTVCase("with-return",   [10_000, -5_000],                        5_000),
]


@pytest.mark.parametrize(
    "case",
    CASES,
    ids=lambda c: c.name,           # ids from the dataclass field
)
def test_dim_customer_ltv(postgresql, case: CustomerLTVCase):
    """Seed source rows for one customer; run the dbt model; assert LTV."""
    # 1. Seed the source table
    with postgresql.cursor() as cur:
        cur.execute("""
            INSERT INTO public.customers(id, email, name)
            VALUES (%s, %s, %s) RETURNING id
        """, (1, "test@example.com", "Test User"))
        for i, cents in enumerate(case.orders, start=1):
            cur.execute("""
                INSERT INTO public.orders(id, customer_id, total_cents, status)
                VALUES (%s, %s, %s, %s)
            """, (i, 1, cents, "completed" if cents > 0 else "returned"))
    postgresql.commit()

    # 2. Materialise the dbt model (uses postgresql connection)
    run_dbt_model("dim_customer_lifetime_value", conn=postgresql)

    # 3. Assert the LTV row equals the expected cents
    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT ltv_cents
            FROM   analytics.dim_customer_lifetime_value
            WHERE  customer_id = 1
        """)
        row = cur.fetchone()

    if case.orders:
        assert row is not None
        assert row[0] == case.expected_ltv_cents
    else:
        # No orders means no LTV row (or zero-valued row, depending on the model)
        assert row is None or row[0] == 0

Step-by-step explanation.

The CustomerLTVCase dataclass captures every axis of one test case — the name (which drives the ID), the input orders, and the expected LTV. Grouping into a dataclass beats a bare tuple because named fields survive refactoring.
ids=lambda c: c.name extracts the ID from the dataclass, giving test IDs like test_dim_customer_ltv[one-hundred] and test_dim_customer_ltv[with-return]. Adding a new case is one dataclass entry.
Each parametrized invocation gets its own fresh postgresql (function-scope) DB, so the four cases don't leak data into each other. This is why parametrization works cleanly with per-test DB isolation.
run_dbt_model is a thin wrapper around dbt run --select dim_customer_lifetime_value that points at the injected connection — this lets tests exercise real dbt behaviour without a full dbt-core install ceremony.
The zero-orders case tests a subtle boundary: does the model emit a zero row or no row? The test asserts both possibilities (is None or == 0) because either is a reasonable design; the test locks in whichever the current model chooses.

Output.

Test ID	Orders seeded	Expected LTV	Actual LTV	pass/fail
test_dim_customer_ltv[one-hundred]	1 × 10,000	10,000	10,000	pass
test_dim_customer_ltv[thousand]	100 × 1,000	100,000	100,000	pass
test_dim_customer_ltv[zero-orders]	0	0 (or no row)	no row	pass
test_dim_customer_ltv[with-return]	10,000, -5,000	5,000	5,000	pass

Rule of thumb. Model dbt tests as parametrized cases with a dataclass per axis. Every dbt model gets at least the four cases: single, bulk, zero, edge-case. The dataclass approach outperforms tuple lists for anything past three fields.

Worked example — schema-contract test parametrized over every table

Detailed explanation. A canonical DE contract test: every table in the public schema must have (a) a primary key, (b) a created_at column, (c) a NOT NULL constraint on the primary key. Writing one test per table is tedious; pytest_generate_tests discovers the tables at collection time and generates one test per discovered table. Walk through the hook.

Discovery. pytest_generate_tests(metafunc) runs at collection; queries information_schema.tables.
Generation. metafunc.parametrize supplies the table names as params.
Assertion. Each generated test validates one table's contract.
Effect. Adding a new table automatically adds a new test — zero test-file changes.

Question. Implement the pytest_generate_tests hook that generates one contract test per table in public.

Input.

Component	Value
Discovery source	information_schema.tables
Filter	schema = 'public' AND table_type = 'BASE TABLE'
Params per table	(table_name,)
Assertions per test	has PK, has created_at, PK is NOT NULL

Code.

# tests/component/test_schema_contract.py — dynamic parametrization
import pytest
import psycopg2

# ------------------------------------------------------------------
# The pytest_generate_tests hook — discovers tables at collection
# ------------------------------------------------------------------
def pytest_generate_tests(metafunc):
    """Generate one test per table in public schema."""
    if "table_name" not in metafunc.fixturenames:
        return          # this hook only fires for tests requesting `table_name`

    # Query the schema (uses a static connection URL; template DB is up)
    conn = psycopg2.connect(
        "host=/tmp dbname=template_db"      # or use env vars
    )
    with conn.cursor() as cur:
        cur.execute("""
            SELECT table_name
            FROM   information_schema.tables
            WHERE  table_schema = 'public'
              AND  table_type   = 'BASE TABLE'
            ORDER  BY table_name
        """)
        tables = [r[0] for r in cur.fetchall()]
    conn.close()

    metafunc.parametrize(
        "table_name",
        tables,
        ids=tables,             # ID = table name itself
    )


# ------------------------------------------------------------------
# The single test — runs once per discovered table
# ------------------------------------------------------------------
def test_table_has_primary_key(postgresql, table_name):
    """Every public table must have a primary key constraint."""
    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT c.contype
            FROM   pg_constraint c
            JOIN   pg_class      t ON t.oid = c.conrelid
            WHERE  t.relname = %s
              AND  c.contype = 'p'
        """, (table_name,))
        pk = cur.fetchone()

    assert pk is not None, f"table {table_name} has no primary key"


def test_table_has_created_at(postgresql, table_name):
    """Every public table must have a created_at column."""
    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT column_name
            FROM   information_schema.columns
            WHERE  table_schema = 'public'
              AND  table_name   = %s
              AND  column_name  = 'created_at'
        """, (table_name,))
        col = cur.fetchone()

    assert col is not None, f"table {table_name} missing created_at"

Step-by-step explanation.

pytest_generate_tests is a hook pytest calls during collection for every test function. Inside the hook, we check whether the current test requests the table_name fixture — if not, we skip (the hook fires for all tests, not just ours).
When the hook fires for test_table_has_primary_key, it queries the template DB for the current schema, gets back ['customers', 'orders'] (or whatever exists), and calls metafunc.parametrize("table_name", [...]) — this is exactly like the decorator, but supplied dynamically.
ids=tables sets the test IDs to the table names themselves: test_table_has_primary_key[customers], test_table_has_primary_key[orders]. When one fails, you know which table is the culprit.
Adding a new table to schema.sql automatically adds a new test to the run — zero test-file changes required. This is the amplifier's leverage.
Note that the discovery query targets template_db, not the function-scoped postgresql. The discovery must happen at collection time (before any function fixture is available), so we open a direct connection.

Output.

Test ID	Table	PK found	created_at found	pass/fail
test_table_has_primary_key[customers]	customers	yes	—	pass
test_table_has_primary_key[orders]	orders	yes	—	pass
test_table_has_created_at[customers]	customers	—	no	FAIL
test_table_has_created_at[orders]	orders	—	yes	pass

Rule of thumb. For any DE contract that must hold across every table in a schema, use pytest_generate_tests — never hand-write one test per table. The dynamic generation makes new-table addition automatically covered.

Senior interview question on pytest parametrize discipline

A senior interviewer might ask: "You have a Spark job that computes retention cohorts and a dbt model that computes MRR. You need to cover null/edge/valid axes plus a manifest-driven axis 'every model in the marts folder'. Walk me through the parametrization strategy — decorator vs fixture vs pytest_generate_tests, how you generate readable IDs, and how you keep the parametrize table maintainable."

Solution Using layered parametrize decorators + fixture params + `pytest_generate_tests`

# 1. Layer A — decorator-level, static, axis parametrize
import pytest
from dataclasses import dataclass


@dataclass
class RetentionCase:
    name: str
    signup_dates: list[str]
    activity_dates: list[list[str]]     # per user
    cohort: str
    expected_retention: float


RETENTION_CASES = [
    RetentionCase("all-active",    ["2026-01-01"] * 3,
                  [["2026-01-08"], ["2026-01-15"], ["2026-01-22"]],
                  "2026-01", 1.0),
    RetentionCase("half-churned",  ["2026-01-01"] * 4,
                  [["2026-01-08"], ["2026-01-08"], [], []],
                  "2026-01", 0.5),
    RetentionCase("zero-cohort",   [], [], "2026-01", 0.0),
]


@pytest.mark.parametrize("case", RETENTION_CASES, ids=lambda c: c.name)
def test_retention_cohort(spark_session, case: RetentionCase):
    from mypkg.retention import cohort_retention
    df = spark_session.createDataFrame(
        [(i, sd, ad) for i, (sd, ad) in enumerate(zip(case.signup_dates, case.activity_dates))],
        schema=["user_id", "signup_date", "activity_dates"],
    )
    result = cohort_retention(df, cohort=case.cohort)
    assert abs(result - case.expected_retention) < 1e-9

# 2. Layer B — fixture-level, params, backend switching
import pytest


@pytest.fixture(params=["postgres", "duckdb"], ids=["pg", "duck"])
def sql_backend(request, postgresql, tmp_path):
    """Yield a connection to either Postgres or DuckDB."""
    if request.param == "postgres":
        yield postgresql
    else:
        import duckdb
        conn = duckdb.connect(str(tmp_path / "test.duckdb"))
        # (schema-load elided)
        yield conn
        conn.close()


def test_mrr_query_matches_across_backends(sql_backend):
    """Same query, both backends, must agree on MRR."""
    from mypkg.mrr import compute_mrr
    result = compute_mrr(sql_backend, month="2026-01")
    assert result >= 0     # basic sanity

# 3. Layer C — pytest_generate_tests, manifest-driven
def pytest_generate_tests(metafunc):
    """Discover every model in the dbt manifest at collection."""
    if "dbt_model" not in metafunc.fixturenames:
        return

    import json
    from pathlib import Path
    manifest = json.loads(Path("target/manifest.json").read_text())
    # Filter to models under models/marts/
    marts_models = [
        node["name"]
        for node in manifest["nodes"].values()
        if node["resource_type"] == "model"
        and node["path"].startswith("marts/")
    ]
    metafunc.parametrize("dbt_model", marts_models, ids=marts_models)


def test_mart_has_row_count_gt_zero(postgresql, dbt_model):
    """Every mart model must materialise to >= 1 row."""
    with postgresql.cursor() as cur:
        cur.execute(f"SELECT COUNT(*) FROM analytics.{dbt_model}")
        row_count = cur.fetchone()[0]
    assert row_count > 0, f"mart {dbt_model} materialised zero rows"

Step-by-step trace.

Layer	Style	Discovery	ID example	Test count
A (decorator)	static list	at import	test_retention_cohort[all-active]	3
A (decorator)	static list	at import	test_retention_cohort[half-churned]	1
A (decorator)	static list	at import	test_retention_cohort[zero-cohort]	1
B (fixture params)	fixture-level	at collection	test_mrr_query_matches_across_backends[pg]	1
B (fixture params)	fixture-level	at collection	test_mrr_query_matches_across_backends[duck]	1
C (generate_tests)	dynamic hook	at collection	test_mart_has_row_count_gt_zero[fct_orders]	N (manifest-driven)

After deployment, the same parametrized suite covers three orthogonal axes with three distinct techniques: (A) static known cases for retention logic, (B) fixture-level cases for cross-backend agreement, (C) manifest-driven cases for every mart model. Adding a new mart is zero test-file changes; adding a new retention edge case is one dataclass row.

Output:

Case	Style	Effort to add new case	Runs
new retention axis	Layer A	1 dataclass row	+1 test
add MySQL backend	Layer B	1 fixture param + branch	+N tests (one per test)
add new mart model	Layer C	zero (manifest picks it up)	+1 test automatically

Why this works — concept by concept:

Decorator-level parametrize — the classic; best for known, static axes; readable IDs via lambda over dataclass fields.
Fixture-level parametrize — routes the same params through a fixture; ideal for backend/config axes that need per-param setup; every test that requests the fixture inherits the param multiplication.
pytest_generate_tests — the escape hatch; discovers axes at collection time from runtime sources (dbt manifest, information_schema, YAML config); the amplifier that lets one test cover every model.
ids= discipline — the difference between test_foo[0] (hostile) and test_foo[with-return] (grep-friendly). Use lambda c: c.name when working with dataclasses.
Cost — parametrization overhead is negligible (~µs per case at collection); the dominant cost is the test body itself. The eliminated cost is copy-pasted test functions (typically 10-20 lines each) that drift out of sync. Net: three techniques cover 90% of DE parametrization needs. O(1) code per axis added.

Data validation
Topic — data-validation
Data validation and contract-testing problems

Practice →

Python Topic — defensive-coding Defensive coding and edge-case coverage

Practice →

4. Docker-Compose + Testcontainers integration tests

`testcontainers-python` + a session-scoped `DockerCompose` fixture is the pattern every senior DE has converged on — hermetic, fast, and CI-safe

The mental model in one line: pytest docker-compose integration testing means declaring a docker-compose.test.yml next to your source, wrapping it in a session-scoped pytest fixture that spins the whole networked stack (Postgres + Kafka + MinIO + whatever) up once per pytest run and tears it down at exit — and if you use testcontainers-python instead of shelling out to docker-compose, you get first-class wait_for health checks, per-container get_service_host accessors, and clean handling of xdist-worker port allocation for free. Every senior data engineer has debugged the "tests pass on my laptop but fail on CI because Postgres wasn't ready yet" flake and every senior data engineer has fixed it with a wait_for on SELECT 1 inside a retry loop.

Two container primitives — pick one and commit.

testcontainers-python per-service. Instantiate PostgresContainer("postgres:15"), KafkaContainer("confluentinc/cp-kafka:7.6.0"), MinioContainer("minio/minio:latest") in Python. Best for tests that need one or two containers. Handles wait-for-ready, port publishing, and network isolation. Idiomatic in 2026.
testcontainers.compose.DockerCompose. Wraps a docker-compose.yml file. Best for tests that need the full stack (Airflow scheduler + webserver + Postgres + Redis + Celery worker + Kafka). Handles docker-compose up -d, health-check waits, and docker-compose down -v at teardown.
pytest-docker-compose plugin. Older wrapper; still maintained. Slightly less flexible than testcontainers-python; retained in legacy suites.
Raw subprocess.run(['docker-compose', 'up']). The anti-pattern. Loses wait_for, forces you to hand-roll health checks, and breaks under xdist. Only appropriate for one-off manual scripts, never for a test suite.

The three lifecycle patterns for container fixtures.

Session scope + shared state. Spin the stack up once; every test hits the same DB. Fast (one startup ~10 s amortised over 100 tests) but requires tests to clean up after themselves or use per-test schemas / topics. This is the pattern for most DE integration suites.
Session scope + per-test isolation. Spin the stack up once; each test creates its own schema (SET search_path) or Kafka topic (test_<uuid>). Fast and hermetic. This is the pattern for Layer-4 tests that must be independent.
Function scope + full teardown. Spin the stack up per test; tear down completely at teardown. Slow (~10 s per test); use only for tests that need pristine state and can't share containers safely.

The wait-for-ready idiom — non-optional for every container.

docker-compose up -d returns as soon as the containers are started, not when they are ready. Postgres takes ~2-5 s after start to accept connections; Kafka ~5-10 s; MinIO ~2 s. Racing this window is the #1 source of container-test flakes.
HTTP health-check pattern. testcontainers.compose.DockerCompose.wait_for("http://localhost:9200/_cluster/health") polls until the endpoint returns 200. Set a 30-second timeout.
TCP-connect pattern. For services without HTTP (raw Postgres, Kafka broker), poll a psycopg2.connect(...) or KafkaConsumer(bootstrap_servers=...) in a retry loop. Third-party libs like tenacity (@retry(stop_after_delay=30, wait_fixed=0.5)) make this concise.
Docker-compose healthchecks. Define healthcheck in the yml (test: pg_isready -U test) and use depends_on: postgres: condition: service_healthy — compose itself waits, and the fixture just waits on the top-level service.

Common interview probes on pytest docker-compose.

"How do you avoid port collisions under xdist?" — either OS-allocated ports (published on the container side) or per-worker compose projects (COMPOSE_PROJECT_NAME=test_${WORKER_ID}).
"How do you clean up volumes?" — docker-compose down -v at teardown; DockerCompose context manager does this automatically.
"What's the wait-for-ready story?" — HTTP endpoint poll, TCP connect retry, or compose-native healthcheck + depends_on.condition.
"When would you pick testcontainers-python over compose?" — small stack (1-2 containers) and per-test isolation is required.

Worked example — `pytest-postgresql` fixture with a schema template

Detailed explanation. The Layer-3 equivalent of a docker-compose stack: a real Postgres process, but no container. pytest-postgresql bundles a Postgres binary invocation directly — no Docker required — and provides postgresql_proc (session, one binary) plus postgresql (function, one DB clone). This is faster than dockerised Postgres for local dev and works identically on GitHub Actions Ubuntu runners. Walk through the wiring.

What it starts. A postgres binary running against a temp datadir; no image pull, no Docker daemon.
How it isolates tests. Function-scoped DB via template clone.
Where the schema lives. load=[Path('schema.sql')] applied to the template at session start.
What breaks. Nothing if the runner has postgresql-15 installed; a clear error message if not.

Question. Wire the pytest-postgresql template + a single component test that inserts a row.

Input.

Component	Value
Postgres binary	postgresql-15 on the runner
Data dir	ephemeral temp dir
Schema	tests/schema.sql
Test DB	fresh template clone per test

Code.

# tests/conftest.py — pytest-postgresql fixtures
from pathlib import Path
from pytest_postgresql import factories

TEST_DIR = Path(__file__).parent
SCHEMA_SQL = TEST_DIR / "schema.sql"

postgresql_proc = factories.postgresql_proc(
    port=None,                          # OS-allocated
    unixsocketdir="/tmp",
)

postgresql = factories.postgresql(
    "postgresql_proc",
    dbname="test_db",
    load=[SCHEMA_SQL],
)

# tests/component/test_pg_template.py
def test_schema_loaded(postgresql):
    """Template clone must include the schema.sql tables."""
    with postgresql.cursor() as cur:
        cur.execute("""
            SELECT tablename FROM pg_tables
            WHERE  schemaname = 'public'
            ORDER  BY tablename
        """)
        tables = [r[0] for r in cur.fetchall()]
    assert "orders"    in tables
    assert "customers" in tables


def test_insert_and_select(postgresql):
    """Round-trip an insert; assert the row is visible."""
    with postgresql.cursor() as cur:
        cur.execute("""
            INSERT INTO public.orders(customer_id, total_cents, status)
            VALUES (1, 500, 'pending') RETURNING id
        """)
        oid = cur.fetchone()[0]
    postgresql.commit()

    with postgresql.cursor() as cur:
        cur.execute("SELECT total_cents FROM public.orders WHERE id = %s", (oid,))
        row = cur.fetchone()
    assert row == (500,)

Step-by-step explanation.

postgresql_proc at session scope starts one Postgres binary. The port=None argument tells pytest-postgresql to bind to an OS-allocated port; the caller can read the actual port from postgresql_proc.port. This is what makes xdist safe.
postgresql at function scope creates a fresh database per test by (a) waiting for postgresql_proc to be up, (b) issuing CREATE DATABASE test_db_<random>, (c) applying schema.sql, (d) opening a psycopg2 connection, (e) at teardown dropping the DB.
SCHEMA_SQL is loaded on every clone, so the schema is guaranteed to be present. For faster startup, some teams pre-load the schema onto a template_db and clone from that — pytest-postgresql supports both patterns via template_dbname.
test_schema_loaded proves the schema arrived; test_insert_and_select proves the DB is writable and reads back. Both are isolated — the second test starts with an empty orders table even though the first test would have inserted something.
Compared to a dockerised Postgres, this pattern skips the Docker daemon entirely — ~500 ms faster per pytest session and works on runners where Docker-in-Docker isn't available (some CI providers).

Output.

Test	Startup	Body	Teardown	Total
test_schema_loaded	~1 s (session, first only) + ~100 ms (clone)	~10 ms	~30 ms	~1.15 s
test_insert_and_select	~100 ms (clone)	~15 ms	~30 ms	~145 ms
(subsequent tests)	~100 ms (clone)	~10-50 ms	~30 ms	~140-180 ms

Rule of thumb. For Layer-3 SQL-emitting tests, prefer pytest-postgresql over a dockerised Postgres — it's faster to start, easier to reason about, and works on runners without Docker access. Save containers for Layer 4.

Worked example — Testcontainers Kafka + a smoke consumer test

Detailed explanation. A test that produces a message to Kafka and consumes it back, using testcontainers-python to spin up a single Kafka container. This is the Layer-4 pattern for CDC / event-driven code where the interaction with Kafka is the thing under test. Walk through the fixture + test.

Container. KafkaContainer("confluentinc/cp-kafka:7.6.0").
Fixture scope. Session — one Kafka broker for all Kafka tests in the run.
Per-test isolation. Each test creates its own topic named test_<uuid>.
Assertion. Produce a message; consume with a fresh consumer; assert value matches.

Question. Write the Kafka fixture + a smoke test that round-trips a message through a per-test topic.

Input.

Component	Value
Kafka image	confluentinc/cp-kafka:7.6.0
Fixture scope	session
Isolation	per-test topic
Producer / consumer lib	kafka-python

Code.

# tests/conftest.py — Kafka session fixture
import pytest
import uuid
from testcontainers.kafka import KafkaContainer


@pytest.fixture(scope="session")
def kafka_container():
    """Spin up one Kafka broker per pytest run."""
    with KafkaContainer(image="confluentinc/cp-kafka:7.6.0") as k:
        # KafkaContainer.get_bootstrap_server() blocks until ready
        yield k


@pytest.fixture
def kafka_topic(kafka_container):
    """Per-test unique topic; consumers use fresh group-id."""
    topic = f"test_{uuid.uuid4().hex[:8]}"
    yield topic
    # (topic auto-cleaned when broker exits; explicit delete optional)

# tests/integration/test_kafka_roundtrip.py — smoke test
import json
from kafka import KafkaProducer, KafkaConsumer


def test_kafka_produce_and_consume(kafka_container, kafka_topic):
    """Produce one JSON message; consume it back; assert equality."""
    bootstrap = kafka_container.get_bootstrap_server()

    # 1. Produce
    producer = KafkaProducer(
        bootstrap_servers=[bootstrap],
        value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    )
    payload = {"order_id": 42, "status": "placed"}
    producer.send(kafka_topic, payload)
    producer.flush()
    producer.close()

    # 2. Consume from earliest
    consumer = KafkaConsumer(
        kafka_topic,
        bootstrap_servers=[bootstrap],
        auto_offset_reset="earliest",
        group_id=f"grp_{kafka_topic}",
        consumer_timeout_ms=5_000,          # stop after 5 s of no messages
        value_deserializer=lambda v: json.loads(v.decode("utf-8")),
    )
    messages = list(consumer)
    consumer.close()

    # 3. Assert
    assert len(messages) == 1
    assert messages[0].value == payload

Step-by-step explanation.

KafkaContainer at session scope pulls the Kafka image (once per CI run — cached by Docker), starts the broker, and blocks on get_bootstrap_server() until the broker accepts client connections. This is the wait-for-ready idiom baked into testcontainers.
kafka_topic at function scope generates a per-test topic name using uuid.uuid4().hex[:8] — the 8-char hex is short enough to be readable and long enough to collision-resist across 1000+ concurrent test runs.
The producer serialises Python dicts as JSON bytes and sends one message. producer.flush() blocks until the broker acks; producer.close() shuts down cleanly. Both are necessary to guarantee the message is durable before the consumer starts.
The consumer uses a per-topic group-id (grp_<topic>) and auto_offset_reset="earliest" — this guarantees a fresh consumer group with no committed offset reads from the start of the topic. consumer_timeout_ms=5_000 ensures the list(consumer) returns even if no more messages arrive.
The assertion is pure equality on the deserialised value. Because the topic is per-test, there's no way for a previous test's messages to leak in — the count is deterministic.

Output.

Step	Duration	Detail
Kafka startup (session, first test)	~8 s	image pull cached, broker init
kafka_topic creation	<1 ms	uuid gen only
Producer send + flush	~50 ms	one message
Consumer poll (5s timeout)	~5 s	waits for stall
Test total (first)	~13 s	mostly Kafka startup
Test total (subsequent)	~5.1 s	just consumer stall

Rule of thumb. For Kafka tests, use testcontainers-python's KafkaContainer at session scope; give every test its own uuid-suffixed topic; give every consumer a matching group-id. This is the cheapest correct pattern.

Worked example — Airflow DAG integration test spinning up scheduler + Postgres

Detailed explanation. An Airflow DAG that reads from a source table and writes to a target table needs to be tested end-to-end: metadata DB running, scheduler running, DAG file discovered, task instances actually executing. DockerCompose fixture with an Airflow compose file is the standard pattern. Walk through the yml + fixture + test.

Compose services. postgres (metadata) + airflow-init + airflow-scheduler + airflow-webserver.
Fixture scope. Session — one Airflow up per test session; every DAG test triggers via CLI.
Test isolation. Each test triggers a fresh DAG run with a unique dag_run_id.
Assertion. After airflow dags trigger, poll airflow tasks state until success/failure.

Question. Write the compose fixture + a DAG test that triggers ingest_daily_orders and asserts success.

Input.

Component	Value
Compose file	docker-compose.test.yml
Services	postgres, airflow-scheduler, airflow-webserver
Trigger	airflow dags trigger ingest_daily_orders --run-id test_
Poll	airflow tasks state ... every 2 s, up to 60 s

Code.

# docker-compose.test.yml — minimal Airflow stack for tests
version: "3.9"

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      timeout: 5s
      retries: 5

  airflow-init:
    image: apache/airflow:2.9.2
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    entrypoint: /bin/bash
    command: -c "airflow db init && airflow users create --role Admin --username admin --password admin --email a@b.c --firstname a --lastname b"

  airflow-scheduler:
    image: apache/airflow:2.9.2
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    environment:
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    volumes:
      - ./dags:/opt/airflow/dags
    command: scheduler

# tests/conftest.py — Airflow compose fixture
import pytest
from testcontainers.compose import DockerCompose


@pytest.fixture(scope="session")
def airflow_stack():
    """Spin up Airflow + Postgres via docker-compose."""
    with DockerCompose(
        ".",
        compose_file_name="docker-compose.test.yml",
        pull=True,
    ) as compose:
        # Wait for the scheduler to actually parse the DAGs
        compose.wait_for("http://localhost:8080/health")  # webserver
        yield compose

# tests/integration/test_ingest_dag.py — end-to-end DAG test
import subprocess
import time
import uuid


def _run(cmd: list[str]) -> str:
    """Shell out to airflow via docker exec on the scheduler container."""
    full = ["docker", "exec", "airflow-scheduler"] + cmd
    return subprocess.check_output(full).decode()


def test_ingest_daily_orders_dag_succeeds(airflow_stack):
    """Trigger ingest_daily_orders; poll for success; assert green."""
    run_id = f"test_{uuid.uuid4().hex[:8]}"

    # 1. Trigger
    _run(["airflow", "dags", "trigger", "ingest_daily_orders",
          "--run-id", run_id])

    # 2. Poll state up to 60 s
    deadline = time.time() + 60
    state = "queued"
    while time.time() < deadline and state not in ("success", "failed"):
        time.sleep(2)
        out = _run(["airflow", "dags", "state", "ingest_daily_orders", run_id])
        state = out.strip()

    # 3. Assert
    assert state == "success", f"DAG {run_id} ended in {state}"

Step-by-step explanation.

The compose file uses Airflow's official image with LocalExecutor — no Celery, no Redis, minimum surface for a test stack. The metadata DB is a per-test-run Postgres; the scheduler picks up DAGs from a mounted ./dags directory.
airflow-init runs airflow db init + creates an admin user, then exits. depends_on: condition: service_completed_successfully on the scheduler ensures the init has finished before the scheduler starts.
The airflow_stack fixture wraps DockerCompose — the with block handles docker-compose up -d on entry and docker-compose down -v on exit. pull=True ensures images are pulled once per session.
The test triggers a DAG run via airflow dags trigger executed inside the scheduler container (via docker exec). The run-id is uuid-suffixed for per-test isolation.
The polling loop calls airflow dags state every 2 seconds up to a 60-second deadline. success or failed breaks the loop; the test assertion pins the required outcome.

Output.

Step	Duration	State
Compose startup (session first)	~30 s	postgres, init, scheduler, webserver
DAG trigger	~200 ms	subprocess exec
Task execution	~5-20 s	depends on DAG
Poll loop	~5-20 s	2 s poll interval
Test total (first)	~55 s	mostly compose startup
Test total (subsequent)	~25 s	just trigger + poll

Rule of thumb. For Airflow DAG integration tests, use DockerCompose at session scope with a compose file that pins Airflow 2.9+ and Postgres 15+. Every test gets a fresh run_id; the scheduler+worker are shared. Session-scope amortises the ~30 s compose startup across all DAG tests.

Senior interview question on pytest docker-compose integration testing

A senior interviewer might ask: "You have a Layer-4 integration suite with 40 tests, each currently spinning up its own docker-compose stack (Postgres + Kafka + MinIO) at ~15 s startup, so the suite takes 10 minutes. You need to bring it under 3 minutes. Walk me through the session-scoped compose fixture, the per-test schema/topic isolation pattern, the wait-for-ready idiom, and how you keep the pattern xdist-safe."

Solution Using session-scoped `DockerCompose` + per-test schema + wait-for-ready + xdist-safe port allocation

# 1. docker-compose.test.yml — the full stack, one project per xdist worker
version: "3.9"

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: test
    ports:
      - "5432"                          # published, port assigned by Docker
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "test"]
      interval: 3s
      retries: 10

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS:            PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT:    zookeeper:2181
    depends_on: [zookeeper]
    ports:
      - "9092"

  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment: { ZOOKEEPER_CLIENT_PORT: 2181 }

  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    ports:
      - "9000"
      - "9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 3s
      retries: 10

# 2. tests/conftest.py — session compose fixture with xdist safety
import os
import pytest
import uuid
from testcontainers.compose import DockerCompose


@pytest.fixture(scope="session")
def compose_stack(worker_id):
    """One compose project per xdist worker; session-scoped."""
    project = f"test_{worker_id}"       # 'master' when serial; 'gw0', 'gw1', ...
    os.environ["COMPOSE_PROJECT_NAME"] = project

    with DockerCompose(
        ".",
        compose_file_name="docker-compose.test.yml",
        pull=True,
        wait=True,                      # wait for healthchecks
    ) as compose:
        # Belt-and-braces: also block on our own SELECT 1
        pg_port = int(compose.get_service_port("postgres", 5432))
        _wait_postgres_ready("localhost", pg_port)

        yield compose


def _wait_postgres_ready(host, port, timeout=30):
    """Retry SELECT 1 until success or timeout."""
    import time, psycopg2
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            with psycopg2.connect(host=host, port=port, user="test",
                                  password="test", dbname="test") as c:
                with c.cursor() as cur:
                    cur.execute("SELECT 1")
                    if cur.fetchone() == (1,):
                        return
        except Exception:
            time.sleep(0.5)
    raise RuntimeError("postgres never became ready")

# 3. Per-test isolation — new schema per test
@pytest.fixture
def isolated_pg(compose_stack):
    """Yield a connection to a per-test schema in the shared Postgres."""
    import psycopg2
    port = int(compose_stack.get_service_port("postgres", 5432))
    schema = f"t_{uuid.uuid4().hex[:8]}"

    conn = psycopg2.connect(host="localhost", port=port,
                            user="test", password="test", dbname="test")
    with conn.cursor() as cur:
        cur.execute(f'CREATE SCHEMA "{schema}"')
        cur.execute(f'SET search_path TO "{schema}"')
        # (schema DDL for this test)
        cur.execute("""
            CREATE TABLE orders(id BIGSERIAL PRIMARY KEY, cents BIGINT)
        """)
    conn.commit()

    yield conn

    # Teardown — drop the whole schema
    with conn.cursor() as cur:
        cur.execute(f'DROP SCHEMA "{schema}" CASCADE')
    conn.commit()
    conn.close()


@pytest.mark.integration
def test_ingest_writes_orders(isolated_pg):
    """Real Postgres, per-test schema, no leakage."""
    from mypkg.ingest import ingest_batch
    ingest_batch([("A", 100), ("B", 200)], conn=isolated_pg)
    with isolated_pg.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM orders")
        assert cur.fetchone()[0] == 2

Step-by-step trace.

Step	Before (per-test compose)	After (session + per-schema)
Compose startups	40 × ~15 s = 10 min	1 × ~15 s per worker
Test body	40 × ~1 s = 40 s	40 × ~1 s = 40 s
Schema create/drop	none	40 × ~30 ms = ~1.2 s
xdist worker safety	port-collision risk	COMPOSE_PROJECT_NAME per worker
Total (4-worker xdist)	~4 min	~1 min
Wait-for-ready	ad-hoc sleeps	health checks + SELECT 1 retry

After deployment, the same 40 tests run in ~1 minute on 4-worker xdist, share one compose stack per worker, and every test isolates via a per-test schema. COMPOSE_PROJECT_NAME=test_<worker_id> ensures each xdist worker gets its own container network with distinct published ports — no collision.

Output:

Metric	Before	After
Runtime (serial)	10 min	~2 min
Runtime (4-worker xdist)	~4 min (port collisions)	~1 min (isolated)
Compose startups	40	1 per worker (4 total)
Test isolation	full (own stack)	full (own schema)
Wait-for-ready	absent (flaky)	health check + SELECT 1
Memory pressure	40 × 2 GB stacks	1 × 2 GB per worker (~8 GB)

Why this works — concept by concept:

Session-scoped DockerCompose fixture — one compose stack per pytest session (per xdist worker). Amortises ~15 s of startup across all Layer-4 tests. The wait=True argument makes testcontainers-python block on the healthcheck-defined ready conditions.
Per-test schema isolation — each test creates its own Postgres schema (CREATE SCHEMA t_<uuid>) and sets search_path; all its DDL and DML live under that schema. Cleanup is one DROP SCHEMA CASCADE. This gives per-test isolation on top of a shared DB.
COMPOSE_PROJECT_NAME per worker — worker_id is pytest-xdist's per-worker identifier (gw0, gw1, ...). Setting the project name isolates each worker's compose stack — separate container names, separate networks, no port collision even on the same host.
Wait-for-ready idiom — compose healthchecks + a belt-and-braces SELECT 1 retry loop. The healthcheck handles ~95% of races; the SELECT 1 retry catches the last 5% (schema not yet migrated, etc).
Cost — one compose stack per xdist worker (~2 GB RAM), ~15 s startup amortised, ~30 ms per test for schema create/drop. The eliminated cost is 40 × ~15 s = 10 minutes of compose startups. Net O(1) startup per worker versus O(N) startup per test. Volume cleanup is automatic via docker-compose down -v inside the context manager's exit.

ETL
Topic — etl
ETL integration and end-to-end pipeline tests

Practice →

API Topic — api-integration API integration testing patterns

Practice →

5. CI wiring — coverage, xdist parallelism, flaky-test discipline

`pytest -n auto --cov --cov-fail-under=85` is the two-line CI recipe every DE team eventually converges on

The mental model in one line: pytest for data engineering in CI is the discipline of running the four-layer suite in parallel via pytest-xdist, gating merges on branch-coverage via pytest-cov with a --cov-fail-under threshold, uploading --junit-xml to a flaky-test dashboard, and quarantining known-flaky tests with pytest-rerunfailures plus a nightly retriage job — the combination that turns a 25-minute serial suite into a 7-minute PR gate that catches regressions the day the code is written. Every senior data engineer has inherited a CI that runs pytest serially, has no coverage gate, and ignores flakes; and every senior data engineer has converted that same CI into a fast, gated, parallel suite over the course of a two-week focus block that pays for itself in developer time within the first month.

The three-pillar CI recipe for DE pytest suites.

Pillar 1 — xdist parallelism. pytest -n auto --dist=loadgroup spawns one worker per CPU; distributes tests to workers by --xdist-group marker for fixture affinity. This is the raw-speed win: an N-core runner runs the suite ~N× faster (minus startup + coordination overhead).
Pillar 2 — coverage gate. pytest --cov=mypkg --cov-branch --cov-fail-under=85 measures line-and-branch coverage, fails the run if coverage drops below 85%. This is the anti-regression gate — someone adding untested code breaks the build.
Pillar 3 — flaky discipline. @pytest.mark.flaky(reruns=2) (from pytest-rerunfailures) auto-reruns failing tests up to twice; combined with a quarantine marker (@pytest.mark.flaky_quarantined) and a nightly retriage job, this keeps CI green without hiding real failures.

pytest-cov config — the four settings that matter.

--cov=mypkg. Measure coverage on the mypkg package. Use one flag per package; don't measure tests/ itself.
--cov-branch. Include branch coverage — captures whether both sides of every if have been executed. Line coverage alone hides "we tested the happy path but not the exception branch".
--cov-report=term-missing:skip-covered. Prints uncovered line numbers in the terminal but skips fully-covered files — the report you actually read.
--cov-fail-under=85. The gate. Below 85% (or whatever threshold you commit to) the run fails. Ratchet upward as the suite matures — start at 70%, aim for 90%.

pytest-xdist distribution modes — pick the one that matches your fixture graph.

--dist=load (default). Round-robin tests to workers as they become idle. Best for tests with no shared fixtures.
--dist=loadscope. Group tests by file / class / module. Ensures all tests in a file land on the same worker — useful when a module fixture is expensive.
--dist=loadgroup. Group tests by the @pytest.mark.xdist_group("name") marker. The most flexible — pin all tests that share a Kafka topic to one worker, for example.
--dist=each. Run every test on every worker. Only for very specific benchmarking / matrix cases; almost never right.

Flaky-test discipline — the four-step protocol.

Step 1 — instrument. Add --junit-xml=junit.xml to every CI run; upload the artifact.
Step 2 — surface. A nightly job parses the junit XMLs from the last N runs and flags tests with pass rate <99%.
Step 3 — quarantine. Flagged tests get @pytest.mark.flaky(reruns=2) immediately (fixes the visible CI break) and a @pytest.mark.flaky_quarantined marker for tracking.
Step 4 — retriage. A weekly ticket assigns each quarantined test to an owner; owner has 2 weeks to fix-or-delete. No test stays quarantined forever.

Common interview probes on pytest CI.

"How do you measure coverage?" — pytest-cov --cov=pkg --cov-branch --cov-fail-under=N.
"How do you parallelise safely?" — pytest-xdist -n auto --dist=loadgroup with per-worker fixtures.
"What's your flaky-test policy?" — rerun with pytest-rerunfailures, quarantine, weekly retriage.
"How do you keep DB fixtures xdist-safe?" — per-worker Postgres process (session-scoped in each worker) or per-worker COMPOSE_PROJECT_NAME.

Worked example — GitHub Actions matrix with coverage upload

Detailed explanation. A canonical GitHub Actions workflow: matrix over Python 3.11 and 3.12, install deps, run pytest with xdist + coverage, upload junit XML + coverage HTML as artifacts, gate on --cov-fail-under=85. Walk through the yaml.

Matrix. Python 3.11 and 3.12 in parallel.
Runner. ubuntu-24.04 (has Docker installed).
Cache. pip cache keyed on requirements hash.
Coverage upload. GitHub artifact + optional Codecov push.

Question. Write the .github/workflows/tests.yml that runs the four-layer pytest suite with xdist and coverage.

Input.

Parameter	Value
Matrix	Python 3.11, 3.12
xdist workers	auto (CPU count)
Coverage threshold	85%
Junit output	junit.xml per matrix cell
Timeout	15 min

Code.

# .github/workflows/tests.yml
name: tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  pytest:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip

      - name: Install
        run: |
          pip install -e '.[test]'
          pip install pytest-cov pytest-xdist pytest-rerunfailures \
                      pytest-postgresql testcontainers

      - name: Install Postgres
        run: sudo apt-get install -y postgresql-15

      - name: Run pytest
        run: |
          pytest \
            -n auto \
            --dist=loadgroup \
            --cov=mypkg \
            --cov-branch \
            --cov-report=xml \
            --cov-report=term-missing:skip-covered \
            --cov-fail-under=85 \
            --junit-xml=junit-${{ matrix.python-version }}.xml \
            -v \
            tests/

      - name: Upload junit
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-${{ matrix.python-version }}
          path: junit-${{ matrix.python-version }}.xml

      - name: Upload coverage HTML
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.python-version }}
          path: coverage.xml

Step-by-step explanation.

The strategy.matrix runs the workflow twice — once per Python version — in parallel. fail-fast: false lets both cells complete even if one fails; you get coverage data from both.
actions/setup-python@v5 with cache: pip uses GitHub's built-in pip cache, shaving ~30 s off cold installs.
pytest -n auto uses pytest-xdist to fan out to CPU count workers (ubuntu-24.04 default = 4 vCPUs). --dist=loadgroup respects @pytest.mark.xdist_group markers for fixture affinity.
--cov-fail-under=85 fails the step if branch coverage drops below 85%. The step exits non-zero; GitHub marks the check as failed; the PR is blocked from merge.
if: always() on the artifact-upload steps ensures junit + coverage are uploaded even when the pytest step failed — you get the data to diagnose the failure. This is a small detail that saves hours of debugging.

Output.

Cell	Duration	Tests run	Coverage	Result
Python 3.11	~5 min	1,290	87%	pass
Python 3.12	~5 min	1,290	87%	pass
Total wall clock	~5 min (parallel cells)	—	—	pass
Artifacts	junit-3.11, junit-3.12, coverage-3.11, coverage-3.12	—	—	—

Rule of thumb. For DE pytest CI, use a Python-version matrix with cache: pip, pytest -n auto, --cov-fail-under=<N>, and if: always() on artifact uploads. This is the minimum viable modern setup.

Worked example — xdist + docker-compose shared across workers

Detailed explanation. The subtle problem: session-scoped fixtures are per-worker in xdist, not per-session. If your compose stack is session-scoped, it starts N times (once per worker). Sometimes that's what you want (isolation); sometimes you want the compose stack shared across all workers (for cost reasons). The pytest-xdist idiom tmp_path_factory + a lockfile solves the "shared across workers" case. Walk through the pattern.

Problem. @pytest.fixture(scope="session") on compose_stack starts one compose stack per worker.
Solution. Use a filesystem lock on a shared tmp_path_factory directory to elect one worker to start compose; other workers block until ready.
When. When compose startup is >30 s and you're xdist-parallelising past 4 workers.

Question. Write the shared-compose fixture using a filesystem lock so N workers use one compose stack.

Input.

Component	Value
Lock file	tmp_path_factory / "compose.lock"
Ready file	tmp_path_factory / "compose.ready"
Compose stack	docker-compose.test.yml
Election	one worker starts; others poll ready file

Code.

# tests/conftest.py — cross-worker shared compose fixture
import os
import time
import pytest
import fcntl
from pathlib import Path
from testcontainers.compose import DockerCompose


@pytest.fixture(scope="session")
def compose_stack(tmp_path_factory, worker_id):
    """One compose stack shared across all xdist workers."""
    # tmp_path_factory.getbasetemp() is shared across workers
    root = tmp_path_factory.getbasetemp().parent
    root.mkdir(parents=True, exist_ok=True)
    lock_path  = root / "compose.lock"
    ready_path = root / "compose.ready"

    if worker_id == "master":
        # Serial pytest — no election needed
        with DockerCompose(".", compose_file_name="docker-compose.test.yml") as c:
            c.wait_for("http://localhost:9001/minio/health/ready")
            yield c
        return

    # Parallel — acquire lock
    with open(lock_path, "w") as lock_fp:
        try:
            fcntl.flock(lock_fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
            # We are the elected worker — start compose
            os.environ["COMPOSE_PROJECT_NAME"] = "test_shared"
            compose = DockerCompose(
                ".", compose_file_name="docker-compose.test.yml"
            ).start()
            _wait_ready(compose)
            ready_path.write_text("ok")
            yield compose
            compose.stop()
            ready_path.unlink(missing_ok=True)

        except BlockingIOError:
            # Some other worker started it; wait for ready file
            deadline = time.time() + 120
            while not ready_path.exists() and time.time() < deadline:
                time.sleep(0.5)
            if not ready_path.exists():
                raise RuntimeError("compose never came ready")

            # Yield a lightweight reference (compose is up)
            yield _AttachedCompose(project_name="test_shared")


def _wait_ready(compose):
    """Poll all healthchecks; block until green."""
    # (implementation elided; use compose.wait_for or per-service polls)
    pass


class _AttachedCompose:
    """Read-only view of a compose stack another worker owns."""
    def __init__(self, project_name):
        self.project_name = project_name
    def get_service_port(self, svc, port):
        import subprocess
        out = subprocess.check_output(
            ["docker", "compose", "-p", self.project_name, "port", svc, str(port)]
        )
        return out.decode().strip().split(":")[-1]

Step-by-step explanation.

When pytest runs under xdist, worker_id is gw0, gw1, ... — one per worker. When pytest runs serial, worker_id is master. The fixture branches on this.
tmp_path_factory.getbasetemp() returns a per-worker temp dir, but its .parent is the shared base (/tmp/pytest-of-user). Using the shared parent as the lock location lets multiple workers coordinate.
fcntl.flock(..., LOCK_EX | LOCK_NB) acquires an exclusive lock non-blockingly. Only one worker succeeds; the others get BlockingIOError and fall into the "wait for ready" branch.
The elected worker starts compose, waits for readiness, writes a compose.ready sentinel file, and yields the live compose object. At teardown, it stops compose and cleans the sentinel.
Non-elected workers spin-wait on ready_path.exists() for up to 120 s. Once ready, they yield an _AttachedCompose proxy that lets tests query service ports via docker compose port — no actual container control.

Output.

Worker	Elected?	Compose action	Wait
gw0	yes	start compose	writes ready file
gw1	no	attaches	waits ~15 s for ready
gw2	no	attaches	waits ~15 s for ready
gw3	no	attaches	waits ~15 s for ready

Rule of thumb. For very-expensive compose stacks (>30 s startup) and 4+ xdist workers, elect one worker to start compose and have others attach. For cheaper stacks (<15 s) or ≤2 workers, per-worker isolation is simpler and safer.

Worked example — flaky-test dashboard from `--junit-xml`

Detailed explanation. A tiny Python script that ingests the last N junit XMLs and builds a per-test pass-rate report. Flag anything with pass rate < 99% over the last 20 runs. Walk through the parser + report.

Input. junit-*.xml files from the last 20 CI runs.
Parser. xml.etree.ElementTree — each <testcase> element is one test outcome.
Aggregation. Group by test node id; compute pass rate = passed / total.
Report. Sorted by pass rate ascending; flag < 99%.

Question. Write the parser + reporter for the flaky-test dashboard.

Input.

Component	Value
Input glob	reports/junit-*.xml
Test key	classname::name
Failure marker	or child
Threshold	99%

Code.

# tools/flaky_dashboard.py
"""Compute per-test pass rates from junit XMLs and flag flakes."""
from __future__ import annotations
import glob
import xml.etree.ElementTree as ET
from collections import defaultdict


def parse_junit(path: str) -> list[tuple[str, bool]]:
    """Return [(test_id, passed?), ...] from one junit XML."""
    root = ET.parse(path).getroot()
    out: list[tuple[str, bool]] = []
    for tc in root.iter("testcase"):
        tid = f"{tc.get('classname', '')}::{tc.get('name', '')}"
        failed = any(child.tag in ("failure", "error") for child in tc)
        out.append((tid, not failed))
    return out


def compute_pass_rates(paths: list[str]) -> dict[str, float]:
    """Aggregate pass rates over all runs."""
    tallies: dict[str, list[bool]] = defaultdict(list)
    for p in paths:
        for tid, passed in parse_junit(p):
            tallies[tid].append(passed)
    return {tid: sum(v) / len(v) for tid, v in tallies.items() if len(v) >= 5}


def flag_flakes(rates: dict[str, float], threshold: float = 0.99) -> list[tuple[str, float]]:
    """Return tests below the pass-rate threshold, worst first."""
    flakes = [(tid, r) for tid, r in rates.items() if r < threshold]
    flakes.sort(key=lambda x: x[1])
    return flakes


def main():
    paths = glob.glob("reports/junit-*.xml")
    rates = compute_pass_rates(paths)
    flakes = flag_flakes(rates)
    if not flakes:
        print(f"OK — all {len(rates)} tests >= 99% pass rate")
        return
    print(f"FLAKY — {len(flakes)} tests below 99% over {len(paths)} runs:")
    print(f"  {'pass_rate':>10}  test_id")
    for tid, rate in flakes[:20]:
        print(f"  {rate * 100:>9.1f}%  {tid}")


if __name__ == "__main__":
    main()

Step-by-step explanation.

parse_junit walks the XML and extracts every <testcase>. A test passed iff it has no <failure> or <error> child. Skipped tests (which have <skipped>) are counted as passed — they didn't fail.
compute_pass_rates aggregates across all junit files. Tests seen fewer than 5 times are filtered out (insufficient data to call flake vs new). The pass rate is sum(passed) / len(runs).
flag_flakes returns tests below the threshold, sorted worst first — the flakiest tests appear at the top of the report and get owner attention first.
The main function glob-collects reports/junit-*.xml (uploaded by CI) and prints the top 20 offenders. Wire this into a nightly cron job that pushes the report to Slack or a dashboard.
To integrate with pytest-rerunfailures, this report drives which tests get @pytest.mark.flaky(reruns=2) added — a manual review by a triage owner turns "reported flake" into "quarantined test with a tracking ticket."

Output.

FLAKY — 3 tests below 99% over 20 runs:
  pass_rate  test_id
      75.0%  tests.integration.test_kafka::test_kafka_produce_and_consume
      85.0%  tests.integration.test_dag::test_ingest_daily_orders_dag_succeeds
      95.0%  tests.component.test_pg::test_bulk_status_update

Rule of thumb. For any DE test suite past ~200 tests, ship a flaky-test dashboard from --junit-xml. Manual pass-rate tracking doesn't scale; automated flagging + a weekly retriage keeps the suite honest.

Senior interview question on pytest CI and flaky-test discipline

A senior interviewer might ask: "Your team's pytest CI runs serially, has no coverage gate, and has 15 tests that flake ~10% of the time — devs have started -p 'not integration'-skipping the suite on their branches. Walk me through the two-week focus block that gets you to pytest -n auto --cov-fail-under=85 with a quarantine + retriage process, and quantify the CI budget savings."

Solution Using xdist + coverage gate + rerunfailures + quarantine markers + nightly retriage

# 1. pytest.ini — strict-marker gate + coverage + rerunfailures defaults
[pytest]
minversion = 8.0
addopts =
    -ra
    -q
    --strict-markers
    --strict-config
    --cov=mypkg
    --cov-branch
    --cov-report=term-missing:skip-covered
    --cov-report=xml
    --cov-fail-under=85
    --junit-xml=junit.xml

markers =
    unit:              base-layer pure-function tests
    contract:          schema / pydantic contract tests
    component:         Layer-3 pytest-postgresql tests
    integration:       Layer-4 docker-compose tests
    flaky:             tests known to flake; get reruns=2
    flaky_quarantined: assigned to an owner for fix-or-delete

testpaths = tests

# 2. tests/conftest.py — auto-apply reruns for @pytest.mark.flaky
def pytest_collection_modifyitems(config, items):
    """Attach reruns=2 to every test carrying the flaky marker."""
    for item in items:
        if item.get_closest_marker("flaky"):
            item.add_marker("flaky_quarantined")  # for tracking
            # pytest-rerunfailures picks up the reruns param

# 3. .github/workflows/tests.yml — parallel matrix + artifact uploads
name: tests
on: [push, pull_request]

jobs:
  pytest:
    runs-on: ubuntu-24.04
    strategy:
      matrix:
        python: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ matrix.python }}, cache: pip }
      - run: pip install -e '.[test]'
      - run: sudo apt-get install -y postgresql-15
      - run: |
          pytest -n auto --dist=loadgroup --reruns 2 tests/
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-${{ matrix.python }}
          path: junit.xml

# 4. .github/workflows/flaky-retriage.yml — nightly retriage
name: flaky-retriage
on:
  schedule:
    - cron: "0 6 * * *"      # 06:00 UTC daily

jobs:
  retriage:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Download last 20 junit artifacts
        uses: actions/github-script@v7
        with:
          script: |
            const artifacts = await github.rest.actions.listArtifactsForRepo({
              owner: context.repo.owner, repo: context.repo.repo, per_page: 100,
            });
            // (download loop elided)
      - run: python tools/flaky_dashboard.py > flaky_report.txt
      - name: Post to Slack
        run: |
          curl -X POST $SLACK_WEBHOOK -d "text=$(cat flaky_report.txt)"
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}

# 5. Example flaky-quarantined test
import pytest

@pytest.mark.flaky
@pytest.mark.integration
def test_kafka_lag_below_threshold(kafka_container):
    """Known flaky under CPU pressure — reruns=2 attached automatically."""
    # ... test body ...
    assert measure_lag() < 100

Step-by-step trace.

Layer	Component	Effect
pytest.ini	strict-markers + coverage gate	fails PRs missing coverage or with unknown markers
conftest.py	@flaky → auto-quarantine marker	tracks quarantined tests without hiding them
tests.yml	xdist + reruns + junit upload	~5 min CI, junit stored per run
flaky-retriage.yml	nightly parse + Slack	flaky tests surfaced daily
Example test	@pytest.mark.flaky	rerun up to 2× on failure
Weekly	manual retriage of Slack digest	owners assigned; fix-or-delete in 2 weeks

After the two-week focus block, CI runtime drops from ~25 minutes serial to ~5 minutes parallel; coverage gate blocks PRs below 85%; flaky tests get reruns automatically and a Slack ping if they persist; the retriage ticket ensures nothing stays flaky forever. Dev unblock: teams can trust pytest -n auto as the pre-push check.

Output:

Metric	Before	After
CI runtime	25 min	5 min
Coverage gate	none	85% branch
Flaky visibility	ad-hoc	nightly Slack digest
Quarantine tracking	manual	`@pytest.mark.flaky_quarantined`
Devs skipping suite	~30%	~0% (fast enough to always run)
PRs blocked by real bugs	~40% caught	~85% caught

Why this works — concept by concept:

xdist parallelism — -n auto --dist=loadgroup scales test throughput linearly with CPU count. loadgroup distributes tests grouped by xdist_group marker for fixture affinity, avoiding container-collision.
Coverage gate — --cov-fail-under=85 is the anti-regression ratchet. PRs adding untested code fail; PRs improving coverage pass. The gate is what makes coverage a leading indicator instead of a lagging one.
pytest-rerunfailures — @pytest.mark.flaky auto-reruns; --reruns 2 at the CLI applies globally. Buys time to fix flakes without red CI blocking merges. Combined with a quarantine marker for tracking.
Junit dashboard — --junit-xml per run + a nightly parser + Slack ping = flaky tests get surfaced daily. No test hides in a "we always retry it" limbo.
Cost — pytest-xdist (one process per CPU), pytest-cov (~3% CPU overhead), pytest-rerunfailures (near-zero), junit XML (~1 KB per test). The eliminated cost is 20 minutes of serial CI per PR + hours of debugging flaky failures + weeks of hidden coverage regressions. Net: O(1) developer time to fix a flake vs O(N) developer time to debug it silently. Two-week focus block, ~1 dev-week of net cost, pays back in ~1 month.

Python
Topic — exception-handling
Exception handling and CI-hardening tests

Practice →

Python
Topic — python
Python testing and CI-workflow problems

Practice →

Cheat sheet — pytest recipes for data engineering

Four-layer test pyramid. Layer 1 unit (pure functions, no I/O, sub-ms per test, hundreds of tests, run on save), Layer 2 contract (dbt schema tests, pydantic validation, ~1 ms per test), Layer 3 component-with-DB (pytest-postgresql template clone, ~100 ms per test), Layer 4 integration-with-compose (testcontainers-python session compose, ~5-30 s per test, run on PR + nightly). The pyramid shape — thousands of Layer 1, dozens of Layer 4 — is what keeps CI under 8 minutes for a 1000+ test data-platform suite.
Fixture scope decision matrix. Use function scope for anything that mutates during the test (fresh DB row, tmp file, monkeypatched env). Use session scope for anything with >1 s setup (Postgres process, compose stack, dbt manifest). Use module scope for tests-in-one-file that share a large seed. Avoid class scope in modern pytest; avoid autouse=True except for cross-cutting concerns like UTC timezone or logger reset — never for I/O fixtures.
pytest-postgresql template-clone recipe. postgresql_proc = factories.postgresql_proc(port=None, unixsocketdir="/tmp", postgres_options="-c fsync=off -c synchronous_commit=off -c full_page_writes=off") at session scope; postgresql = factories.postgresql("postgresql_proc", dbname="test_db", load=[SCHEMA_SQL]) at function scope. The port=None is what makes it xdist-safe; the durability flags cut commit latency by ~5×; the template clone is a Postgres-native file-block copy at ~100 ms per test.
Factory-as-fixture template. @pytest.fixture def make_order(postgresql): def _make(customer_id=1, total_cents=1000, status="pending"): with postgresql.cursor() as cur: cur.execute("INSERT INTO orders(...) VALUES(...) RETURNING id", (...)); return Order(id=..., ...); return _make. Every DE table gets a factory; factories close over the DB fixture; tests call the factory zero or more times per test with defaulted-and-overridable kwargs; cleanup is automatic via the template-clone drop at teardown.
@pytest.mark.parametrize with ids=. Always supply ids= for parametrizations with more than 3 rows. Prefer readable strings (ids=["null-in", "empty-str", "valid-with-punct"]) or ids=lambda c: c.name when parametrizing over a dataclass. Never accept the default test_foo[0] / test_foo[test_foo0] IDs — they turn every failing test into a debugging tax.
pytest_generate_tests for dynamic axes. def pytest_generate_tests(metafunc): if "table_name" in metafunc.fixturenames: metafunc.parametrize("table_name", discover_tables(), ids=discover_tables()). Use for manifest-driven axes (every model in the dbt manifest, every table in information_schema, every YAML config in configs/). Amplifies one test function into N tests without hand-writing them; new axis entries are automatically covered.
testcontainers-python compose fixture. @pytest.fixture(scope="session") def compose_stack(worker_id): os.environ["COMPOSE_PROJECT_NAME"] = f"test_{worker_id}"; with DockerCompose(".", compose_file_name="docker-compose.test.yml", pull=True, wait=True) as compose: compose.wait_for("http://localhost:9001/minio/health/ready"); yield compose. The per-worker project name is what makes xdist safe (isolated networks, no port collision); wait=True blocks on healthchecks; the with statement handles docker-compose down -v at teardown.
Wait-for-ready idiom. Never trust docker-compose up -d alone — the containers are started, not ready. Use compose.wait_for("http://.../health") for HTTP-endpoint services (MinIO, Airflow webserver, Elasticsearch) and a SELECT 1 retry loop with tenacity for raw Postgres or Kafka. Timeout at 30 s; log clearly on failure ("postgres never became ready"); belt-and-braces both patterns for high-value fixtures.
pytest-xdist distribution modes. -n auto for CPU-count workers; --dist=load default is round-robin per test; --dist=loadgroup respects @pytest.mark.xdist_group("name") for fixture affinity (pin all Kafka-consumer tests to one worker); --dist=loadscope groups by file/class. Session-scoped fixtures are per-worker, so each worker gets its own Postgres process — this is the correct default; only elect a shared-across-workers stack via a filesystem lock when compose startup >30 s.
pytest-cov config. --cov=mypkg --cov-branch --cov-report=term-missing:skip-covered --cov-report=xml --cov-fail-under=85. Line-only coverage lies about happy-path bias; branch coverage catches "we tested the if but not the else". term-missing:skip-covered prints only files with gaps; xml report feeds Codecov / SonarQube. Start --cov-fail-under at 70%, ratchet to 85%, aim for 90% on the core packages; never lower the threshold.
pytest-rerunfailures flaky discipline. @pytest.mark.flaky(reruns=2) on the test (or CLI --reruns 2 globally); combined with a @pytest.mark.flaky_quarantined tracking marker (auto-applied via pytest_collection_modifyitems). A nightly job parses --junit-xml files, computes per-test pass rates, and posts <99% offenders to Slack. Weekly retriage: owner assigned; fix-or-delete deadline; no test stays quarantined forever.
monkeypatch + tmp_path isolation. For any test touching env vars, cwd, or filesystem writes, use monkeypatch.setenv("KEY", "value") (auto-reverts), monkeypatch.chdir(tmp_path) (auto-reverts), and tmp_path (auto-deletes). Never modify os.environ or os.chdir directly — global-state leakage between tests is the #2 source of "works on my machine" flakes.
GitHub Actions minimal recipe. strategy.matrix.python: ["3.11", "3.12"] on ubuntu-24.04; actions/setup-python@v5 with cache: pip; install postgresql-15 via apt for pytest-postgresql; pytest -n auto --dist=loadgroup --cov-fail-under=85 --junit-xml=junit.xml; if: always() on artifact uploads for junit + coverage. Wire the Slack notifier separately as a nightly cron job on the retriage workflow.
Airflow / dbt / Spark specific gotchas. Airflow DAG tests need the airflow-init service to have condition: service_completed_successfully before the scheduler starts. dbt tests need monkeypatch.chdir(project_root) because dbt_project.yml is discovered from cwd. Spark UDF tests can use a session-scoped SparkSession fixture with .master("local[2]") — never local[*] in tests because it stalls other xdist workers. All three: pin the version in requirements-test.txt; version-drift is a silent flaky-test source.

Frequently asked questions

What is pytest in one sentence for a data engineer?

Pytest for data engineering is the Python test framework that unifies four test layers under one runner: unit tests for pure transforms, contract tests for dbt/pydantic schemas, component tests for SQL-emitting code against a real Postgres via pytest-postgresql, and integration tests for end-to-end pipelines spun up via pytest docker-compose or pytest testcontainers. Every plugin senior DE teams reach for — pytest-xdist for parallelism, pytest-cov for coverage, pytest-rerunfailures for flaky discipline, pytest-postgresql for template DBs, testcontainers-python for Kafka/MinIO/Airflow stacks — plugs into pytest's fixture graph without ceremony, and the same pytest command runs on a laptop, in a Codespace, and on GitHub Actions with identical behaviour. It is the load-bearing tool that makes a 1000-test data-platform suite feasible.

Pytest fixtures vs unittest setUp — which do I pick in 2026?

Pick pytest fixtures for every greenfield project and every serious refactor. pytest fixtures compose (fixture A can request fixture B), have explicit scope (function / module / session), support yield-teardown (fixture body reads top-to-bottom like a contextmanager), and are parametrizable (@pytest.fixture(params=[...])). unittest.setUp / tearDown do none of those — they're per-class-method, imperative, and force you into class TestFoo(unittest.TestCase): boilerplate. The only reason to touch unittest in 2026 is if you inherit a legacy suite and the migration cost exceeds the improvement (rare; pytest can run unittest-style tests natively, so you can migrate one file at a time). Every senior DE codebase has converged on pytest fixtures because scope discipline is what makes the difference between a 2-minute test run and a 20-minute one — and only pytest exposes scope as a first-class primitive.

How do I parametrize a pytest test over dbt models?

Use pytest_generate_tests(metafunc) to discover models from the dbt manifest at collection time and inject them as a parameter. Concretely: implement def pytest_generate_tests(metafunc): if "dbt_model" in metafunc.fixturenames: models = [n["name"] for n in json.loads(Path("target/manifest.json").read_text())["nodes"].values() if n["resource_type"]=="model"]; metafunc.parametrize("dbt_model", models, ids=models). Then any test that names dbt_model as a parameter runs once per model with a readable ID like test_mart_has_row_count_gt_zero[fct_orders]. Adding a new dbt model automatically adds a new test — zero test-file changes required. This is the amplifier that turns one contract test into hundreds and lets you enforce invariants like "every mart has ≥1 row" or "every staging model has a _ingested_at column" across the entire warehouse.

What's the difference between docker-compose fixtures and testcontainers-python?

docker-compose fixtures (via testcontainers.compose.DockerCompose or the older pytest-docker-compose plugin) wrap a docker-compose.yml file — best when your test needs the full networked stack (Airflow scheduler + webserver + Postgres + Redis + Celery, or Kafka + Zookeeper + Schema Registry). testcontainers-python per-service (PostgresContainer, KafkaContainer, MinioContainer) instantiates one container at a time in Python code — best when your test needs one or two containers with programmatic setup. Both handle wait-for-ready health checks and clean teardown; both are pytest docker-compose-compatible in the sense that the underlying tech is Docker. Rule of thumb: if your docker-compose.yml has ≥3 services, use DockerCompose; if it's 1-2 services, use per-service testcontainers-python and skip the yaml file. Both patterns are session-scoped; both need per-test isolation via schemas (Postgres) or uuid-suffixed topics (Kafka).

How do I run pytest in parallel with pytest-xdist without breaking Postgres fixtures?

Two rules: (1) make the Postgres fixture per-worker via session scope + OS-allocated port (factories.postgresql_proc(port=None) for pytest-postgresql, or COMPOSE_PROJECT_NAME=test_{worker_id} for docker-compose); (2) make every DB itself per-test via template clone or per-test schema (CREATE SCHEMA t_<uuid>). Concretely, pytest -n auto --dist=loadgroup spawns one worker per CPU; each worker gets its own session-scoped Postgres process on its own port; each test in a worker clones a fresh DB from the template. Under 4-worker xdist on an 8-core machine, a 400-test suite goes from ~2 minutes serial to ~30 seconds — with full isolation, no shared state, no port collisions. The --dist=loadgroup mode respects @pytest.mark.xdist_group("name") markers so all tests sharing a Kafka topic can pin to one worker (fixture affinity). Never try to share a single Postgres process across workers via a filesystem lock unless you've measured that it's actually necessary — per-worker Postgres is the simpler, safer default.

How do I test an Airflow DAG with pytest?

Use a session-scoped DockerCompose fixture with a minimal Airflow compose file — postgres for metadata + airflow-init (runs airflow db init, exits) + airflow-scheduler + optionally airflow-webserver. The compose file uses depends_on: airflow-init: condition: service_completed_successfully so the scheduler waits for init to finish. Test code triggers a DAG run via subprocess.check_output(["docker", "exec", "airflow-scheduler", "airflow", "dags", "trigger", "my_dag", "--run-id", f"test_{uuid.uuid4().hex[:8]}"]), then polls airflow dags state my_dag <run_id> every 2 seconds up to a 60-second deadline. Assert on the final state (success or failed). For unit-testing individual tasks in isolation, use Airflow's Task.test() method (no compose needed) or the @task-decorator's .function() accessor to call the underlying Python directly. For end-to-end DAG behaviour where you need the scheduler + metadata DB, the docker-compose pattern is the standard; ~30-second startup amortised across all DAG tests via session scope keeps the suite fast enough to run on every PR.

Practice on PipeCode

Drill the Python practice library → for the pytest, fixture-graph, parametrize, and mocking problems senior interviewers love.
Rehearse on the ETL practice library → for the ingestion-pipeline testing, DB-fixture design, and integration-suite scenarios.
Sharpen the validation axis with the data-validation practice library → for the contract-test, schema-assertion, and dbt-model-verification patterns.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-layer testing decision matrix against real graded inputs.

Lock in pytest-for-DE muscle memory

Docs explain fixtures. PipeCode drills explain the decision — when session-scope makes the difference between a 2-minute and a 20-minute suite, when parametrize with readable ids beats copy-pasted tests, when testcontainers-python earns its place over a shell-out to docker-compose, when the coverage gate + rerunfailures + quarantine protocol turns a flaky suite green. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production testing trade-offs senior data engineers actually face.

Practice Python problems →
Practice ETL problems →

LLM Evaluation for Data Pipelines: LangSmith, TruLens, Ragas & Snowflake Cortex Search Ops

Gowtham Potureddi — Fri, 31 Jul 2026 04:28:51 +0000

llm evaluation for data pipelines is the load-bearing correctness discipline of the 2026 data stack — the difference between a RAG chatbot that quietly regresses for three weeks before a customer notices and a RAG chatbot whose retrieval quality is gated on every pull request. Every LLM call your pipeline makes — the summariser inside a nightly ingestion DAG, the classifier fronting a Kafka stream, the RAG chain answering support tickets, the SQL-generation copilot inside Snowsight — has failure modes that unit tests do not catch: hallucinations that look confident, retrieval misses that return the wrong policy document, prompt drift where a model swap flips the tone, cost explosions where a runaway agent burns a quarter's budget in one afternoon. Those failures do not raise exceptions; they degrade the output. The only way to catch them is to measure the output on every run and gate on the score.

This guide is the senior-DE / MLOps walkthrough you wished existed the first time an interviewer asked "walk me through your llm evaluation for data pipelines stack" or "how would you compare langsmith, trulens, and ragas?" or "how do you gate a RAG deployment on retrieval quality metrics?" It walks through the four axes every senior discussion converges on (groundedness, answer relevance, context precision / recall, latency + cost), the four production-hardened tools that ship those metrics in 2026 — LangSmith for trace-first eval and versioned datasets, TruLens for feedback functions and the rag evaluation triad, Ragas for reference-free metrics inside CI, and Snowflake Cortex Search Ops for eval at the semantic-search layer — and finishes each section with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data-validation practice library →, and sharpen the systems axis with the design practice library →.

On this page

Why LLM pipelines fail silently without eval — the four axes
LangSmith — trace-first eval, dataset versioning, hosted judges
TruLens — feedback functions and the RAG triad
Ragas — reference-free RAG metrics for CI
Snowflake Cortex Search Ops — eval at the semantic-search layer
Cheat sheet — LLM eval recipes
Frequently asked questions
Practice on PipeCode

1. Why LLM pipelines fail silently without eval — the four axes

Four axes, four measurement tools — the choice binds every downstream release gate

The one-sentence invariant: llm evaluation for data pipelines is the discipline of measuring model + retrieval output on every run against four independent axes — groundedness (did the answer cite the retrieved context?), answer relevance (did the answer address the user's question?), context precision / recall (did retrieval surface the right chunks?), and latency + cost — because unit tests cannot catch hallucination, retrieval miss, prompt drift, or budget blowup, and shipping without a numeric score on each axis is the modern equivalent of shipping without regression tests. The four axes are orthogonal: a RAG chain can be perfectly grounded in the retrieved context and still fail answer relevance because the retrieval was wrong; retrieval can be perfect and grounding low because the model paraphrased poorly; both can be high while latency p99 blows past your SLO or per-request cost triples on a model swap. Measuring one axis and calling it "LLM eval" is the single most common mistake senior candidates make in interviews.

The four axes senior interviewers actually probe.

Groundedness. Of every claim in the generated answer, what fraction is directly supported by the retrieved context? An answer that invents a fact the retrieved documents do not contain scores low on groundedness even if the fact is correct in the world. TruLens calls this "groundedness"; Ragas calls it "faithfulness"; both compute it as an LLM-judge score over (context, answer) pairs. This is the primary anti-hallucination metric and the first metric any RAG deployment should ship.
Answer relevance. Does the answer actually address the user's question? A perfectly grounded answer that ignores the question ("here is the entire retrieved policy document") scores low on answer relevance. Ragas measures this by having an LLM generate hypothetical questions the answer could address, then computing cosine similarity between those questions and the original query. High answer relevance means the answer stays on topic.
Context precision / recall. Of the top-k retrieved chunks, how many are relevant (precision), and of the relevant chunks in the corpus, how many appear in top-k (recall)? These are the classic IR metrics reborn for RAG. Ragas ships context_precision and context_recall; TruLens exposes "context relevance" as the top vertex of the RAG triad. Low precision means noisy retrieval; low recall means missing information.
Latency + cost. The pipeline axis. p50 / p95 / p99 latency per LLM call, tokens per call, dollars per call, tokens per user query end-to-end. A model swap that improves groundedness by 3 points but triples cost is a regression on this axis. Every eval framework should ship latency + cost alongside quality scores; if it doesn't, wire it up separately with a @traceable wrapper.

The 2026 reality — four tools, four sweet spots, they coexist.

LangSmith is the trace-first hosted platform: every LLM call is a Run inside a hierarchical Trace; datasets are versioned; experiments diff scores over the same dataset; hosted LLM-as-judge evaluators run in the LangSmith cloud. Sweet spot: teams already using LangChain / LangGraph who want a hosted dashboard and dataset store.
TruLens is the local-first open-source eval library: TruChain / TruLlama wrap your app, Feedback functions score every Record, and the RAG triad (context relevance, groundedness, answer relevance) ships as a canonical pattern. Sweet spot: teams that want to keep eval inside their VPC, avoid a hosted vendor, and instrument any Python app (not just LangChain).
Ragas is the reference-free metric library: EvaluationDataset + evaluate() returns a pandas DataFrame of faithfulness, answer_relevancy, context_precision, context_recall. Sweet spot: CI gates — a nightly Airflow job runs Ragas over a golden query set and fails the build if any metric regresses beyond a threshold.
Snowflake Cortex Search Ops brings the eval into SQL: CORTEX SEARCH SERVICE is the semantic search index; AI_COMPLETE('claude-3-5-sonnet', ...) is the SQL judge; recall@k / MRR / nDCG are computed as plain aggregate SQL over (query, expected_doc_id, retrieved_doc_ids) tables. Sweet spot: teams whose entire data + eval + model stack lives inside one Snowflake account and one governance boundary.

What interviewers listen for.

Do you name all four axes without prompting? — senior signal.
Do you distinguish groundedness from answer relevance rather than treating "hallucination" as one blob? — senior signal.
Do you name at least two eval tools and state their sweet spots? — required answer.
Do you describe eval as a CI gate, not as "a dashboard we look at sometimes"? — senior signal.
Do you name latency + cost as an eval axis, not as "just an SRE concern"? — required answer.

Worked example — the four-axis measurement plan for a support-ticket RAG

Detailed explanation. The single most useful artifact for an LLM eval interview is a memorised 4-axis measurement plan against a concrete scenario. Every senior LLM eval discussion converges on this plan within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the plan for a support-ticket RAG chatbot that answers customer questions from a Confluence corpus.

Application. RAG chatbot; input = customer question; output = plain-language answer + citations to Confluence pages.
Corpus. ~80,000 Confluence documents, ~1.2 million chunks.
Retrieval. Hybrid (BM25 + vector) top-8 chunks.
LLM. Claude Sonnet 4.7 in production; Claude Haiku for the judge.

Question. Build the four-axis measurement plan for the support-ticket RAG and name the tool that ships each metric.

Input.

Axis	Metric	Tool	Threshold (v1)
Groundedness	fraction of answer claims supported by context	Ragas `faithfulness`	≥ 0.85
Answer relevance	question-answer topical alignment	Ragas `answer_relevancy`	≥ 0.80
Context precision	fraction of top-8 chunks that are relevant	Ragas `context_precision`	≥ 0.70
Context recall	fraction of relevant chunks in top-8	Ragas `context_recall`	≥ 0.75
Latency p95	end-to-end wall-clock	LangSmith trace	≤ 4 s
Cost per query	tokens × price	LangSmith trace	≤ $0.015

Code.

# The four-axis eval plan, expressed as a dataclass
from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalAxis:
    name: str
    metric: str
    tool: str
    threshold: float
    higher_is_better: bool

PLAN = [
    EvalAxis("Groundedness",       "faithfulness",       "Ragas",     0.85, True),
    EvalAxis("Answer relevance",   "answer_relevancy",   "Ragas",     0.80, True),
    EvalAxis("Context precision",  "context_precision",  "Ragas",     0.70, True),
    EvalAxis("Context recall",     "context_recall",     "Ragas",     0.75, True),
    EvalAxis("Latency p95",        "end_to_end_ms",      "LangSmith", 4000, False),
    EvalAxis("Cost per query",     "cost_usd",           "LangSmith", 0.015, False),
]

def gate(scores: dict[str, float]) -> tuple[bool, list[str]]:
    """Return (passed, list_of_failures). Runs in the CI job."""
    failures = []
    for axis in PLAN:
        score = scores.get(axis.metric)
        if score is None:
            failures.append(f"{axis.metric}: missing")
            continue
        ok = (score >= axis.threshold) if axis.higher_is_better else (score <= axis.threshold)
        if not ok:
            failures.append(f"{axis.metric}: {score:.3f} vs threshold {axis.threshold}")
    return (len(failures) == 0, failures)

Step-by-step explanation.

Each axis maps to one metric and one tool that ships that metric out of the box. Ragas is the workhorse for the four quality metrics because it computes all four from a single EvaluationDataset in one evaluate() call. LangSmith is the workhorse for latency + cost because every @traceable call emits a Run with wall-clock and token counts already attached.
Thresholds start conservative — 0.85 for faithfulness is aggressive but achievable with Sonnet-tier models and a well-tuned retriever. Set thresholds slightly below your first stable measurement so day-two changes have headroom; lift thresholds as the pipeline matures.
The higher_is_better flag flips the comparison for latency and cost — those are "lower is better" axes. Every gate library gets this wrong once; codifying the direction in the dataclass prevents it.
The gate() function is the CI contract. It runs in the GitHub Action after Ragas + LangSmith have populated the scores dict. Any failure fails the build; the failure list is posted as a PR comment. This is the pattern that turns eval from "a dashboard" into "a release gate."
In production, thresholds evolve. Version the PLAN list in git alongside your prompts and your dataset version; a threshold change is a code review just like a prompt change. Never bump a threshold to make a failing PR green without a written justification.

Output.

Scenario	Passed?	Failures
v1 baseline	yes	none
Model swap Sonnet → Haiku	no	`faithfulness: 0.78 vs 0.85`
Retriever change hybrid → vector-only	no	`context_precision: 0.62 vs 0.70`
Prompt tweak (add citation format)	yes	none
Cost-optimised (batch calls)	no	`end_to_end_ms: 5200 vs 4000`

Rule of thumb. Never pick an LLM eval tool without first writing down the four-axis plan and the numeric threshold on each axis. The tool is downstream of the plan; if the plan is missing, the tool choice is arbitrary and the eval will drift into "vibes."

Worked example — the senior LLM-eval interview grading rubric

Detailed explanation. The senior LLM-eval interview grades a small set of specific signals. Memorise the rubric; every answer should touch every row.

Axes named in minute 1. All four (groundedness, answer relevance, context precision/recall, latency + cost) — senior signal.
Tools per axis. Ragas for the four quality metrics, LangSmith for latency/cost, TruLens for VPC deployments, Cortex Search Ops for Snowflake-native — required.
Gate mechanism. "GitHub Action fails the PR on regression" — required.
Judge validation. Quarterly re-validation against human labels; correlation ≥ 0.75 — senior signal.

Question. Score a candidate's answer against the rubric and pick the one differentiator that separates senior from mid-level responses.

Input.

Row	Weak answer	Senior answer
Axes	"accuracy"	"four axes: groundedness, answer relevance, context precision/recall, latency + cost"
Tools	"we look at LangSmith"	"Ragas in CI, LangSmith for traces, TruLens for VPC"
Gate	"we watch a dashboard"	"GitHub Action fails PR on any axis regression"
Judge	"LLM judge is fine"	"re-validated quarterly against 100 human-labelled rows; correlation tracked"

Code.

# rubric.py — score a candidate answer against the LLM-eval rubric
RUBRIC = [
    ("axes",   ["groundedness", "answer relevance", "context precision", "latency", "cost"]),
    ("tools",  ["ragas", "langsmith", "trulens"]),
    ("gate",   ["github action", "fail", "regression"]),
    ("judge",  ["human", "correlation", "re-validate"]),
]

def score(answer: str) -> dict[str, float]:
    a = answer.lower()
    return {row: sum(1 for k in keys if k in a) / len(keys) for row, keys in RUBRIC}

candidate = ("I'd measure groundedness, answer relevance, context precision, and latency. "
             "Ragas + LangSmith. GitHub Action fails the PR on regression. "
             "The LLM judge is re-validated quarterly against human-labelled rows.")

print(score(candidate))
# → {'axes': 0.8, 'tools': 0.67, 'gate': 1.0, 'judge': 0.67}

Step-by-step explanation.

Each rubric row is a set of keyword indicators. Naming all axes hits row 1 fully; naming two tools scores partial credit on row 2.
The gate row is binary — either the candidate names an automated PR-blocking mechanism or they don't. This is the single biggest senior differentiator.
Judge validation is the row most candidates miss. Naming "human correlation" or "quarterly re-validation" separates the mid-level from senior.

Output.

Row	Weak candidate	Senior candidate
axes	0.2	0.8-1.0
tools	0.0-0.33	0.67-1.0
gate	0.0	1.0
judge	0.0	0.67-1.0

Rule of thumb. If you touch three of four rows at 0.6+, you pass the senior bar. The gate row is the mandatory one; missing it means "mid-level" regardless of the other three.

Senior interview question on LLM eval axes and CI

A senior interviewer often opens with: "You inherit a RAG chatbot that has no eval — just a LangChain chain wired to Postgres pgvector and Claude Sonnet. The product team keeps shipping prompt tweaks and quality is regressing. Walk me through the eval stack you'd stand up in the first sprint, the four axes you'd measure, and the CI gate you'd wire into GitHub Actions."

Solution Using Ragas as the CI gate with LangSmith traces for latency and cost

# eval_gate.py — the CI job that scores + gates every PR
import json, sys
from pathlib import Path
from datasets import Dataset
from langsmith import Client
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                            context_precision, context_recall)

GOLDEN = Path("eval/golden_v3.jsonl")
THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75}

def main() -> int:
    from myapp.rag import rag_chain
    predictions = []
    for row in [json.loads(l) for l in GOLDEN.read_text().splitlines()]:
        resp = rag_chain.invoke({"query": row["question"]})
        predictions.append({"question": row["question"],
                             "answer": resp["answer"],
                             "contexts": [d.page_content for d in resp["source_documents"]],
                             "ground_truth": row["ref_answer"]})

    result = evaluate(Dataset.from_list(predictions),
                      metrics=[faithfulness, answer_relevancy,
                                context_precision, context_recall])
    scores = result.to_pandas().mean(numeric_only=True).to_dict()

    # Attach latency + cost from LangSmith traces
    ls = Client()
    runs = list(ls.list_runs(project_name="rag-ci", limit=len(predictions)))
    latencies = sorted(r.latency for r in runs if r.latency is not None)
    scores["latency_p95_ms"] = latencies[int(0.95 * len(latencies))] * 1000
    scores["cost_usd_avg"]   = sum(r.total_cost or 0 for r in runs) / len(runs)

    failures = [f"{k}={scores[k]:.3f} < {v}" for k, v in THRESHOLDS.items()
                if scores.get(k, 0) < v]
    Path("eval/report.json").write_text(json.dumps(scores, indent=2))
    if failures:
        print("EVAL GATE FAILED:", failures); return 1
    print("EVAL GATE PASSED", scores); return 0

if __name__ == "__main__":
    sys.exit(main())

# .github/workflows/eval-gate.yml — the PR gate
name: LLM Eval Gate
on:
  pull_request:
    paths: ["prompts/**", "src/rag/**", "eval/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - name: Run gate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          LANGSMITH_PROJECT: "rag-ci"
        run: python eval_gate.py
      - name: Comment scores on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require("fs");
            const report = fs.readFileSync("eval/report.json", "utf8");
            github.rest.issues.createComment({
              owner: context.repo.owner, repo: context.repo.repo,
              issue_number: context.issue.number,
              body: "## LLM eval scores\n```
{% endraw %}
json\n" + report + "\n
{% raw %}
```" });

Step-by-step trace.

Step	Before (no eval)	After (Ragas + LangSmith CI gate)
Regression detection	user complaint (weeks later)	PR fails within minutes
Metrics tracked	none	4 quality + 2 pipeline axes
Judge	none	Claude Haiku as Ragas judge
Dataset	ad-hoc curl commands	versioned `golden_v3.jsonl`
PR feedback	none	GitHub bot comments scores
Threshold governance	verbal	code-reviewed dict
Cost visibility	monthly bill shock	per-PR cost delta

After deployment, every PR that touches prompts/** or src/rag/** runs the full eval gate, the four Ragas metrics + latency p95 + cost/query land as a JSON on the PR, and any axis regression below its threshold fails the check. The product team's prompt tweaks either improve the numbers or fail the merge; there is no "ship first, measure later" path anymore.

Output:

Metric	Baseline (main)	PR #482 (Haiku swap)	PR #483 (retriever tweak)
faithfulness	0.89	0.78 (fail)	0.90
answer_relevancy	0.86	0.81	0.87
context_precision	0.74	0.72	0.79 (improved)
context_recall	0.78	0.77	0.82
latency_p95_ms	3800	2100	4300 (fail)
cost_usd_avg	0.011	0.003	0.013

Why this works — concept by concept:

Ragas reference-free metrics — faithfulness, answer_relevancy, context_precision, and context_recall all compute from (question, answer, contexts, ground_truth?) alone. faithfulness and answer_relevancy don't even need ground_truth; context_precision and context_recall do. This is why Ragas is the CI workhorse — no reference-answer curation required for two of the four axes.
LangSmith @traceable runs — every LLM call inside the RAG chain emits a Run with wall-clock latency, prompt tokens, completion tokens, and computed cost. The ls.list_runs() call at gate time pulls the last N runs for the rag-ci project. This is how latency and cost enter the gate without a separate instrumentation layer.
Versioned golden dataset — golden_v3.jsonl in git is the single source of truth for what "the pipeline should do." Growing the dataset is monotonic (append new curated rows; never delete); a dataset bump is a PR just like a prompt bump. Two experiments over the same dataset version are the only fair comparison.
Threshold dict in code — THRESHOLDS lives in eval_gate.py, code-reviewed with every change. This is the difference between "we lowered the threshold to make the PR pass" (a compliance failure) and "we lowered the threshold with a written justification approved by the on-call reviewer" (a governed change).
Cost — one CI run costs roughly 120 questions × ($0.011 per RAG call + $0.002 per Ragas judge call) ≈ $1.60 per PR. Compared to the cost of a bad model swap reaching production undetected for a week (support tickets, refunds, reputation), this is the cheapest quality gate you will ever ship. O(dataset_size) per PR, O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on eval pipelines and gates

Practice →

Data validation Topic — data-validation Data validation problems on quality gates

Practice →

2. LangSmith — trace-first eval, dataset versioning, hosted judges

Every LLM call is a `Run` inside a `Trace` — datasets are versioned, experiments diff over them, judges score them

The mental model in one line: langsmith is the trace-first eval platform where every LLM call your code makes is wrapped in @traceable to emit a hierarchical Run (child of a parent Trace), your golden inputs live in a versioned Dataset, an Experiment runs your target function against every example in the dataset and pipes the outputs through a set of evaluators (LLM-as-judge or Python), and the LangSmith UI diffs experiments side by side so a prompt change or a model swap has an immediately visible before/after — it is the tool of choice when your app already lives inside LangChain or LangGraph and you want a hosted dashboard plus a dataset store without building either from scratch. Every senior DE / MLOps engineer working with LangChain has stood up LangSmith at some point; the pattern is the same across products.

The four axes for LangSmith.

Groundedness. Ships as a hosted LLM-as-judge evaluator (langsmith.evaluation.LangChainStringEvaluator("labeled_criteria", criteria="correctness") or a custom judge prompt). You can also plug Ragas into a LangSmith experiment via the RunEvaluator interface; LangSmith records the Ragas scores as annotations on each Run.
Answer relevance. Same story — a hosted criteria evaluator with criteria="relevance", or a custom Python judge. LangSmith's differentiator is that the score attaches to the exact trace, so you can drill from "answer relevance dropped 0.05" to the specific 6 runs that regressed.
Context precision / recall. LangSmith itself doesn't ship retrieval metrics out of the box, but the trace captures the retrieved documents, and you can wire a custom RunEvaluator that computes precision/recall against a labelled dataset. The pattern: dataset row carries expected_context_ids; evaluator compares against the trace's retrieved IDs.
Latency + cost. Free. Every @traceable decorator emits latency, prompt tokens, completion tokens, and cost per token computed against the model price sheet. The LangSmith dashboard groups by run name, model, project, and dataset. This is the killer feature that keeps teams on LangSmith even when they'd prefer a self-hosted stack.

The trace shape — nested runs.

Trace. The top-level unit of work. One user query = one trace. Traces have a name, project_name, tags, and a wall-clock start/end.
Run. A single operation inside the trace — an LLM call, a retriever call, a tool call, a chain step. Runs are nested (a chain run contains child runs for each component). Runs have inputs, outputs, run_type (llm / chain / tool / retriever), and cost/latency metadata.
Feedback. A score attached to a run by an evaluator (LLM judge, Python function, or human). Multiple feedback entries per run are allowed (e.g. one for faithfulness, one for relevance, one for latency).
Dataset. A versioned collection of Example rows (inputs, outputs optional). Datasets are immutable per version; adding rows creates a new version.
Experiment. A run of a target function against every example in a dataset version, with a set of evaluators. Experiments have a name, a dataset version pin, and produce one Feedback per (Example, evaluator) pair.

Dataset versioning — the disciplined pattern.

Immutable versions. Every dataset write (add / edit / delete row) creates a new version. Experiments pin to a version by ID; a rerun months later against the same version is byte-for-byte comparable.
Naming. rag_eval_v3_2026_07_30 — semantic version + date. Never overwrite rag_eval without a version suffix; two experiments named the same but against different underlying rows is the debugging pain from hell.
Tagging. Rows are tagged (golden, hard, edge_case, regression_from_pr_482). Tags let you slice eval scores by category without duplicating rows.
Bootstrap. First dataset comes from a spreadsheet of "questions our support team gets" curated by a product manager. Second dataset comes from real production traces flagged as "interesting." Third dataset comes from regression cases (a specific question that broke, immortalised as a row).

The hosted judge — LLM-as-judge inside LangSmith.

Prebuilt criteria. correctness, relevance, coherence, conciseness, harmfulness. Each is a prompt template that scores an output on a 0-1 scale.
Custom criteria. Pass a natural-language criterion string; LangSmith wraps it in a scoring prompt and runs it against a model you pick.
Pairwise evaluators. Given two candidate outputs (from two experiments), pick which is better. Useful for A/B comparisons of prompt versions.
Cost. Every hosted-judge call is a paid LLM call. Budget: judge cost ≈ 1-2× the production call cost per dataset row.

Common interview probes on LangSmith.

"What's a Run vs a Trace?" — required answer: Trace is the top-level unit of work; Runs are the nested operations inside it.
"How do you version an eval dataset?" — required answer: immutable versions, pin experiments to a version ID.
"What's the difference between a hosted evaluator and a custom evaluator?" — hosted = LangSmith cloud; custom = your Python callable receiving (run, example).
"How do you gate a PR on LangSmith scores?" — required answer: run an experiment in CI, fetch scores via SDK, compare against thresholds.

Worked example — instrument a LangChain RAG with `@traceable`

Detailed explanation. The first thing any LangSmith deployment does is turn on tracing. The @traceable decorator (or the auto-instrumented langchain package) captures every LLM call, retriever call, and chain step and ships them to the LangSmith cloud. Once tracing is on, every subsequent query is a debugging goldmine; without it, LangSmith is empty. Walk through the setup for a Postgres-pgvector + Claude RAG chain.

Chain. RetrievalQA from LangChain: pgvector retriever → prompt template → Claude Sonnet.
Env vars. LANGSMITH_API_KEY, LANGSMITH_PROJECT=support-rag, LANGSMITH_TRACING=true.
Traceable functions. The chain, the retriever, the LLM call — all auto-traced by the LangChain integration.

Question. Wire up @traceable around a LangChain RAG so every user query lands as a Trace in the LangSmith UI with per-step timings and token counts.

Input.

Component	Type	Traced?
`rag_chain.invoke`	Chain	yes (auto)
`pgvector.similarity_search`	Retriever	yes (auto)
`ChatAnthropic.invoke`	LLM	yes (auto)
`format_answer` (custom)	Utility	yes (via `@traceable`)
`answer_endpoint` (FastAPI)	Endpoint	yes (via `@traceable`)

Code.

# rag_app.py — LangChain RAG with LangSmith tracing
import os
os.environ.setdefault("LANGSMITH_TRACING", "true")
os.environ.setdefault("LANGSMITH_PROJECT", "support-rag")

from langchain.chains import RetrievalQA
from langchain_anthropic import ChatAnthropic
from langchain_community.vectorstores import PGVector
from langchain_openai import OpenAIEmbeddings
from langsmith import traceable
from fastapi import FastAPI
from pydantic import BaseModel

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
retriever  = PGVector(
    connection_string=os.environ["PG_URL"],
    embedding_function=embeddings,
    collection_name="support_docs",
).as_retriever(search_kwargs={"k": 8})

llm = ChatAnthropic(model="claude-sonnet-4-7", temperature=0.1)

rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={"verbose": False},
)

@traceable(run_type="tool", name="format_answer")
def format_answer(question: str, resp: dict) -> dict:
    """Trim, cite, and structure the LLM output for the client."""
    return {
        "answer":     resp["result"].strip(),
        "citations":  [d.metadata.get("source_url") for d in resp["source_documents"]],
        "confidence": min(1.0, len(resp["source_documents"]) / 8),
    }

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/answer")
@traceable(run_type="chain", name="answer_endpoint",
           tags=["prod", "v2.3"])
async def answer(q: Query):
    resp = rag_chain.invoke({"query": q.question})
    return format_answer(q.question, resp)

Step-by-step explanation.

Setting LANGSMITH_TRACING=true and LANGSMITH_PROJECT before importing langchain auto-instruments every LangChain primitive. No decorator needed on rag_chain.invoke — the LangChain integration wraps it for you. This is the fastest way to get useful traces.
The @traceable(run_type="tool", name="format_answer") on the custom formatter captures the non-LangChain utility inside the same trace tree. run_type="tool" is a hint to the UI (colours the run icon); name overrides the auto-detected function name.
The @traceable(run_type="chain", name="answer_endpoint", tags=["prod", "v2.3"]) on the FastAPI endpoint makes the endpoint the top-level trace. Tags like prod and v2.3 are filterable in the LangSmith UI — you can pull all traces for v2.3 in one query.
After deploy, every /answer POST creates one trace with three or four nested runs: the endpoint (chain), the RAG chain (chain), the retriever (retriever), the LLM call (llm), and the formatter (tool). Each run has latency, tokens, and cost.
Sampling in production: setting LANGSMITH_TRACING=true traces 100% of requests by default. For high-volume prod, use LANGSMITH_TRACING_SAMPLE_RATE=0.01 to trace 1%. Debugging incidents: flip to 1.0 for 15 minutes, capture the incident traces, flip back.

Output.

Trace field	Value
Trace name	`answer_endpoint`
Total latency	2,340 ms
Total tokens	4,120 prompt + 380 completion
Total cost	$0.0148
Nested runs	4 (endpoint → chain → retriever → llm → tool)
Tags	`prod`, `v2.3`
Project	`support-rag`

Rule of thumb. Turn on LangSmith tracing on day one — even before you have any evaluators wired up. Tracing alone gives you latency + cost per LLM call, per model, per prompt version, and per endpoint, which is more observability than most LLM apps ever get. Add evaluators once the trace tree feels rich enough.

Worked example — run a pairwise experiment over a versioned dataset

Detailed explanation. The killer LangSmith workflow is the pairwise experiment: two candidate configs (say, "prompt v1" vs "prompt v2") both run against the same dataset version, and a hosted pairwise judge picks the better output per row. The result is a win-rate ("prompt v2 wins on 68 of 100 rows"), which is directly consumable by a product review. Walk through the setup.

Dataset. support_rag_v3 — 100 curated {question, expected_answer_gist} rows.
Configs. prompt_v1 (current prod) vs prompt_v2 (proposed change).
Judge. Hosted pairwise evaluator with a "which answer is more helpful?" criterion.
Output. Win-rate table + drill-down to disagreements.

Question. Run a pairwise LangSmith experiment comparing prompt_v1 and prompt_v2 against support_rag_v3 and interpret the result.

Input.

Field	Value
Dataset	`support_rag_v3` (100 examples)
Baseline function	`run_with_prompt_v1`
Candidate function	`run_with_prompt_v2`
Judge	Claude Sonnet, criterion "helpfulness"
Metric	pairwise win-rate

Code.

# pairwise_experiment.py
from langsmith import Client
from langsmith.evaluation import evaluate_comparative
from langchain_anthropic import ChatAnthropic

ls = Client()

# --- The two target functions ---
def run_with_prompt_v1(inputs: dict) -> dict:
    from myapp.rag import rag_chain_v1
    return {"answer": rag_chain_v1.invoke({"query": inputs["question"]})["result"]}

def run_with_prompt_v2(inputs: dict) -> dict:
    from myapp.rag import rag_chain_v2
    return {"answer": rag_chain_v2.invoke({"query": inputs["question"]})["result"]}

# --- The pairwise judge ---
JUDGE_PROMPT = """You are comparing two answers to a support question.
Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}
Which answer is more helpful, accurate, and grounded?
Respond with exactly one of: "A", "B", "TIE"."""

judge_llm = ChatAnthropic(model="claude-sonnet-4-7", temperature=0)

def pairwise_helpfulness(runs: list, example: dict) -> dict:
    a, b = runs[0].outputs["answer"], runs[1].outputs["answer"]
    resp = judge_llm.invoke(JUDGE_PROMPT.format(
        question=example.inputs["question"], answer_a=a, answer_b=b,
    ))
    label = resp.content.strip().upper()
    return {
        "key": "helpfulness",
        "scores": [
            1 if label == "A" else 0,
            1 if label == "B" else 0,
        ],
    }

# --- Step 1: run each candidate as a normal experiment ---
r1 = ls.evaluate(
    run_with_prompt_v1,
    data="support_rag_v3",
    experiment_prefix="prompt_v1",
)
r2 = ls.evaluate(
    run_with_prompt_v2,
    data="support_rag_v3",
    experiment_prefix="prompt_v2",
)

# --- Step 2: run the pairwise judge over the two experiments ---
comparative = evaluate_comparative(
    experiments=[r1.experiment_name, r2.experiment_name],
    evaluators=[pairwise_helpfulness],
)
print(comparative)

Step-by-step explanation.

Two separate ls.evaluate calls run each candidate against the same dataset version. Each produces its own experiment with its own outputs stored per-example. Using the same dataset version is what makes the comparison fair — a change of dataset between runs would confuse the diff.
The pairwise judge is a custom evaluate_comparative evaluator. It receives both runs (one per experiment) plus the example, calls the judge LLM with a "which is better?" prompt, and returns per-experiment scores (1 for winner, 0 for loser, 0.5 for tie in some conventions).
Temperature 0 on the judge is mandatory — you want deterministic scoring so a re-run yields the same result. The judge model can be the same as the production model or a stronger one; the common pattern is "production uses Haiku for cost, judge uses Sonnet for quality."
evaluate_comparative returns a win-rate summary and stores per-row judge outputs in LangSmith. The UI shows a side-by-side diff for every disagreement, so you can eyeball whether the judge is agreeing with human intuition.
The gate: PR is merged if prompt_v2_wins / (prompt_v1_wins + prompt_v2_wins + ties) > 0.60. Anything less than 60% win-rate on a pairwise judge is not a meaningful improvement; ties count against both.

Output.

Metric	Value
Dataset version	`support_rag_v3` (100 rows)
`prompt_v1` wins	22
`prompt_v2` wins	68
Ties	10
`prompt_v2` win-rate (excl. ties)	75.6%
Judge cost	$0.42 total
Verdict	ship `prompt_v2`

Rule of thumb. Prefer pairwise judges over pointwise judges when the goal is "is this change better?" — pairwise is more sensitive to small quality deltas because the judge is asked to compare, not to score in an absolute frame. Save pointwise judges for absolute thresholds ("faithfulness ≥ 0.85").

Senior interview question on LangSmith trace + dataset ops

A senior interviewer might ask: "You've got a LangChain-based RAG in production, no eval, no dataset, complaints piling up. Design the first sprint: turn on tracing, extract a first dataset from production traces, set up a baseline experiment, and wire a pairwise comparison to gate the next prompt change."

Solution Using LangSmith tracing, production-trace-derived dataset, and a pairwise gate

# sprint1_langsmith_setup.py
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "support-rag-prod"

from datetime import datetime, timedelta, timezone
from langsmith import Client
from langsmith.evaluation import evaluate_comparative
from langchain_anthropic import ChatAnthropic

ls = Client()
DATASET = "support_rag_from_prod_v1"

# Step A: extract dataset from last week's low-confidence prod traces
def bootstrap_dataset():
    if any(d.name == DATASET for d in ls.list_datasets()):
        return
    since = datetime.now(timezone.utc) - timedelta(days=7)
    runs = list(ls.list_runs(project_name="support-rag-prod", start_time=since,
                              run_type="chain", limit=300))
    picks = sorted(runs, key=lambda r: (r.outputs or {}).get("confidence", 1.0))[:100]
    ls.create_dataset(dataset_name=DATASET, description="prod-derived seed")
    ls.create_examples(
        dataset_name=DATASET,
        inputs=[{"question": r.inputs["q"]["question"]} for r in picks],
        outputs=[None] * len(picks),
        metadata=[{"source_trace_id": str(r.id)} for r in picks],
    )

# Step B: pinned baseline + pairwise gate for the next prompt PR
def run_with_prompt(v):
    from myapp.rag import make_chain
    chain = make_chain(prompt_version=v)
    return lambda inputs: {"answer": chain.invoke({"query": inputs["question"]})["result"]}

JUDGE = ChatAnthropic(model="claude-sonnet-4-7", temperature=0)
JUDGE_PROMPT = ("Question: {q}\nAnswer A: {a}\nAnswer B: {b}\n"
                'Which is more helpful, accurate, grounded? Reply "A", "B", or "TIE".')

def pairwise_gate(runs, example):
    resp = JUDGE.invoke(JUDGE_PROMPT.format(
        q=example.inputs["question"],
        a=runs[0].outputs["answer"], b=runs[1].outputs["answer"]))
    label = resp.content.strip().upper()
    return {"key": "helpfulness",
            "scores": [1 if label == "A" else 0, 1 if label == "B" else 0]}

def gate_prompt_change(new_version: str, min_win_rate=0.60) -> bool:
    r_old = ls.evaluate(run_with_prompt("v1"), data=DATASET,
                        experiment_prefix="baseline_prompt_v1")
    r_new = ls.evaluate(run_with_prompt(new_version), data=DATASET,
                        experiment_prefix=f"candidate_{new_version}")
    comp = evaluate_comparative(experiments=[r_old.experiment_name, r_new.experiment_name],
                                 evaluators=[pairwise_gate])
    totals = comp["scores"]["helpfulness"]["totals"]
    win_rate = totals["B"] / max(1, totals["A"] + totals["B"])
    return win_rate >= min_win_rate

if __name__ == "__main__":
    bootstrap_dataset()
    print("APPROVED" if gate_prompt_change("v2") else "BLOCKED")

Step-by-step trace.

Step	Before (no eval)	After (LangSmith sprint 1)
Tracing	none	100% of prod traffic (later sampled)
Dataset	none	`support_rag_from_prod_v1` — 100 real prod rows
Baseline	vibes	pinned experiment `baseline_prompt_v1`
Prompt change gate	verbal	pairwise judge, 60% win-rate threshold
Regression memory	Slack thread	permanent rows tagged `regression`
Judge	none	Claude Sonnet, deterministic (temp=0)
Dashboard	none	LangSmith UI with side-by-side diffs

After the sprint ends, the team has a running trace history, a 100-row prod-derived dataset, a pinned baseline experiment, and a gate_prompt_change() function anyone can call before proposing a prompt PR. The vibes-based prompt shipping cycle is over; a prompt change that doesn't win 60% of pairwise comparisons doesn't merge.

Output:

Sprint deliverable	Concrete artifact
Tracing on	`LANGSMITH_TRACING=true` in every service env
First dataset	`support_rag_from_prod_v1` (100 rows)
Baseline experiment	`baseline_prompt_v1` (immutable)
Pairwise gate function	`gate_prompt_change(new_version)`
CI hook	`.github/workflows/prompt-gate.yml` calls the function
Regression tag	`metadata.tag = "regression"` on 3 rows
Cost per gate run	~$0.85 (200 chain calls + 100 judge calls)

Why this works — concept by concept:

@traceable + LangChain auto-instrumentation — one env var turns every LLM call in the app into a Run in a Trace, capturing latency, tokens, cost, inputs, outputs, and error stacks for free. This is the highest-leverage single change any LLM-app team can make.
Production-derived dataset bootstrap — the 100 lowest-confidence prod traces from last week are the perfect seed dataset because they represent actual user friction, not synthetic questions. Ground-truth answers are filled in later by humans; the raw questions are already load-bearing.
Pinned experiment as baseline — pinning baseline_prompt_v1 against dataset support_rag_from_prod_v1 makes every future comparison anchor-referenced. Anyone can rerun the baseline months later against the same dataset version and get the same score.
Pairwise judge with temperature 0 — pairwise judges are more sensitive than pointwise judges to small quality deltas because the judge is asked to compare, not to score in an absolute frame. Temperature 0 makes the score deterministic so a re-run yields the same verdict.
Cost — one gate run = 200 chain calls (100 rows × 2 candidates) + 100 judge calls ≈ $0.85. Compared to shipping a bad prompt to production for a week (support tickets, refunds, reputation), this is the cheapest quality gate you will ever ship. O(dataset_size) per gate; O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on trace-instrumented pipelines

Practice →

Python Topic — python Python problems on decorator patterns and observability

Practice →

3. TruLens — feedback functions and the RAG triad

`Feedback` functions score every `Record` on the three edges of the RAG triad — local-first, open, and framework-agnostic

The mental model in one line: trulens is the open-source, local-first eval library where you wrap your app in TruChain / TruLlama / TruCustomApp, attach Feedback functions that score every Record (one Record = one user query + its outputs), and lean on the canonical RAG triad — context relevance (query ↔ context), groundedness (context ↔ answer), and answer relevance (query ↔ answer) — as the three-edged completeness proof that no single-axis eval can match, and the entire eval history lives in a local SQLite (or your own Postgres) with no vendor round-trip required. TruLens's sweet spot is teams that need eval inside the VPC, want to instrument non-LangChain code, or prefer an unopinionated Feedback function primitive over a hosted platform's canned evaluators.

The four axes for TruLens.

Groundedness. The classic Feedback(provider.groundedness_measure_with_cot_reasons) — split the answer into claims, ask an LLM judge whether each claim is supported by the context, aggregate. TruLens ships this as a named feedback function; you plug in the provider (OpenAI, Anthropic, Bedrock, local via LiteLLM) and it just works.
Answer relevance. Feedback(provider.relevance) on (query, answer). Same LLM-judge pattern as Ragas's answer_relevancy but the prompt is TruLens's own and the API surface is Feedback object composition.
Context precision / recall. Slightly indirect — TruLens exposes "context relevance" per-chunk (Feedback(provider.context_relevance_with_cot_reasons).on_input().on(context)) which is essentially per-chunk precision. Aggregate across chunks and you get context precision; recall requires labelled ground-truth context IDs.
Latency + cost. TruLens records per-Record wall-clock and token counts natively. record.latency and record.cost come out of the box; you can pipe them into your dashboard of choice or query the local SQLite directly.

The Feedback function primitive.

Anatomy. Feedback(provider.metric_fn, name="...").on_input().on_output() — a Feedback binds a callable (usually an LLM-judge call), a name, and selectors that pull the right fields off the Record.
Selectors. .on_input() pulls the user query; .on_output() pulls the app's final answer; .on(Select.RecordCalls.retriever.rets) pulls the retriever's output. Selectors are the composable glue between the app trace and the feedback function.
Aggregation. By default, feedback returns a scalar. For per-chunk metrics like context_relevance, aggregation strategies include np.mean, np.min, np.max, or a custom callable.
Providers. LiteLLM, OpenAI, Anthropic, Bedrock, Huggingface — any provider with a chat completion API can be the judge. LiteLLM is the local-first choice because it wraps every provider under one API.

App vs Session vs Record scope.

Record. One user query → one Record. Feedback runs per Record. Most metrics live here.
Session. A sequence of Records tied to a user session (multi-turn chat). Session-scoped feedback measures things like coherence across turns.
App. The app-level aggregate (all Records for one app version). App-scoped feedback is where you compute p95 latency, aggregate faithfulness, etc.
App version. TruChain(chain, app_name="rag", app_version="v2.3") — every deploy bumps the version; the dashboard slices by version so a bad deploy shows up as a scored-lower app version.

Local-first — the offline story.

Storage. Default backend is SQLite at default.sqlite. Every Record + every Feedback score lives there. Tru().get_records_and_feedback() returns a pandas DataFrame you can query, join, or export.
Dashboard. Tru().run_dashboard() starts a Streamlit UI on localhost with per-Record drill-down, feedback distributions, and app-version diffs. No cloud round-trip.
VPC deploy. For team use, swap SQLite for Postgres (Tru(database_url="postgresql://...")) — the schema is one-command migratable and the dashboard reads Postgres directly. Everything stays in your VPC.

Common interview probes on TruLens.

"What's the RAG triad?" — required answer: context relevance, groundedness, answer relevance — the three edges of a triangle whose vertices are query, context, and answer.
"Where does TruLens store its eval history?" — required answer: local SQLite by default, Postgres in production; no vendor round-trip.
"How is Feedback different from LangSmith's evaluator?" — TruLens Feedback is a Python callable + selector composition, framework-agnostic; LangSmith evaluators are cloud-hosted primitives tied to Runs.
"Why does groundedness split into claims?" — required answer: to score each atomic claim independently, otherwise a single unsupported sentence in a paragraph is invisible.

Worked example — instrument a RAG chain with `TruChain` and run the triad

Detailed explanation. The canonical TruLens setup: wrap an existing LangChain RAG in TruChain, define three Feedback functions (the triad), and let every subsequent chain.invoke produce a scored Record. Walk through the setup end-to-end.

Chain. LangChain RetrievalQA with FAISS + Claude Sonnet.
Provider. LiteLLM pointed at Claude Haiku (cheap judge).
Feedbacks. Context relevance (per-chunk, mean-aggregated), groundedness, answer relevance.
Storage. Default SQLite.

Question. Wire up TruChain around a LangChain RAG and produce the RAG triad scores for every query.

Input.

Component	Object
App	LangChain `RetrievalQA`
Wrapper	`TruChain(app_name="support-rag", app_version="v2.3")`
Provider	`LiteLLM(model_engine="claude-3-5-haiku-20241022")`
Feedbacks	`f_context_relevance`, `f_groundedness`, `f_answer_relevance`
Backend	SQLite (`default.sqlite`)

Code.

# trulens_rag.py — instrument a LangChain RAG with the RAG triad
import numpy as np
from trulens.core import TruSession, Feedback, Select
from trulens.apps.langchain import TruChain
from trulens.providers.litellm import LiteLLM
from langchain.chains import RetrievalQA
from langchain_anthropic import ChatAnthropic
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# 1. Build the base RAG chain (unchanged from prod)
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vs         = FAISS.load_local("faiss_index", embeddings, allow_dangerous_deserialization=True)
retriever  = vs.as_retriever(search_kwargs={"k": 8})
llm        = ChatAnthropic(model="claude-sonnet-4-7", temperature=0.1)
rag_chain  = RetrievalQA.from_chain_type(llm=llm, retriever=retriever,
                                          return_source_documents=True)

# 2. Start TruLens session + provider
session  = TruSession()
provider = LiteLLM(model_engine="claude-3-5-haiku-20241022")

# 3. Define the RAG triad
context_selector = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content

f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons,
             name="Context Relevance")
    .on_input()
    .on(context_selector)
    .aggregate(np.mean)
)

f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons,
             name="Groundedness")
    .on(context_selector.collect())
    .on_output()
)

f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons,
             name="Answer Relevance")
    .on_input()
    .on_output()
)

# 4. Wrap the chain
tru_chain = TruChain(
    rag_chain,
    app_name="support-rag",
    app_version="v2.3",
    feedbacks=[f_context_relevance, f_groundedness, f_answer_relevance],
)

# 5. Every call is now scored
with tru_chain as recording:
    resp = rag_chain.invoke({"query": "How do I rotate my API key without downtime?"})

record = recording.records[-1]
print(f"latency: {record.latency:.2f}s, cost: ${record.cost:.4f}")
print("scores:", {k: v.result for k, v in record.feedback_and_future_results.items()})

# 6. Fetch the full history as a DataFrame
df = session.get_records_and_feedback(app_ids=[tru_chain.app_id])[0]
print(df[["input", "output", "Context Relevance", "Groundedness", "Answer Relevance"]].tail())

Step-by-step explanation.

The base LangChain RAG is unchanged from production — TruLens is a wrapper, not a replacement. This is TruLens's key adoption story: you don't rewrite your app to add eval.
The TruSession() singleton is the entry point; by default it uses SQLite (default.sqlite) in the current directory. Swap for Postgres by passing database_url= at construction. Every subsequent TruChain, TruLlama, or TruCustomApp registers with the session.
The context_selector is a TruLens selector expression — Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content navigates the recorded trace: pull the retriever call, get all returned documents, extract page_content from each. Selectors are the composable primitive that makes TruLens framework-agnostic.
The three Feedback functions are the RAG triad. Note how each uses different selectors: f_context_relevance scores per-chunk (mean-aggregated), f_groundedness collects all chunks as one context blob then scores the answer against it, f_answer_relevance compares query directly to answer.
The with tru_chain as recording context manager captures every rag_chain.invoke inside as a Record. recording.records is the list of Records made in the block; each has .latency, .cost, and .feedback_and_future_results. The dataframe view at the end is what you'd surface in a dashboard.

Output.

Query	Context Relevance	Groundedness	Answer Relevance	Latency	Cost
"How do I rotate my API key?"	0.86	0.92	0.94	2.1 s	$0.014
"What's the SLA at 3am ET?"	0.71	0.68	0.90	2.3 s	$0.015
"Why can't I export PDF?"	0.55	0.44	0.82	2.4 s	$0.016
Overall (mean)	0.71	0.68	0.89	2.27 s	$0.015

Rule of thumb. Wrap first, evaluate second. Get TruChain around the existing app before you fine-tune Feedback functions. The first useful signal comes from just seeing per-Record latencies + costs in the SQLite; the triad scores are the second layer.

Worked example — per-chunk context relevance and the drill-down that saves your retriever

Detailed explanation. The RAG triad scores at the Record level are useful, but the real diagnostic power of TruLens comes from per-chunk context relevance. When a query scores low on context relevance, the drill-down reveals which chunks are irrelevant, which points directly at retriever tuning: switch to hybrid retrieval, increase k, add a reranker, or fix the chunking strategy. Walk through the diagnosis flow for a low-scoring query.

Low-scoring query. "Why can't I export PDF from a shared workspace?" — context relevance 0.55.
Per-chunk drill-down. Select.RecordCalls.retriever.get_relevant_documents.rets[:] — inspect each chunk's individual score.
Diagnosis. 6 of 8 chunks are billing-related; 2 are the actual "role-based export" chunks. Retriever is over-weighting a common bigram.
Fix. Switch from pure-vector to hybrid (BM25 + vector) retrieval; add a reranker.

Question. Extract per-chunk context relevance for the failing query and use it to diagnose the retriever problem.

Input.

Chunk #	Source	Vector score	Context relevance score
1	billing-policy.md	0.88	0.12
2	billing-policy.md	0.86	0.10
3	shared-workspaces.md#export	0.84	0.94
4	payment-methods.md	0.83	0.08
5	export-formats.md#roles	0.82	0.91
6	billing-policy.md	0.81	0.11
7	account-limits.md	0.80	0.15
8	billing-history.md	0.79	0.09

Code.

# per_chunk_context_relevance.py
import numpy as np
from trulens.core import Feedback, Select
from trulens.providers.litellm import LiteLLM

provider = LiteLLM(model_engine="claude-3-5-haiku-20241022")

# Per-chunk context relevance — no aggregate; keep the vector of scores
f_context_relevance_per_chunk = (
    Feedback(provider.context_relevance_with_cot_reasons,
             name="Context Relevance (per chunk)")
    .on_input()
    .on(Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content)
    # NOTE: no .aggregate() — returns per-chunk scores as a list
)

# After running the query, drill down
record = recording.records[-1]
per_chunk = record.feedback_and_future_results["Context Relevance (per chunk)"].result

# Zip with retrieved sources for diagnosis
docs = record.calls[-1].rets  # the retriever's returned docs
diag = [
    {"source": d.metadata["source_url"], "vector": d.metadata["score"], "rel": s}
    for d, s in zip(docs, per_chunk)
]

# Print the diagnostic table
for d in sorted(diag, key=lambda x: -x["rel"]):
    print(f"{d['rel']:.2f}  {d['vector']:.2f}  {d['source']}")

# Diagnosis rule: if > 50% of top-k chunks score < 0.30 on context relevance,
# the retriever is over-weighting a common bigram
low = sum(1 for d in diag if d["rel"] < 0.30)
if low / len(diag) > 0.5:
    print(f"RETRIEVER PROBLEM: {low}/{len(diag)} chunks below 0.30 relevance")
    print("Recommend: switch to hybrid retrieval + reranker")

Step-by-step explanation.

Removing the .aggregate() call on the Feedback function is what turns a scalar score into a per-chunk vector. TruLens automatically iterates over the selector's returned list and applies the callable to each element.
The per-chunk table reveals the pathology: chunks 1, 2, 6 are all billing-policy.md, all score above 0.79 on the vector distance, but all score below 0.15 on context relevance. The vector retriever thinks these chunks match; the LLM judge thinks they don't. This is a classic "vector similarity captures surface bigrams" failure mode.
Chunks 3 and 5 (from shared-workspaces.md and export-formats.md) are the actually relevant chunks — high context relevance (0.94, 0.91) and moderate vector score (0.84, 0.82). They're in the top-8 but drowned out by the noise.
The rule of thumb — "if > 50% of top-k chunks score < 0.30 on context relevance, the retriever needs help" — is the operational signal that triggers a retriever change. Options: (a) hybrid BM25 + vector rerank; (b) add a cross-encoder reranker; (c) increase top-k and re-rank; (d) fix the chunking strategy (chunks 1/2/6 might be over-chunked from one long policy page).
After switching to hybrid retrieval + a bge-reranker-large reranker, this query's context relevance jumps from 0.55 to 0.87; the two relevant chunks now sit at ranks 1 and 2. The Record-level triad scores rise across the board.

Output.

Fix applied	Context Relevance (mean)	Faithfulness	Answer Relevance
baseline (pure vector, k=8)	0.55	0.44	0.82
hybrid BM25 + vector	0.72	0.68	0.86
hybrid + cross-encoder rerank	0.87	0.89	0.93

Rule of thumb. Any Record with context relevance below 0.60 should trigger a per-chunk drill-down. The per-chunk vector is the retriever's report card; scalar aggregates hide the pathology. Wire the drill-down as a Slack alert on the first N regressions.

Senior interview question on TruLens triad and production scoring

A senior interviewer might ask: "You've got a support RAG in production, no eval, running inside a bank's VPC — no data can leave the perimeter. Wire up TruLens with the RAG triad, run it locally against a golden set, then deploy a sampled production-time scoring plus a nightly batch back-fill without adding latency to 98% of traffic."

Solution Using TruLens RAG triad locally, sampled sync feedback + nightly batch back-fill

# trulens_vpc_prod.py — VPC-friendly TruLens deployment
import os, random, numpy as np
from trulens.core import TruSession, Feedback, Select
from trulens.apps.langchain import TruChain
from trulens.providers.litellm import LiteLLM

# Postgres inside VPC; Bedrock judge inside AWS — zero external egress
session = TruSession(database_url=os.environ["TRULENS_DATABASE_URL"])
provider = LiteLLM(model_engine="bedrock/anthropic.claude-3-5-haiku-20241022-v1:0")

from myapp.rag import make_chain
rag_chain = make_chain()

CTX = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content

f_ctx = (Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
         .on_input().on(CTX).aggregate(np.mean))
f_grd = (Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
         .on(CTX.collect()).on_output())
f_ans = (Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
         .on_input().on_output())

tru_chain = TruChain(rag_chain, app_name="support-rag",
                     app_version=os.environ.get("APP_VERSION", "v2.3"),
                     feedbacks=[f_ctx, f_grd, f_ans])

# 1. Local golden-set run (bootstrap eval)
def run_golden_set():
    import json; from pathlib import Path
    with tru_chain as _:
        for row in [json.loads(l) for l in Path("eval/golden.jsonl").read_text().splitlines()]:
            rag_chain.invoke({"query": row["question"]})
    df, _ = session.get_records_and_feedback(app_ids=[tru_chain.app_id])
    print(df[["Context Relevance", "Groundedness", "Answer Relevance"]].mean())

# 2. Production endpoint: 2% sync scoring, 98% record-only
SAMPLE_RATE = 0.02

def prod_answer(question: str) -> dict:
    ctx = tru_chain if random.random() < SAMPLE_RATE else tru_chain.record_only()
    with ctx as rec:
        resp = rag_chain.invoke({"query": question})
    mode = "sync" if ctx is tru_chain else "deferred"
    return {"answer": resp["result"], "feedback_mode": mode,
            "record_id": rec.records[-1].record_id}

# 3. Nightly back-fill for unscored Records
def nightly_backfill():
    df, _ = session.get_records_and_feedback(app_ids=[tru_chain.app_id])
    for _, row in df[df["Groundedness"].isna()].iterrows():
        record = session.get_record(row["record_id"])
        for f in [f_ctx, f_grd, f_ans]:
            session.add_feedback(feedback_result=f.run(record=record))

Step-by-step trace.

Layer	Component	Purpose
Storage	Postgres in VPC	zero-egress persistence
Provider	LiteLLM → Bedrock Claude Haiku	judge stays inside AWS
App wrapper	`TruChain`	captures Records for the LangChain RAG
Feedbacks	triad (context, groundedness, answer)	three-edge completeness
Sync path	2% of prod requests	real-time alerts
Async path	98% of prod requests	zero added latency
Nightly back-fill	batch judge calls	full coverage within 24h

After deployment, the RAG's every prod query is stored as a Record in Postgres, 2% of Records get scored instantly with the triad, and a nightly Airflow DAG back-fills the remaining 98% using batch LLM calls. The bank's compliance team sees no data leaving the VPC; the product team sees a real-time dashboard of triad scores per app version; a bad deploy shows up as a scored-lower app version within an hour.

Output:

Metric	Value
Records stored / day	600,000
Sync-scored Records / day	12,000
Batch-scored Records / day	588,000
Feedback cost / day (batch discount)	~$4,580
Data egress from VPC	0 bytes
Time from bad deploy to alert	≤ 15 min (on sync tail)
Storage in Postgres / day	~600 MB

Why this works — concept by concept:

RAG triad completeness — the three edges (query↔context, context↔answer, query↔answer) form a triangle; a failure on any edge is a distinct failure mode. Context-relevance failure = bad retriever; groundedness failure = hallucination; answer-relevance failure = off-topic answer. Measuring only one edge lets the other two regress invisibly.
Postgres backend inside VPC — TruLens's SQLAlchemy backend is swappable; pointing it at an in-VPC Postgres keeps every Record and every score inside the bank's perimeter. No data crosses a vendor boundary at any point.
LiteLLM + Bedrock judge — the judge LLM also stays inside AWS (Bedrock is region-local). LiteLLM's unified provider API means switching between Bedrock, self-hosted vLLM, or Azure OpenAI is a one-line change with the rest of the eval code unchanged.
2% sync + 98% async — the sampled-sync tier gives real-time alerts (a bad deploy shows up in 12,000 scored Records within 15 minutes), while the async tier back-fills the aggregate view without adding latency to 98% of user requests. Coverage is 100% within 24 hours.
Cost — 2% × $0.015 sync + 98% × $0.007 (batch discount) per Record ≈ $4,580/day for 600k queries, versus $9,000/day if all 100% went sync. Compared to shipping a bad model without any triad scoring, the alert-lead-time saving alone is worth 10× the eval cost. O(1) latency for 98% of prod requests; O(1) latency to detect regressions.

Python
Topic — python
Python problems on wrapper and observability patterns

Practice →

Design Topic — design Design problems on VPC-first eval systems

Practice →

4. Ragas — reference-free RAG metrics for CI

Reference-free metrics (`faithfulness`, `answer_relevancy`) plus reference-based (`context_precision`, `context_recall`) — one `evaluate()` call, one scored DataFrame

The mental model in one line: ragas is the metric library that turns rag evaluation into a one-line evaluate(dataset, metrics=[...]) call returning a pandas DataFrame of scored rows — with two metrics (faithfulness, answer_relevancy) that need no reference answer and two metrics (context_precision, context_recall) that use a labelled ground-truth, all judged by any LLM you configure via LiteLLM — and its sweet spot is the batch CI gate where a nightly Airflow job scores a fixed golden set, uploads the DataFrame, and a GitHub Action fails the PR when any metric regresses beyond a threshold. Ragas is the least opinionated of the four tools: it doesn't wrap your app, it doesn't own a dashboard, it just scores.

The four axes for Ragas.

Groundedness. faithfulness — split the answer into atomic claims, ask the LLM judge whether each claim is entailed by the retrieved contexts, aggregate. Same intent as TruLens's groundedness; different prompt.
Answer relevance. answer_relevancy — generate 3 hypothetical questions the answer could address, compute cosine similarity between each and the actual query, aggregate. Requires an embedding model in addition to the judge LLM.
Context precision / recall. context_precision — for each retrieved chunk, is it relevant to the question given the ground-truth answer? Ranked precision. context_recall — decompose the ground-truth answer into statements; check which are covered by the retrieved contexts. Both require a labelled ground truth.
Latency + cost. Not measured by Ragas itself. Wire LangSmith or TruLens for those axes, or compute them in your own wrapper. Ragas is metric-only.

Reference-free vs reference-based — the split that matters.

Reference-free: faithfulness, answer_relevancy. Need only (question, answer, contexts). Cheap to curate the dataset — you don't need a golden answer written by a human. Perfect for bootstrapping eval when you have zero ground-truth data.
Reference-based: context_precision, context_recall, answer_correctness, answer_similarity. Need (question, answer, contexts, ground_truth). Ground truth costs human time to curate; budget 5-10 minutes per row.
Bootstrap pattern. Start with reference-free (day one). Add reference-based as you accumulate labelled rows (weeks in). Ragas's DataFrame skips reference-based metrics gracefully if ground_truth is missing.

The EvaluationDataset and evaluate() shape.

Row shape. {question, answer, contexts: list[str], ground_truth?} per row. Contexts is a list of chunk strings; ground_truth is optional.
Dataset. Dataset.from_list(rows) (HuggingFace datasets.Dataset); or Ragas's own EvaluationDataset in v0.2+. Both interoperate.
evaluate. evaluate(dataset, metrics=[faithfulness, answer_relevancy, ...], llm=..., embeddings=...). Returns a Result with .to_pandas() giving per-row scores.
LLM + embeddings. Any LiteLLM-compatible LLM as judge; any HF or provider embeddings for answer_relevancy. Common: Claude Haiku judge + OpenAI text-embedding-3-large.

Common interview probes on Ragas.

"What's reference-free vs reference-based?" — required answer: reference-free needs no ground-truth answer; reference-based does.
"Which Ragas metrics need ground truth?" — context_precision, context_recall, answer_correctness.
"How is faithfulness computed?" — split answer into claims; judge each against contexts; aggregate.
"How would you gate a PR on Ragas?" — nightly Airflow → evaluate() → JSON → GitHub Action compares to thresholds.

Worked example — score a golden set with the four Ragas metrics

Detailed explanation. The canonical Ragas workflow: a golden .jsonl file of (question, answer, contexts, ground_truth) rows, one evaluate() call returning a scored DataFrame, threshold comparison, JSON dump. Walk through the setup.

Golden set. 100 rows curated by the product manager + support team.
Metrics. All four.
Judge. Claude Haiku via LiteLLM.
Embeddings. OpenAI text-embedding-3-large.

Question. Score the golden set with the four Ragas metrics and emit a JSON report suitable for CI.

Input.

Column	Type	Required for
`question`	str	all metrics
`answer`	str	all metrics
`contexts`	list[str]	all metrics
`ground_truth`	str	`context_precision`, `context_recall`

Code.

# ragas_gate.py
import json
from pathlib import Path
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy, context_precision, context_recall,
)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings

# 1. Load the golden set
rows = [json.loads(l) for l in Path("eval/golden_v3.jsonl").read_text().splitlines()]

# 2. Run the pipeline on each question, populating `answer` and `contexts`
from myapp.rag import rag_chain
for r in rows:
    resp = rag_chain.invoke({"query": r["question"]})
    r["answer"]   = resp["result"]
    r["contexts"] = [d.page_content for d in resp["source_documents"]]

ds = Dataset.from_list(rows)

# 3. Score with Ragas
judge      = LangchainLLMWrapper(ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0))
embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-large"))

result = evaluate(
    ds,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge,
    embeddings=embeddings,
)

# 4. Aggregate and emit
df    = result.to_pandas()
means = df[[m.name for m in [faithfulness, answer_relevancy, context_precision, context_recall]]].mean()
report = {"means": means.to_dict(), "rows": len(df)}
Path("eval/report.json").write_text(json.dumps(report, indent=2))

THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75}
failures = [f"{k}={v:.3f} < {THRESHOLDS[k]}" for k, v in means.items() if v < THRESHOLDS[k]]

if failures:
    print("RAGAS GATE FAILED:", failures)
    raise SystemExit(1)
print("RAGAS GATE PASSED:", means.to_dict())

Step-by-step explanation.

The golden .jsonl starts with just (question, ground_truth) per row; the pipeline call in step 2 populates answer and contexts on the fly. This means the eval is scoring the current pipeline, not a stale snapshot.
LangchainLLMWrapper and LangchainEmbeddingsWrapper adapt any LangChain LLM/embeddings object to Ragas's provider interface. Any LiteLLM-supported model works; picking Haiku for the judge keeps the per-row judge cost around $0.003.
evaluate() runs each metric against each row in parallel (Ragas uses async under the hood). For 100 rows × 4 metrics, expect ~2 minutes wall-clock and ~$1.60 in judge + embedding calls.
The to_pandas() DataFrame has one column per metric, one row per input example, plus the original inputs. Aggregating with .mean() gives you the scalar scores for the gate; retaining the per-row DataFrame lets you drill down to the low-scoring rows.
The gate raises SystemExit(1) on any threshold failure — that's how Airflow / GitHub Actions detect the failure. The report JSON is uploaded as a workflow artifact so the PR comment bot can render it inline.

Output.

Metric	Score	Threshold	Pass?
faithfulness	0.87	0.85	yes
answer_relevancy	0.83	0.80	yes
context_precision	0.71	0.70	yes
context_recall	0.79	0.75	yes

Rule of thumb. For a first Ragas gate, ship reference-free metrics only (faithfulness, answer_relevancy) so you don't need labelled ground truth on day one. Add context_precision / context_recall in week two once you've labelled 30-50 rows. Never gate on a metric whose baseline you haven't measured for two weeks.

Worked example — nightly Airflow DAG + GitHub PR gate

Detailed explanation. The production shape of a Ragas gate is a nightly Airflow DAG (for main-branch trend) plus a per-PR GitHub Action (for the blocking gate). Walk through both.

Nightly DAG. Runs ragas_gate.py against main; stores the JSON in S3; emits Slack alert on regression.
Per-PR gate. GitHub Action runs the same ragas_gate.py against the PR branch; posts scores as a PR comment; fails the check on regression.

Question. Wire the nightly Airflow DAG and the per-PR GitHub Action, sharing the same ragas_gate.py.

Input.

Environment	Trigger	Behaviour on failure
Nightly Airflow	cron 03:00 UTC	Slack alert + preserve trend
Per-PR GitHub Action	pull_request on prompts/, src/rag/	Fail check + comment scores

Code.

# dags/ragas_nightly.py — Airflow DAG
from datetime import datetime, timedelta
from airflow.decorators import dag, task

DEFAULTS = {"owner": "mlops", "retries": 0, "retry_delay": timedelta(minutes=5)}

@dag(dag_id="ragas_nightly",
     start_date=datetime(2026, 7, 1),
     schedule="0 3 * * *",
     catchup=False,
     default_args=DEFAULTS)
def ragas_nightly():
    @task
    def score():
        import subprocess, json
        rc = subprocess.call(["python", "/opt/eval/ragas_gate.py"])
        report = json.loads(open("/opt/eval/eval/report.json").read())
        return {"rc": rc, "report": report}

    @task
    def alert(payload):
        import requests, os
        if payload["rc"] == 0:
            return
        requests.post(os.environ["SLACK_WEBHOOK"], json={
            "text": f":rotating_light: Nightly Ragas gate failed\n```
{% endraw %}
{json.dumps(payload['report'], indent=2)}
{% raw %}
```"
        })

    @task
    def archive(payload):
        import boto3, json, datetime as dt
        s3 = boto3.client("s3")
        key = f"ragas/{dt.date.today():%Y/%m/%d}/report.json"
        s3.put_object(Bucket="mlops-eval",
                       Key=key,
                       Body=json.dumps(payload["report"]).encode())

    p = score()
    alert(p)
    archive(p)

ragas_nightly()

# .github/workflows/ragas-gate.yml
name: Ragas Gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/rag/**"
      - "eval/**"

jobs:
  ragas:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Run Ragas gate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY:    ${{ secrets.OPENAI_API_KEY }}
        run: python eval/ragas_gate.py
      - name: Comment scores on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require("fs");
            let body = "## Ragas gate\n";
            try {
              const rpt = JSON.parse(fs.readFileSync("eval/report.json"));
              body += "| metric | score |\n|---|---|\n";
              for (const [k, v] of Object.entries(rpt.means))
                body += `| ${k} | ${v.toFixed(3)} |\n`;
            } catch (e) { body += "_no report produced_"; }
            github.rest.issues.createComment({
              owner: context.repo.owner, repo: context.repo.repo,
              issue_number: context.issue.number, body
            });
      - name: Upload report artifact
        if: always()
        uses: actions/upload-artifact@v4
        with: { name: ragas-report, path: eval/report.json }

Step-by-step explanation.

Both entry points call the same ragas_gate.py. Sharing the entry point means the nightly trend and the per-PR gate can never diverge in what they measure. Any threshold change is a code review against one file.
The Airflow DAG's score task shells out to the Python script and captures the return code plus the JSON report. alert fires the Slack webhook on failure; archive writes the daily report to S3 at ragas/YYYY/MM/DD/report.json — the historical trend lives in S3, queryable via Athena.
The GitHub Action gates on pull_request events touching prompts, RAG source, or eval configs. paths: filters keep the gate cheap — it doesn't run on markdown-only PRs. if: always() on the comment step ensures the PR gets scored even when the gate fails.
The comment step uses actions/github-script@v7 to render a markdown table of scores. On failure, the PR shows: red X check, comment table with scores, and a workflow artifact holding the raw report.
The 15-minute timeout-minutes guardrails against runaway judge calls (e.g. rate-limit backoffs). If a judge call storms happen, the job fails fast; the gate is treated as "unable to evaluate" and the human reviewer decides whether to re-run or override.

Output.

Scenario	Nightly DAG	PR gate
Baseline (all metrics pass)	archive JSON; no alert	green check; scores as comment
Faithfulness regression on main	Slack alert to `#llm-eval`	n/a
Faithfulness regression on PR	n/a	red X; PR blocked; scores as comment
Judge API rate-limited	retry once, then Slack alert	red X; artifact = partial report

Rule of thumb. Ship the nightly DAG before the per-PR gate. Two weeks of nightly baselines tell you where the thresholds should sit; setting thresholds on day one is guessing. Once the trend is stable, wire the per-PR gate at the observed p10 (the 10th percentile of the last 14 days).

Senior interview question on Ragas as the CI gate

A senior interviewer might ask: "Design a per-PR Ragas gate for a RAG chain. The gate must fail the PR on regression across four metrics, comment scores back on the PR, and archive results daily. Include a custom brand-voice metric. What thresholds do you set on day one?"

Solution Using Ragas evaluate + custom metric + GitHub Action + nightly Airflow

# ragas_full_gate.py — CI gate + custom metric + report emitter
import json, os, sys
from pathlib import Path
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                            context_precision, context_recall)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from custom_metric import BrandVoiceCompliance   # from previous example

GOLDEN = Path("eval/golden_v3.jsonl")
THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75,
              "brand_voice_compliance": 0.75}

def main():
    from myapp.rag import rag_chain
    rows = [json.loads(l) for l in GOLDEN.read_text().splitlines()]
    for r in rows:
        resp = rag_chain.invoke({"query": r["question"]})
        r["answer"]   = resp["result"]
        r["contexts"] = [d.page_content for d in resp["source_documents"]]

    judge = LangchainLLMWrapper(ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0))
    emb   = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-large"))

    result = evaluate(Dataset.from_list(rows),
                      metrics=[faithfulness, answer_relevancy, context_precision,
                                context_recall, BrandVoiceCompliance()],
                      llm=judge, embeddings=emb)
    means = result.to_pandas()[list(THRESHOLDS)].mean().to_dict()

    Path("eval/report.json").write_text(json.dumps(
        {"means": means, "rows": len(rows),
         "commit": os.environ.get("GITHUB_SHA", "local")}, indent=2))

    failures = [f"{k}={v:.3f}<{THRESHOLDS[k]}"
                for k, v in means.items() if v < THRESHOLDS[k]]
    if failures:
        print("GATE FAILED:", failures); sys.exit(1)
    print("GATE PASSED:", means)

if __name__ == "__main__":
    main()

Step-by-step trace.

Layer	Component	Purpose
Dataset	`eval/golden_v3.jsonl`	100 curated `(question, ground_truth)`
Pipeline runner	`run_pipeline()`	populate `answer` + `contexts` from prod code
Metrics	5 (4 Ragas + 1 custom)	faithfulness + answer_relevancy + context_precision + context_recall + brand_voice
Judge	Claude Haiku, temp=0	deterministic scoring
Embeddings	OpenAI text-embedding-3-large	for `answer_relevancy`
Gate	`sys.exit(1)` on any regression	CI-friendly failure
Comment	GitHub Actions bot	markdown table on PR

After deployment, the gate blocks any PR that touches prompts/** or src/rag/** if any of the five metrics drops below its threshold on the 100-row golden set. The nightly Airflow DAG runs the same script against main, alerts on regression, and archives the JSON to S3. Custom metrics live in myapp.eval.metrics and are reviewed like any other code.

Output:

Deliverable	Location
Gate script	`eval/ragas_full_gate.py`
Custom metrics module	`myapp/eval/metrics/brand_voice.py`
PR gate workflow	`.github/workflows/ragas-gate.yml`
Nightly DAG	`dags/ragas_nightly.py`
Archived reports	`s3://mlops-eval/ragas/YYYY/MM/DD/report.json`
Threshold config	`eval/ragas_full_gate.py` (code-reviewed)
Slack alert channel	`#llm-eval`

Why this works — concept by concept:

Reference-free + reference-based split — faithfulness and answer_relevancy bootstrap the gate without needing labelled ground truth; context_precision, context_recall, and custom metrics harden it as labels arrive. This lets the gate ship on day one with two axes and grow to five as the dataset matures.
PydanticPrompt custom metric — extending Ragas with a domain-specific metric is a first-class pattern; the framework handles async, batching, retries, and DataFrame integration. Rolling your own scorer means reimplementing all four; using MetricWithLLM means writing one method.
One entry point for CI and nightly — sharing ragas_full_gate.py between the PR gate and the Airflow DAG guarantees the two can never diverge. Threshold changes are one file, one code review.
JSON report + GitHub bot comment — the eval score becomes a first-class artifact on the PR discussion. Reviewers see the numbers inline; no dashboard round-trip; failures are self-explanatory.
Cost — 100 rows × 5 metrics × ~$0.003 judge cost ≈ $1.50 per PR gate run. Compared to shipping a bad prompt without a gate, this is a rounding error. O(dataset_size × num_metrics) per run; O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on batch quality gates

Practice →

Data validation Topic — data-validation Data validation problems on threshold gating

Practice →

5. Snowflake Cortex Search Ops — eval at the semantic-search layer

`CORTEX SEARCH SERVICE` + `AI_COMPLETE` judges + recall@k in SQL — data, index, model, and eval inside one governance boundary

The mental model in one line: snowflake cortex search is Snowflake's managed hybrid (BM25 + vector) semantic-search primitive, CORTEX SEARCH SERVICE is the DDL that turns a table of text into a searchable index, AI_COMPLETE('claude-3-5-sonnet', ...) is the SQL-level LLM you can use both to generate answers and to judge them, and Cortex Search Ops is the pattern of running eval — recall@k, MRR, nDCG, groundedness — as plain SQL over a labelled (query, expected_doc_id) table so the entire data + index + model + eval stack sits inside one Snowflake account and one governance boundary. This is the eval-in-SQL story that fits banks, healthcare, and government where data-egress is a compliance loss.

The four axes for Cortex Search Ops.

Groundedness. Compute in SQL via AI_COMPLETE('claude-3-5-sonnet', 'Rate 0-1 whether this answer is grounded in the retrieved context: ...'). The judge call is one function invocation; parse the numeric response.
Answer relevance. Same pattern — AI_COMPLETE with a "does the answer address the question?" prompt. Cortex handles the model round-trip inside the account; no external API needed.
Context precision / recall. Pure IR SQL. Label (query, expected_doc_id) pairs in a table; run the search; compute recall@k, MRR, nDCG@k with aggregate SQL. This is the classic search-eval pattern reborn.
Latency + cost. Query history table (SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY) captures every CORTEX SEARCH_PREVIEW and every AI_COMPLETE call with wall-clock and credits burned. Join to your eval table for per-query cost.

The CORTEX SEARCH SERVICE shape.

DDL. CREATE OR REPLACE CORTEX SEARCH SERVICE svc ON <text_col> ATTRIBUTES <cols> WAREHOUSE = <wh> TARGET_LAG = '<n> minute' AS <SELECT ...> — one command; Snowflake handles chunking, embedding, and indexing.
Refresh. TARGET_LAG sets how stale the index can be before Snowflake re-indexes. TARGET_LAG = '1 hour' for near-real-time; '1 day' for cheaper batch.
Query. SELECT PARSE_JSON(SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', 'my query', 8)) returns the top-8 matching rows plus their scores.
Hybrid retrieval. Cortex Search is hybrid (BM25 + vector) under the hood; no separate config needed.

Eval-in-SQL — the recall@k pattern.

Labels. eval_labels(query_id, query_text, expected_doc_id) — one row per (query, relevant doc) pair. Multi-relevance = multiple rows per query.
Retrievals. eval_retrievals(query_id, retrieved_doc_id, rank) — populated by running Cortex Search on every query.
recall@k. SUM(CASE WHEN retrieved.doc_id = labels.doc_id AND rank <= k THEN 1 ELSE 0 END) / COUNT(DISTINCT labels.query_id) per query, then averaged.
MRR / nDCG. Standard IR formulas, expressible in pure SQL with window functions.

Common interview probes on Cortex Search Ops.

"Why eval in SQL?" — required answer: data + index + model + eval in one governance boundary; no data egress.
"How does Cortex Search compare to Postgres pgvector?" — Cortex is managed hybrid retrieval + indexing + refresh; pgvector is a vector column you index yourself.
"How would you compute recall@k in SQL?" — required answer: join labels to retrievals, aggregate the match-rank ≤ k signal.
"How is AI_COMPLETE different from Bedrock direct?" — Cortex bills through Snowflake credits, stays inside the account, no external API.

Worked example — build a Cortex Search Service and score recall@5

Detailed explanation. The canonical Snowflake eval workflow: create a CORTEX SEARCH SERVICE on a docs table, label a (query, expected_doc_id) table, run search across all labelled queries, compute recall@5 with a single SELECT. Walk through it end-to-end.

Docs table. docs(doc_id, title, body, updated_at) — 80,000 rows.
Search service. docs_svc on the body column, target lag 1 hour.
Labels. eval_labels(query_id, query_text, expected_doc_id) — 200 rows across 100 queries (multi-relevance).
Metric. recall@5.

Question. Create the search service, run it across the label queries, and compute recall@5 in one SQL query.

Input.

Object	Purpose
`docs`	source text
`docs_svc`	Cortex Search Service (hybrid)
`eval_labels`	100 queries × ~2 expected docs each
`eval_retrievals`	populated by search-per-query

Code.

-- 1. Source docs table (assume already populated)
-- CREATE TABLE docs (doc_id STRING, title STRING, body STRING, updated_at TIMESTAMP);

-- 2. Create the search service (indexes body; keeps title as an attribute)
CREATE OR REPLACE CORTEX SEARCH SERVICE docs_svc
    ON body
    ATTRIBUTES title, updated_at
    WAREHOUSE  = eval_wh
    TARGET_LAG = '1 hour'
AS (
    SELECT doc_id, title, body, updated_at
    FROM   docs
);

-- 3. Labels table (populate by hand or via product-team CSV)
CREATE OR REPLACE TABLE eval_labels (
    query_id         STRING,
    query_text       STRING,
    expected_doc_id  STRING
);
-- INSERT INTO eval_labels VALUES ('q1', 'how to rotate API key', 'doc_042'), ...

-- 4. Populate eval_retrievals by running search on every label query
CREATE OR REPLACE TABLE eval_retrievals AS
SELECT
    l.query_id,
    l.query_text,
    r.value:doc_id::STRING AS retrieved_doc_id,
    r.value:score::FLOAT   AS score,
    r.index + 1            AS rank
FROM   (SELECT DISTINCT query_id, query_text FROM eval_labels) l,
LATERAL FLATTEN (
    input => PARSE_JSON(
        SNOWFLAKE.CORTEX.SEARCH_PREVIEW(
            'docs_svc',
            l.query_text,
            10                                 -- top-10 for headroom
        )
    ):results
) r;

-- 5. recall@5 in one SQL — for each query, did any expected doc appear in top-5?
WITH per_query AS (
    SELECT
        l.query_id,
        MAX(CASE WHEN r.retrieved_doc_id = l.expected_doc_id AND r.rank <= 5 THEN 1 ELSE 0 END) AS hit
    FROM       eval_labels    l
    LEFT JOIN  eval_retrievals r
      ON r.query_id = l.query_id
    GROUP BY l.query_id
)
SELECT AVG(hit) AS recall_at_5
FROM   per_query;

Step-by-step explanation.

CREATE CORTEX SEARCH SERVICE is the one-command index. Snowflake chunks the body column, embeds each chunk with its managed embedding model, and stores the hybrid index. TARGET_LAG = '1 hour' means the index is at most 1 hour behind writes to the source table.
SNOWFLAKE.CORTEX.SEARCH_PREVIEW('docs_svc', query, 10) returns a JSON structure with a results array of up to 10 top-k rows. LATERAL FLATTEN unpacks that JSON into rows; r.index + 1 gives the 1-based rank.
The eval_retrievals table is a snapshot of what the search returned for each label query at eval time. Persisting it (rather than joining on-the-fly) means you can re-analyse different k values, compute MRR / nDCG, or diff two eval snapshots without re-running the search.
The recall@5 SELECT joins labels to retrievals on query_id; per query, it checks whether any expected doc appears in the top-5 retrieved. MAX(CASE WHEN ...) is the "any match" reduction; AVG(hit) across queries gives the scalar recall@5.
For multi-relevance recall (average of "fraction of expected docs hit"), swap MAX for AVG. For strict recall (all expected docs must appear), swap for MIN. The SQL primitive is flexible; the reporting choice depends on the domain.

Output.

query_id	expected in top-5?	rank hit
q1 (rotate API key)	yes	1
q2 (SLA at 3am)	yes	3
q3 (export PDF)	no	—
... (98 more)	...	...
recall@5	0.88	—

Rule of thumb. Store eval_retrievals as a persistent table, not an inline subquery. Re-computing metrics on the stored retrievals is a millisecond query; re-running the retrievals is a minutes-long recall of the search service. Persistence unlocks fast metric iteration.

Worked example — MRR + nDCG@10 + `AI_COMPLETE` groundedness in SQL

Detailed explanation. Recall@k is the entry-level metric; MRR (Mean Reciprocal Rank) rewards top-ranked hits; nDCG@k weighs earlier positions higher; AI_COMPLETE groundedness scores generated answers. All four fit in one SQL script. Walk through them.

MRR. For each query, take 1 / rank_of_first_expected_doc; average across queries.
nDCG@10. For each query, sum (2^rel - 1) / log2(rank + 1) up to rank 10; normalise by ideal DCG.
Groundedness. For each generated answer + its retrieved contexts, AI_COMPLETE a rating prompt; parse the score.

Question. Write the SQL that computes MRR, nDCG@10, and groundedness in one script.

Input.

Metric	Requires
MRR	`eval_retrievals` + `eval_labels`
nDCG@10	+ graded relevance in `eval_labels`
Groundedness	`eval_answers(query_id, answer)` + `eval_retrievals` (as contexts)

Code.

-- MRR: mean reciprocal rank of the first expected hit
WITH first_hit AS (
    SELECT l.query_id,
           MIN(r.rank) AS first_hit_rank
    FROM       eval_labels    l
    LEFT JOIN  eval_retrievals r
      ON r.query_id = l.query_id AND r.retrieved_doc_id = l.expected_doc_id
    GROUP BY l.query_id
)
SELECT AVG(CASE WHEN first_hit_rank IS NULL THEN 0 ELSE 1.0 / first_hit_rank END) AS mrr
FROM   first_hit;

-- nDCG@10: normalised discounted cumulative gain (binary relevance)
WITH dcg AS (
    SELECT r.query_id,
           SUM(CASE WHEN EXISTS (SELECT 1 FROM eval_labels l
                                  WHERE l.query_id = r.query_id
                                    AND l.expected_doc_id = r.retrieved_doc_id)
                    THEN 1.0 / LOG(2, r.rank + 1) ELSE 0 END) AS dcg_val
    FROM   eval_retrievals r
    WHERE  r.rank <= 10
    GROUP BY r.query_id
),
idcg AS (
    SELECT l.query_id,
           SUM(1.0 / LOG(2, rn + 1)) AS idcg_val
    FROM (
        SELECT query_id,
               ROW_NUMBER() OVER (PARTITION BY query_id ORDER BY query_id) AS rn
        FROM eval_labels
    ) l
    WHERE rn <= 10
    GROUP BY l.query_id
)
SELECT AVG(COALESCE(dcg.dcg_val, 0) / NULLIF(idcg.idcg_val, 0)) AS ndcg_at_10
FROM   idcg
LEFT   JOIN dcg USING (query_id);

-- Groundedness: AI_COMPLETE judge per (answer, contexts)
CREATE OR REPLACE TABLE eval_groundedness AS
WITH ctx AS (
    SELECT r.query_id,
           LISTAGG(d.body, '\n---\n') WITHIN GROUP (ORDER BY r.rank) AS contexts
    FROM   eval_retrievals r
    JOIN   docs           d USING (retrieved_doc_id, doc_id)
    WHERE  r.rank <= 5
    GROUP BY r.query_id
)
SELECT a.query_id,
       a.answer,
       AI_COMPLETE(
           'claude-3-5-haiku',
           'Rate 0.0-1.0 whether the ANSWER is fully grounded in the CONTEXT. '
           || 'Return only the number.\n\n'
           || 'ANSWER:\n' || a.answer || '\n\n'
           || 'CONTEXT:\n' || ctx.contexts
       )::FLOAT AS groundedness
FROM   eval_answers a
JOIN   ctx          USING (query_id);

SELECT AVG(groundedness) AS mean_groundedness FROM eval_groundedness;

Step-by-step explanation.

MRR uses MIN(rank) per query to find the first hit, then 1/rank. Queries with no hit contribute 0. AVG across queries gives the scalar.
nDCG@10 computes DCG (discounted cumulative gain) per query using the standard 1/log2(rank+1) weighting; IDCG (ideal DCG) is the same formula with all relevant docs assumed at ranks 1..N; the ratio is nDCG. Snowflake's LOG(base, x) is the native log function.
The groundedness pass uses LISTAGG to concatenate the top-5 retrieved doc bodies into one context string per query, then passes answer + contexts to AI_COMPLETE with a rating prompt. AI_COMPLETE returns a string; ::FLOAT casts the numeric response.
All three metrics live in one Snowflake account. No data crosses a boundary; no external API call; the entire eval is auditable via SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY. This is the compliance advantage.
Cost breakdown: recall@5 / MRR / nDCG are pure SQL (~cents per run against a small labels table); groundedness costs one AI_COMPLETE call per query (~$0.002 per query with Haiku). 100 queries × $0.002 = ~$0.20 per full eval run.

Output.

Metric	Score
recall@5	0.88
MRR	0.72
nDCG@10	0.81
mean_groundedness	0.86
Total cost per run	~$0.20
Data egress	0 bytes

Rule of thumb. Persist per-metric result tables (eval_recall, eval_mrr, eval_groundedness) with a run_at timestamp. Diffing two runs is one SELECT; watching a trend is a Snowsight chart; alerting on a regression is a Snowflake Task + email. All four are already Snowflake-native.

Senior interview question on eval-in-SQL at the Cortex Search layer

A senior interviewer might ask: "You've got a Snowflake-native RAG using CORTEX SEARCH SERVICE and AI_COMPLETE. Nothing can leave the account. Design the nightly eval — recall@5, MRR, groundedness — as pure SQL with a TASK, ALERT, and email-on-regression. What are the four objects you deploy?"

Solution Using Cortex Search Service + AI_COMPLETE judge + TASK + ALERT

-- 1. Search service + history table (deployed once)
CREATE OR REPLACE CORTEX SEARCH SERVICE docs_svc
    ON body ATTRIBUTES title, updated_at
    WAREHOUSE = eval_wh TARGET_LAG = '1 hour'
AS (SELECT doc_id, title, body, updated_at FROM docs);

CREATE OR REPLACE TABLE eval_history (
    run_at TIMESTAMP_NTZ, run_id STRING, metric STRING, score FLOAT);

-- 2. Stored procedure: recall@5 + MRR + groundedness in one call
CREATE OR REPLACE PROCEDURE sp_run_cortex_eval() RETURNS STRING LANGUAGE SQL AS $$
DECLARE run_id STRING DEFAULT UUID_STRING();
BEGIN
    CREATE OR REPLACE TEMP TABLE _retr AS
    SELECT l.query_id, l.query_text,
           r.value:doc_id::STRING AS retrieved_doc_id, r.index + 1 AS rank
    FROM (SELECT DISTINCT query_id, query_text FROM eval_labels) l,
    LATERAL FLATTEN(input => PARSE_JSON(
        SNOWFLAKE.CORTEX.SEARCH_PREVIEW('docs_svc', l.query_text, 10)):results) r;

    INSERT INTO eval_history
    SELECT CURRENT_TIMESTAMP(), :run_id, 'recall_at_5', AVG(hit)
    FROM (SELECT l.query_id, MAX(CASE WHEN r.retrieved_doc_id = l.expected_doc_id AND r.rank <= 5 THEN 1 ELSE 0 END) hit
          FROM eval_labels l LEFT JOIN _retr r ON r.query_id = l.query_id GROUP BY l.query_id);

    INSERT INTO eval_history
    SELECT CURRENT_TIMESTAMP(), :run_id, 'mrr',
           AVG(CASE WHEN first_rank IS NULL THEN 0 ELSE 1.0 / first_rank END)
    FROM (SELECT l.query_id, MIN(r.rank) first_rank
          FROM eval_labels l LEFT JOIN _retr r
            ON r.query_id = l.query_id AND r.retrieved_doc_id = l.expected_doc_id
          GROUP BY l.query_id);

    INSERT INTO eval_history
    WITH ctx AS (SELECT r.query_id, LISTAGG(d.body, '\n---\n') WITHIN GROUP (ORDER BY r.rank) contexts
                 FROM _retr r JOIN docs d ON r.retrieved_doc_id = d.doc_id
                 WHERE r.rank <= 5 GROUP BY r.query_id)
    SELECT CURRENT_TIMESTAMP(), :run_id, 'groundedness',
           AVG(AI_COMPLETE('claude-3-5-haiku',
               'Rate 0.0-1.0 whether ANSWER is grounded in CONTEXT. Return the number only.\n\n'
               || 'ANSWER:\n' || a.answer || '\n\nCONTEXT:\n' || ctx.contexts)::FLOAT)
    FROM eval_answers a JOIN ctx USING (query_id);

    RETURN 'ok:' || :run_id;
END; $$;

-- 3. TASK: nightly at 03:00 UTC
CREATE OR REPLACE TASK t_nightly_eval
    WAREHOUSE = eval_wh SCHEDULE = 'USING CRON 0 3 * * * UTC'
AS CALL sp_run_cortex_eval();
ALTER TASK t_nightly_eval RESUME;

-- 4. ALERT: fire when any metric drops below trailing 30-day p10 × 0.95
CREATE OR REPLACE ALERT a_eval_regression
    WAREHOUSE = eval_wh SCHEDULE = '30 MINUTE'
    IF (EXISTS (
        WITH latest AS (
            SELECT metric, score FROM eval_history
            QUALIFY ROW_NUMBER() OVER (PARTITION BY metric ORDER BY run_at DESC) = 1),
        baseline AS (
            SELECT metric, PERCENTILE_CONT(0.10) WITHIN GROUP (ORDER BY score) p10
            FROM eval_history WHERE run_at >= DATEADD(day, -30, CURRENT_TIMESTAMP())
            GROUP BY metric)
        SELECT 1 FROM latest l JOIN baseline b USING (metric) WHERE l.score < b.p10 * 0.95))
    THEN CALL SYSTEM$SEND_EMAIL('mlops_alerts', 'mlops@acme.com',
        'Cortex Search eval regression',
        'One or more Cortex Search metrics dropped below the 30-day p10.');
ALTER ALERT a_eval_regression RESUME;

Step-by-step trace.

Layer	Object	Purpose
Index	`CORTEX SEARCH SERVICE docs_svc`	hybrid BM25 + vector search on `docs.body`
Labels	`eval_labels`	curated `(query_id, query_text, expected_doc_id)`
Retrievals	temp `_retr` inside SP	rebuilt each run
History	`eval_history`	time series of all metric scores
Judge	`AI_COMPLETE('claude-3-5-haiku', ...)`	in-account LLM judge
Scheduler	`TASK t_nightly_eval`	daily at 03:00 UTC
Alert	`ALERT a_eval_regression`	every 30 min; email on p10 regression

After deployment, every night at 03:00 UTC Snowflake reruns the eval end-to-end, appends four rows to eval_history, and the ALERT fires an email if any metric drops 5% below the 30-day p10. The entire pipeline is Snowflake-native — no Airflow, no Kubernetes, no external SMTP; the compliance team has one governance boundary to audit.

Output:

Deliverable	Snowflake object
Search index	`CORTEX SEARCH SERVICE docs_svc`
Metric history	`eval_history` table
Runner	`PROCEDURE sp_run_cortex_eval`
Scheduler	`TASK t_nightly_eval`
Regression detector	`ALERT a_eval_regression`
Alert channel	email to `mlops@acme.com`
Total credits/day	~2-5 (small warehouse × short runtime)
Data egress	0 bytes

Why this works — concept by concept:

One governance boundary — data (docs), index (docs_svc), model (AI_COMPLETE), judge (same AI_COMPLETE), eval history (eval_history), scheduler (TASK), and alerter (ALERT) are all inside one Snowflake account. Nothing crosses a compliance boundary. This is the decisive advantage over any external eval stack in regulated environments.
Cortex Search hybrid retrieval — Snowflake's CORTEX SEARCH SERVICE is BM25 + vector out of the box; no separate embedding pipeline, no separate index cluster. TARGET_LAG handles refresh; the DDL is one command.
AI_COMPLETE as SQL judge — LLM calls are just another SQL function. You can wrap them in AVG, join them, filter them, LISTAGG inputs, cast outputs. This is the "eval in SQL" pattern taken to its logical conclusion.
TASK + ALERT native scheduler — Snowflake's TASK is a cron; ALERT is a change-detection primitive; both are managed, both are governed by RBAC. No external Airflow or PagerDuty needed for the core loop.
Cost — ~2-5 credits/day for a small eval warehouse running a ~3-minute nightly job, plus ~$0.20 in AI_COMPLETE judge calls. Compared to shipping a bad Cortex Search config without a nightly gate, the alert-lead-time saving alone is worth 100× the eval cost. O(N_labels × k) for retrieval; O(N_queries) for judge; O(1) latency to detect regressions.

SQL
Topic — sql
SQL problems on IR metrics and window functions

Practice →

Design
Topic — design
Design problems on in-warehouse eval systems

Practice →

Cheat sheet — LLM eval recipes

Which tool when. LangSmith when you're already inside LangChain/LangGraph and want a hosted trace + dataset + experiment UI. TruLens when you need local-first, VPC-safe, framework-agnostic instrumentation with the RAG triad. Ragas when you need a metric library that returns a scored DataFrame for the CI gate (nightly Airflow + per-PR GitHub Action). Cortex Search Ops when the entire data + index + model + eval stack lives inside Snowflake and compliance forbids data egress.
The four axes to score. Groundedness (fraction of answer claims supported by context — anti-hallucination), answer relevance (does the answer address the question), context precision + recall (retriever quality), latency + cost (pipeline axis). Any single-axis eval hides regressions on the other three.
LangSmith trace boilerplate. os.environ["LANGSMITH_TRACING"]="true"; os.environ["LANGSMITH_PROJECT"]="prod" before importing LangChain — auto-traces every LLM call. @traceable(run_type="chain", name="answer_endpoint", tags=["prod","v2.3"]) on FastAPI handlers to add the top-level Trace. Sample in production with LANGSMITH_TRACING_SAMPLE_RATE=0.01.
LangSmith dataset + pairwise gate. ls.evaluate(target_fn, data="dataset_v3", experiment_prefix="candidate_v2") runs a candidate against a pinned dataset version. evaluate_comparative(experiments=[old, new], evaluators=[pairwise_helpfulness]) scores which experiment wins per row. Gate on win-rate ≥ 0.60 (excluding ties).
TruLens RAG triad boilerplate. Wrap the app with TruChain(chain, app_name="rag", app_version="v2.3", feedbacks=[f_ctx_rel, f_grounded, f_answer_rel]). Each Feedback binds a provider metric plus selectors: context_selector = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content. Aggregate per-chunk relevance with np.mean; collect contexts for groundedness with .collect().
TruLens VPC production pattern. TruSession(database_url="postgresql://...") keeps every Record + score inside a VPC Postgres. LiteLLM(model_engine="bedrock/anthropic.claude-3-5-haiku-...") keeps the judge in-region. Score 2% sync for real-time alerts; nightly batch back-fill the remaining 98% for zero user-latency impact and batch-price discounts.
Ragas gate script. evaluate(Dataset.from_list(rows), metrics=[faithfulness, answer_relevancy, context_precision, context_recall], llm=judge, embeddings=emb) returns a Result with .to_pandas(). Aggregate with .mean(), compare against THRESHOLDS dict, sys.exit(1) on any regression. Ship reference-free metrics first (faithfulness, answer_relevancy); add reference-based (context_precision, context_recall) as labels arrive.
Ragas CI wiring. One ragas_gate.py shared between the per-PR GitHub Action (pull_request on prompts/**, src/rag/**) and the nightly Airflow DAG (schedule="0 3 * * *"). PR posts markdown-table scores as a comment; nightly alerts Slack on regression and archives report JSON to S3 at ragas/YYYY/MM/DD/report.json.
Cortex Search Service + eval-in-SQL. CREATE CORTEX SEARCH SERVICE svc ON body WAREHOUSE=wh TARGET_LAG='1 hour' AS (SELECT ... FROM docs) for the managed hybrid index; SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', q, k) for retrieval. Eval: recall@k = AVG(MAX(CASE WHEN retrieved_doc_id=expected_doc_id AND rank<=k THEN 1 ELSE 0 END)); MRR = AVG(1.0 / MIN(first_hit_rank)); groundedness = AVG(AI_COMPLETE('claude-3-5-haiku', prompt)::FLOAT).
Cortex Search Ops scheduler. TASK t_nightly_eval SCHEDULE='USING CRON 0 3 * * * UTC' AS CALL sp_run_cortex_eval(); + ALERT a_eval_regression SCHEDULE='30 MINUTE' IF (EXISTS <latest score < 30-day p10 × 0.95>) THEN CALL SYSTEM$SEND_EMAIL(...). Data, index, model, judge, eval, scheduler, alerter — all in one Snowflake account.
Metric decision matrix. Groundedness → Ragas faithfulness OR TruLens groundedness OR Cortex AI_COMPLETE judge. Answer relevance → Ragas answer_relevancy OR TruLens relevance. Retriever quality → Ragas context_precision/context_recall OR Cortex recall@k/MRR/nDCG. Latency + cost → LangSmith Run.latency/Run.total_cost OR TruLens record.latency/record.cost OR Snowflake QUERY_HISTORY.
CI gate thresholds — day-one starting points. Faithfulness ≥ 0.85, answer_relevancy ≥ 0.80, context_precision ≥ 0.70, context_recall ≥ 0.75, latency p95 ≤ 4s, cost per query ≤ $0.015. Measure the observed baseline for two weeks before setting thresholds; day-one thresholds are guesses. Lift thresholds as the pipeline matures; never lower them without a written justification.
Judge selection + validation. Use a cheap fast judge (Claude Haiku, Gemini Flash) with temperature=0 for determinism. Re-validate quarterly against 100 human-labelled rows; if judge-human correlation drops below 0.75, swap the judge model or add few-shot examples to the prompt. Pairwise judges are more sensitive than pointwise judges for A/B comparisons; use pointwise for absolute thresholds.
Cost budgeting rules. Judge cost ≈ 1-2× production LLM cost per evaluated row. Sample 1-5% of production for sync scoring; back-fill 100% via batch (50% discount on most providers). CI gate over 100-row golden set ≈ $1-2 per PR run. Never run 100% sync judges in production; the extra latency and cost destroy the value proposition.

Frequently asked questions

What is LLM evaluation for data pipelines in one sentence?

LLM evaluation for data pipelines is the discipline of scoring every LLM-driven output your pipeline produces — RAG answers, summaries, classifications, SQL-generation results — against a small set of orthogonal quality axes (groundedness, answer relevance, context precision / recall, latency + cost) so hallucinations, retrieval misses, prompt drift, and cost explosions are caught by an automated gate rather than by a customer three weeks later. In 2026 the four canonical tools are LangSmith (trace-first hosted platform with versioned datasets and hosted judges), TruLens (local-first, VPC-safe, framework-agnostic with the RAG triad), Ragas (reference-free metric library that returns a scored DataFrame for CI gates), and Snowflake Cortex Search Ops (eval-in-SQL with AI_COMPLETE judges when the entire data + model stack lives in Snowflake). The tools coexist because their sweet spots don't overlap; the wrong senior answer is picking one and calling the eval done. Every senior data-engineering interview probes LLM eval because it's the load-bearing correctness discipline for the modern RAG + agent stack.

LangSmith vs TruLens vs Ragas — which do I pick?

Default to LangSmith when your app already lives inside LangChain or LangGraph and you want a hosted UI: trace tree, versioned datasets, hosted evaluators, pairwise experiments, all in one dashboard with per-Run latency and cost attached for free. Pick TruLens when you need eval inside a VPC or want to instrument a non-LangChain Python app: TruChain / TruLlama / TruCustomApp wrap any Python function; the RAG triad (context relevance, groundedness, answer relevance) is the canonical three-edge completeness proof; storage defaults to local SQLite and swaps to Postgres for team use; no vendor round-trip is required at any point. Pick Ragas when you want a metric library that plays well with any CI stack: evaluate() returns a pandas DataFrame; two metrics (faithfulness, answer_relevancy) need no reference answer so bootstrapping is cheap; two metrics (context_precision, context_recall) use labelled ground truth; the whole thing runs headless in an Airflow DAG or a GitHub Action. In practice most teams end up running two of the three: LangSmith or TruLens for traces + interactive UI, Ragas for the CI gate. The three are complementary, not competitive.

What is the RAG triad?

The RAG triad is TruLens's canonical eval completeness proof for a Retrieval-Augmented Generation system: a triangle whose three vertices are the query, the retrieved context, and the generated answer, and whose three edges are the three orthogonal quality axes that must all pass for the RAG to be trustworthy. The edges are (a) context relevance — is the retrieved context actually relevant to the query? — a low score points at the retriever; (b) groundedness — is every claim in the generated answer supported by the retrieved context? — a low score points at hallucination; (c) answer relevance — does the generated answer actually address the query, or does it wander off topic? — a low score points at the prompt or the model. Any one of the three can regress independently, so scoring only one hides regressions on the other two. The triad is why TruLens's default instrumentation ships three Feedback functions rather than one, and why senior interviewers probe the triangle rather than a single "hallucination score" — a candidate who names all three edges signals they understand RAG eval as a multi-dimensional discipline.

Can I do LLM eval inside Snowflake?

Yes — snowflake cortex search plus AI_COMPLETE plus the standard TASK + ALERT primitives give you a fully in-warehouse eval loop. The CORTEX SEARCH SERVICE DDL is one command; it manages hybrid BM25 + vector indexing, chunking, embedding, and refresh via TARGET_LAG. SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', query, k) returns top-k results as JSON, which LATERAL FLATTEN unpacks into rows for eval joins against a labelled eval_labels(query_id, expected_doc_id) table. Recall@k, MRR, and nDCG@k are pure aggregate SQL; groundedness is AI_COMPLETE('claude-3-5-haiku', 'Rate 0-1 ...')::FLOAT wrapped in AVG. A Snowflake TASK schedules the whole thing on cron; a Snowflake ALERT fires an email when the latest score drops below the trailing 30-day p10 by more than 5%. The decisive advantage is governance — data, index, model, judge, history, scheduler, and alerter all live inside one Snowflake account with one RBAC boundary. For banks, healthcare, and government workloads where data egress is a compliance loss, Cortex Search Ops is the default answer.

Groundedness vs answer relevance — how are they different?

Groundedness asks "is every claim in the answer supported by the retrieved context?" — it's the anti-hallucination axis. An answer that invents a fact the retrieved documents do not contain scores low on groundedness even if the fact happens to be correct in the world. Ragas calls this metric faithfulness; TruLens calls it groundedness; both split the answer into atomic claims, ask an LLM judge whether each claim is entailed by the retrieved context, and aggregate. Answer relevance asks "does the answer address the user's question?" — it's the on-topic axis. An answer that is perfectly grounded (every claim is in the context) but doesn't address the question ("here is the entire policy document") scores low on answer relevance. Ragas measures answer_relevancy by generating hypothetical questions the answer could address, then computing cosine similarity between those questions and the original query. The two metrics are orthogonal: a RAG chain can be perfectly grounded and off-topic, or on-topic and hallucinating; measuring only one hides the other's failure. Every RAG eval must score both.

Do I need reference answers to evaluate a RAG?

No — this is why Ragas ships two reference-free metrics: faithfulness and answer_relevancy. faithfulness needs only the generated answer and the retrieved contexts — an LLM judge scores whether the answer's claims are entailed by the context, and no ground-truth answer is required. answer_relevancy needs only the question and the answer — the metric generates hypothetical questions the answer could address and compares to the original query via cosine similarity. Both metrics let you bootstrap an eval gate on day one, before any human has written a single labelled answer. Reference-based metrics like context_precision, context_recall, and answer_correctness do need labelled ground truth (an expected_doc_id per query, or a written ground_truth answer), and those take human curation time — budget 5-10 minutes per row. The standard adoption path is: start with the two reference-free metrics + a small human-curated (question, expected_doc_id) table for retrieval metrics; grow the ground-truth answers as the eval matures. TruLens's RAG triad and LangSmith's hosted criteria evaluators also work reference-free, so the three tools all support this bootstrap-first pattern.

Practice on PipeCode

Drill the ETL practice library → for the eval-gate, incremental scoring, and RAG ingestion pipeline problems senior interviewers love.
Rehearse on the data-validation practice library → for the threshold-gating, batch scoring, and quality-metric patterns.
Sharpen the systems axis with the design practice library → for VPC-first eval systems, in-warehouse eval loops, and pairwise experiment design.
Level up your Python instrumentation on the Python practice library → for decorator patterns, wrapper design, and async LLM-judge batching.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis measurement plan against real graded inputs.

Lock in LLM eval muscle memory

Docs explain metrics. PipeCode drills explain the decision — when to gate on faithfulness, when the RAG triad's context edge is the retriever's report card, when Ragas beats LangSmith for CI, when Cortex Search Ops wins on governance. Pipecode.ai is Leetcode for Data Engineering — axis-first practice tuned for the production trade-offs senior data and MLOps engineers actually face.

Practice ETL problems →
Practice design problems →